Mohamed Houri’s Oracle Notes

August 12, 2017

Unnesting of coalesced subqueries

Filed under: Oracle — hourim @ 3:49 pm

The 12cR1 way used by Oracle to coalesce two correlated disjunctive subqueries has prevented a real life query from unnesting its two subqueries causing a serious performance issue. This article examines a representative model of this real life query. The first paragraph explains the any-to-exists subquery conversion. The second paragraph shows how two correlated exists subqueries can be coalesced to form a single subquery. Once these two transformations introduced the third paragraph will examine how the previously coalesced subqueries can be unnested. It will particularly show that the ability of Oracle to unnest two coalesced correlated subqueries depends strongly on the coalescing method as this one differs from version to version.

1. Any-to-exists subquery conversion

Before coalescing two (or more) subqueries Oracle starts first by transforming ANY and ALL subqueries into EXISTS and NOT EXISTS respectively. Consider, for example, the following query:

select
    a.id1
   ,a.n1
   ,a.start_date
from 
    t1 a
where 
   (a.id1 in
         (select b.id
          from t2 b
          where b.status = 'COM')
   or a.id1 in
         (select c.id1
          from t2 c
          where c.status = 'ERR')
     );

If you examine its corresponding 10053 CBO trace file you will find that its two disjunctive subqueries have been first converted from an any-to-exists subqueries as shown below:

Query After Exists Conversion:******* UNPARSED QUERY IS *******

select
  a.id1 id1,
  a.n1 n1,
  a.start_date start_date
from
  t1 a
where
  exists
  ( select b.id id
    from t2 b
    where b.status='COM'
    and   a.id1 =b.id)
or exists
  ( select c.id1 id1
    from t2 c
    where c.status='ERR'
    and a.id1 =c.id1)
  );
 

This is generally the preduled to or, a preliminary part of, a subquery coalescing transformation presented in the next section

2. Subquery Coalescing

The preceding any-to-exists transformed query is nothing else than an instance of the following simplified one:

select 
  {list of columns}
from
  table
where 
  predicate

The predicate in this particular case is acting as two disjunctive correlated subqueries of the same type:

-- first subquery
exists
  ( select b.id id
    from t2 b
    where b.status = 'COM'
    and   a.id1    = b.id)

-- second subquery
 exists
  ( select c.id1 id1
    from t2 c
    where c.status ='ERR'
    and a.id1 = c.id1)

If these two subqueries are not coalesced they will probably generate two accesses to the same table t2. This is why Oracle can, under certain conditions, coalesce two conjunctive or disjucntive subqueries into a single one. Applied to the current query the coalescing process yields the following new transformed query:

Query After Ex/Any Disj SQ coalescing:******* UNPARSED QUERY IS *******

select
  a.id1 id1,
  a.n1 n1,
  a.start_date start_date
from
  t1 a
where
  exists
  (
    select
      0 id
    from
      t2 b
    where
      b.status  = 'COM'
    and a.id1   = b.id
    or b.status ='ERR'
    and a.id1   = b.id1
  );

The execution plan of the doubly transformed query is shown below:

alter session set statistics_level=all;
select * from table(dbms_xplan.display_cursor(null,null, 'allstats last'));

--------------------------------------------------------------
| Id  | Operation          | Name | Starts | E-Rows | A-Rows |
--------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |      1 |        |   9890 |
|*  1 |  FILTER            |      |      1 |        |   9890 |
|   2 |   TABLE ACCESS FULL| T1   |      1 |  10000 |  10000 |
|*  3 |   TABLE ACCESS FULL| T2   |  10000 |      1 |   9890 |
--------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter( IS NOT NULL)
   3 - filter((("B"."ID"=:B1 AND "B"."STATUS"='COM') OR ("B"."ID1"=:B2 AND
              "B"."STATUS"='ERR')))

If you still have any doubts about whether the two subqueries have been coalesced or not then the single apparition of table T2 in the plan together with the predicate n°3 will certainly clear out this doubt for good. Indeed, it is the subquery coalescing process that resulted into Oracle visiting table T2 only one time.

In addition in the outline data used to generate the above execution plan reproduced here below:

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('12.2.0.1')
      DB_VERSION('12.2.0.1')
      ALL_ROWS
      OUTLINE_LEAF(@"SEL$ACD206C8")
      COALESCE_SQ(@"SEL$3")
      OUTLINE_LEAF(@"SEL$1")
      OUTLINE(@"SEL$7C83FF7B")
      COALESCE_SQ(@"SEL$2")
      OUTLINE(@"SEL$3")
      OUTLINE(@"SEL$2")
      FULL(@"SEL$1" "A"@"SEL$1")
      PQ_FILTER(@"SEL$1" SERIAL)
      FULL(@"SEL$ACD206C8" "B"@"SEL$2")
      END_OUTLINE_DATA
  */

The COALESCE_SQ(@”SEL$3″) and COALESCE_SQ(@”SEL$2″)represent the hints used to force a subquery coalescing transformation. If you want to cancel such a subquery coalescing transformation then the NO_COALESCE_SQ(qb_name) hint will be the one you should use. You can also change the hidden parameter _optimizer_coalesce_subqueries governing this transformation which defaults to true in 12cR1.

3. Subquery unnesting
        3.1. The 12cR1 effect

The any-to-exist conversion followed by the subquery coalescing should have normally paved the way for subquery unnesting transformation. Unfortunately, the real life query, which the actual model is a true representation of, the coalesced subqueries have not been unnested altering dramatically the performance of the original query as shown via its corresponding sql monitor report:

-- Get the report_id corresponding to the triplet (sql_id,sql_exec_id, sql_exec_start)
 select 
      report_id
    , key1 sql_id
    , key2 sql_exec_id
    , key3 sql_exec_start
  from
     dba_hist_reports
  where
     component_name ='sqlmonitor'
  and key1 = '97r22vwap7x9t';

--  Use this report_id to get the historical SQL monitor report
select
     dbms_auto_report.report_repository_detail
     (rid   => 8934
     ,type => 'text')
  from dual;

Global Information
---------------------  
 Status                :  DONE (ALL ROWS)       
 Instance ID           :  1                     
 Session               :  skskslsls (1343:42100) 
 SQL ID                :  97r22vwap7x9t         
 SQL Execution ID      :  16777216              
 Execution Started     :  06/27/2017 11:11:35   
 First Refresh Time    :  06/27/2017 11:11:41 
 Last Refresh Time     :  06/27/2017 12:02:57 
 Duration              :  3082s                    
 Module/Action         :  xxxxxxxxxxxx          
 Service               :  xxxxxxxxx          
 Program               :  xxxxxxxxxxx                  
 Fetch Calls           :  13             

Global Stats
=================================================
| Elapsed |   Cpu   |  Other   | Fetch | Buffer |
| Time(s) | Time(s) | Waits(s) | Calls |  Gets  |
=================================================
|    3083 |    3079 |     3.67 |    13 |  294M  |
=================================================

SQL Plan Monitoring Details (Plan Hash Value=4071256796)
========================================================================
| Id |               Operation |   Name   |  Rows   | Execs |   Rows   |
|    |                         |          | (Estim) |       | (Actual) |
========================================================================
|  0 | SELECT STATEMENT        |          |         |     1 |     1010 |
|  1 |   FILTER                |          |         |     1 |     1010 |
|  2 |    TABLE ACCESS FULL    | TABLE1   |   89457 |     1 |    89457 |
|  3 |    TABLE ACCESS FULL    | TABLE2   |       1 | 89457 |     1010 |
========================================================================

The FILTER at operation n°1 acts exactly as NESTED LOOPS join would do: for each row from table TABLE1 (A-Rows=89,457) looks for rows in table TABLE2 (Starts = 89,457) that satisfy the filter predicate n°3.

This is exactly what happens when a correlated subquery is not unnested: it will be evaluated as many times as there are rows in the outer query block.

       3.2. The 11gR2 effect

The on call DBA told me that this query was performing very well before the upgrade. My immediate reflex was then to run the same query under the previous 14 OFE(Optimizer Features Enable) . Useless to say that it completed very quickly using the following execution plan:

Plan hash value: 3372386476
-------------------------------------------------------------
| Id  | Operation             | Name     | Rows  | Time     |
-------------------------------------------------------------
|   0 | SELECT STATEMENT      |          |       |          |
|*  1 |  HASH JOIN            |          |  2202 | 00:00:01 |
|   2 |   VIEW                | VW_NSO_1 |  2202 | 00:00:01 |
|   3 |    HASH UNIQUE        |          |  2202 | 00:00:01 |
|   4 |     UNION-ALL         |          |       |          |
|*  5 |      TABLE ACCESS FULL| TABLE2   |  1101 | 00:00:01 |
|*  6 |      TABLE ACCESS FULL| TABLE2   |  9890 | 00:00:01 |
|   7 |   TABLE ACCESS FULL   | TABLE1   | 89457 | 00:00:01 |
-------------------------------------------------------------

The presence of the term VW_NSO_1 at operation id n°2 in the previous plan is a sign of Oracle having unnested the two correlated subqueries in order to eliminate the prolific FILTER effect of multi-starting the TABLE2outer full scan operation.

However, the presence of TABLE2 twice in the execution plan tends to suggest that Oracle has not used the 12cR1 technic to coalesce the two correlated subqueries. Indeed in contrast to 12cR1, in 11gR2 Oracle has merged the two subqueries into a single one but without eliminating the two select against TABLE2. In the corresponding 10053 trace file we can read the following lines:

Final query after transformations:******* UNPARSED QUERY IS *******

select
  a.id1 id1,
  a.n1 n1,
  a.start_date start_date
from
  (  (select c.id1 id1 from t2 c where c.status='ERR')
    union
     (select b.id id from t2 b where b.status='COM')
  ) vw_nso_1,
  t1 a
where
  a.id1= vw_nso_1.id1;

-- 11.2.0.4
*****************************
Cost-Based Subquery Unnesting
*****************************
SU: Unnesting query blocks in query block SEL$1 (#1) that are valid to unnest.
Subquery removal for query block SEL$3 (#2)
RSW: Not valid for subquery removal SEL$3 (#2)
Subquery unchanged.
Subquery Unnesting on query block SEL$1 (#1)SU: Performing unnesting that does not require costing.
SU: Considering subquery unnest on query block SEL$1 (#1).
SU:   Checking validity of unnesting subquery SEL$2 (#3)
SU:   Checking validity of unnesting subquery SEL$3 (#2)
../..
SU:   Checking validity of unnesting subquery SET$E74BECDC (#6)
SU:   Passed validity checks.
SU:   Transform an ANY subquery to semi-join or distinct.

With the OFE set to 11.2.0.4 Oracle has transformed the two disjunctive subqueries into a single subquery in which the dijsunction has been replaced by a union operation (instead of union all because a distinct operation has to be applied for the in-list elements). The result of this transformation has been transformed into a inline view before being unnested with the outer block query:

Registered qb: SET$7FD77EFD 0x1c710c88 (SUBQ INTO VIEW FOR COMPLEX UNNEST SET$E74BECDC)

Thanks to the unnesting of the inline view ( VW_NSO_1) a more peformant and suitable hash join (at least for the real life query) has been used instead of the mutli-executed filter operation.

4. Why the unnesting has not been possible in 12cR1?

As always the 10053 trace file reproduced below (reduced to the bare minimum) gives an answer to the posed question:

-- 12.1.0.1
*****************************
Cost-Based Subquery Unnesting
*****************************
SU: Unnesting query blocks in query block SEL$1 (#1) that are valid to unnest.
Subquery Unnesting on query block SEL$1 (#1)SU: Performing unnesting that does not require costing.
SU: Considering subquery unnest on query block SEL$1 (#1).
SU: Checking validity of unnesting subquery SEL$ACD206C8 (#2)
SU: SU bypassed: Invalid correlated predicates.
SU: Validity checks failed.
*******************************

It seems that Oracle has been unable to unnest the two subqueries it has coalesced because they are using two different correlated predicates as shown below:

where
  exists
  (
    select
      0 id
    from
      t2 b
    where
      b.status  = 'COM'
    and a.id1   = b.id  --1st correlated predicate
    or b.status ='ERR'
    and a.id1   = b.id1 -- 2nd correlated predicate
  );

Had those two correlated predicates been identical the unnesting of the coalesced disjunctive correlated subqueries would have been possible as demonstrated below:

explain plan for
select
    a.id1
   ,a.n1
   ,a.start_date
from 
    t1 a
where 
   (a.id1 in
         (select b.id1 –- spot the change here
          from t2 b
          where b.status = 'COM')
   or a.id1 in
         (select c.id1
          from t2 c
          where c.status = 'ERR')
        );

select * from table(dbms_xplan.display(format =>'+outline'));

------------------------------------------------------
| Id  | Operation                    | Name  | Rows  |
------------------------------------------------------
|   0 | SELECT STATEMENT             |       |    10 |
|   1 |  NESTED LOOPS                |       |    10 |
|   2 |   NESTED LOOPS               |       |    10 |
|   3 |    SORT UNIQUE               |       |  9890 |
|*  4 |     TABLE ACCESS FULL        | T2    |  9890 |
|*  5 |    INDEX UNIQUE SCAN         | T1_PK |     1 |
|   6 |   TABLE ACCESS BY INDEX ROWID| T1    |     1 |
------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      SEMI_TO_INNER(@"SEL$82F4A621" "B"@"SEL$2")
      NLJ_BATCHING(@"SEL$82F4A621" "A"@"SEL$1")
      USE_NL(@"SEL$82F4A621" "A"@"SEL$1")
      LEADING(@"SEL$82F4A621" "B"@"SEL$2" "A"@"SEL$1")
      INDEX(@"SEL$82F4A621" "A"@"SEL$1" ("T1"."ID1"))
      FULL(@"SEL$82F4A621" "B"@"SEL$2")
      OUTLINE(@"SEL$2")
      OUTLINE(@"SEL$3")
      COALESCE_SQ(@"SEL$2")
      OUTLINE(@"SEL$7C83FF7B")
      COALESCE_SQ(@"SEL$3")
      OUTLINE(@"SEL$ACD206C8")
      OUTLINE(@"SEL$1")
      UNNEST(@"SEL$ACD206C8")
      OUTLINE_LEAF(@"SEL$82F4A621")
      ALL_ROWS
      DB_VERSION('12.2.0.1')
      OPTIMIZER_FEATURES_ENABLE('12.2.0.1')
      IGNORE_OPTIM_EMBEDDED_HINTS
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------

   4 - filter("B"."STATUS"='COM' OR "B"."STATUS"='ERR')
   5 - access("A"."ID1"="B"."ID1")

Note
-----
   - this is an adaptive plan

5. Summary

The Oracle way of coalescing subqueries seems to be going back and forth between different versions. Two dijsunctive correlated subqueries have been differently coalesced in 11.2.0.4 and 12.1.0.1. While in the former they are classically transformed into a single subquery combining two branches of a union of two tables, in the later release they have been coalesced to form a single subquery accesing a single table. The 12cR1 and 12cR2 way of coalescing two correlated subqueries introduces a new requirement for their unnesting: they need to have the same correlated predicates. Thereby you should watch out carrefully your 12c queries having two (or more) correlated subqueries. If they are using different (invalid:-)) predicates they might be still coalesced but they will not be unnested. If you judge that the unnesting of your subqueries is better than using their coalesced version as a predicate of a FILTER operation then you have at your dispoable the NO_COALESCE_SQ(qb_name)or the _optimizer_coalesce_subqueries hidden parameter to cancel the subqueries coalescing transformation.

6. Bonus

Following this article from Mauro Pagano blog about v$sql_diag_repository view I decided to see what this view shows in this particular case of subquery coalescing and unnesting. Here’s below the demonstration:

SQL> show parameter optimizer_features

NAME                                 TYPE        VALUE
------------------------------------ ----------- --------
optimizer_features_enable            string      12.2.0.1
SQL> alter session set "_sql_diag_repo_retain"=true;
-- execute original query
SQL> alter session set "_optimizer_coalesce_subqueries"=false;
-- execute original query
select
     a.sql_id
    ,a.type
    ,a.feature
    ,a.state
    ,b.reason
  from
     gv$sql_diag_repository a
    ,gv$sql_diag_repository_reason b
where a.sql_id = b.sql_id
and   a.sql_diag_repo_id = b.sql_diag_repo_id
and   a.child_number = b.child_number
and   a.sql_id = 'actbyxwt34x4b'
and  a.feature = 'QKSFM_UNNEST'
and state in ('valid','bypass')
SQL_ID        TYPE        FEATURE         STATE      REASON
------------- ----------- --------------- ---------- -----------------------
actbyxwt34x4b qbcdef      QKSFM_UNNEST    valid      passed validity checks
actbyxwt34x4b qcUnknown   QKSFM_UNNEST    valid      passed validity checks
actbyxwt34x4b qcUnknown   QKSFM_UNNEST    valid      passed validity checks
actbyxwt34x4b qcUnknown   QKSFM_UNNEST    valid      passed validity checks

actbyxwt34x4b qbcdef      QKSFM_UNNEST    bypass     invalidated
actbyxwt34x4b qcUnknown   QKSFM_UNNEST    bypass     invalide correlated predicates
actbyxwt34x4b qcUnknown   QKSFM_UNNEST    bypass     invalidated
actbyxwt34x4b qcUnknown   QKSFM_UNNEST    bypass     invalid correlated predicates

The Model

create table t1
   as select
    rownum                id1,
    trunc((rownum-1/3))   n1,
    date '2012-06-07' + mod((level-1)*2,5) start_date,
    lpad(rownum,10,'0')   small_vc,
    rpad('x',1000)        padding
from dual
connect by level <= 1e4;   
 
create table t2
as select
    rownum id
    ,mod(rownum,5) + mod(rownum,10)* 10  as id1
    ,case
       when mod(rownum, 1000) = 7 then 'ERR'
       when rownum <= 9900 then 'COM'
       when mod(rownum,10) between 1 and 5 then 'PRP'
     else
       'UNK'
     end status
     ,lpad(rownum,10,'0')    as small_vc
     ,rpad('x',70)           as padding
from dual
connect by level <= 1e4;
 
alter table t1 add constraint t1_pk primary key (id1);
Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

Tony's Oracle Tips

Tony Hasler's light hearted approach to learning about Oracle

Richard Foote's Oracle Blog

Focusing Specifically On Oracle Indexes, Database Administration and Some Great Music

Hatem Mahmoud Oracle's blog

Just another Oracle blog : Database topics and techniques

Mohamed Houri’s Oracle Notes

Qui se conçoit bien s’énonce clairement

Oracle Diagnostician

Performance troubleshooting as exact science

Raheel's Blog

Things I have learnt as Oracle DBA

Coskan's Approach to Oracle

What I learned about Oracle

So Many Oracle Manuals, So Little Time

“Books to the ceiling, Books to the sky, My pile of books is a mile high. How I love them! How I need them! I'll have a long beard by the time I read them”—Lobel, Arnold. Whiskers and Rhymes. William Morrow & Co, 1988.

EU Careers info

Your career in the European Union

Carlos Sierra's Tools and Tips

Tools and Tips for Oracle Performance and SQL Tuning

Oracle Scratchpad

Just another Oracle weblog

OraStory

Dominic Brooks on Oracle Performance, Tuning, Data Quality & Sensible Design ... (Now with added Sets Appeal)

%d bloggers like this: