Mohamed Houri’s Oracle Notes

August 12, 2017

Unnesting of coalesced subqueries

Filed under: Oracle — hourim @ 3:49 pm

The 12cR1 way used by Oracle to coalesce two correlated disjunctive subqueries has prevented a real life query from unnesting its two subqueries causing a serious performance issue. This article examines a representative model of this real life query. The first paragraph explains the any-to-exists subquery conversion. The second paragraph shows how two correlated exists subqueries can be coalesced to form a single subquery. Once these two transformations introduced the third paragraph will examine how the previously coalesced subqueries can be unnested. It will particularly show that the ability of Oracle to unnest two coalesced correlated subqueries depends strongly on the coalescing method as this one differs from version to version.

1. Any-to-exists subquery conversion

Before coalescing two (or more) subqueries Oracle starts first by transforming ANY and ALL subqueries into EXISTS and NOT EXISTS respectively. Consider, for example, the following query:

select
    a.id1
   ,a.n1
   ,a.start_date
from 
    t1 a
where 
   (a.id1 in
         (select b.id
          from t2 b
          where b.status = 'COM')
   or a.id1 in
         (select c.id1
          from t2 c
          where c.status = 'ERR')
     );

If you examine its corresponding 10053 CBO trace file you will find that its two disjunctive subqueries have been first converted from an any-to-exists subqueries as shown below:

Query After Exists Conversion:******* UNPARSED QUERY IS *******

select
  a.id1 id1,
  a.n1 n1,
  a.start_date start_date
from
  t1 a
where
  exists
  ( select b.id id
    from t2 b
    where b.status='COM'
    and   a.id1 =b.id)
or exists
  ( select c.id1 id1
    from t2 c
    where c.status='ERR'
    and a.id1 =c.id1)
  );
 

This is generally the preduled to or, a preliminary part of, a subquery coalescing transformation presented in the next section

2. Subquery Coalescing

The preceding any-to-exists transformed query is nothing else than an instance of the following simplified one:

select 
  {list of columns}
from
  table
where 
  predicate

The predicate in this particular case is acting as two disjunctive correlated subqueries of the same type:

-- first subquery
exists
  ( select b.id id
    from t2 b
    where b.status = 'COM'
    and   a.id1    = b.id)

-- second subquery
 exists
  ( select c.id1 id1
    from t2 c
    where c.status ='ERR'
    and a.id1 = c.id1)

If these two subqueries are not coalesced they will probably generate two accesses to the same table t2. This is why Oracle can, under certain conditions, coalesce two conjunctive or disjucntive subqueries into a single one. Applied to the current query the coalescing process yields the following new transformed query:

Query After Ex/Any Disj SQ coalescing:******* UNPARSED QUERY IS *******

select
  a.id1 id1,
  a.n1 n1,
  a.start_date start_date
from
  t1 a
where
  exists
  (
    select
      0 id
    from
      t2 b
    where
      b.status  = 'COM'
    and a.id1   = b.id
    or b.status ='ERR'
    and a.id1   = b.id1
  );

The execution plan of the doubly transformed query is shown below:

alter session set statistics_level=all;
select * from table(dbms_xplan.display_cursor(null,null, 'allstats last'));

--------------------------------------------------------------
| Id  | Operation          | Name | Starts | E-Rows | A-Rows |
--------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |      1 |        |   9890 |
|*  1 |  FILTER            |      |      1 |        |   9890 |
|   2 |   TABLE ACCESS FULL| T1   |      1 |  10000 |  10000 |
|*  3 |   TABLE ACCESS FULL| T2   |  10000 |      1 |   9890 |
--------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter( IS NOT NULL)
   3 - filter((("B"."ID"=:B1 AND "B"."STATUS"='COM') OR ("B"."ID1"=:B2 AND
              "B"."STATUS"='ERR')))

If you still have any doubts about whether the two subqueries have been coalesced or not then the single apparition of table T2 in the plan together with the predicate n°3 will certainly clear out this doubt for good. Indeed, it is the subquery coalescing process that resulted into Oracle visiting table T2 only one time.

In addition in the outline data used to generate the above execution plan reproduced here below:

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('12.2.0.1')
      DB_VERSION('12.2.0.1')
      ALL_ROWS
      OUTLINE_LEAF(@"SEL$ACD206C8")
      COALESCE_SQ(@"SEL$3")
      OUTLINE_LEAF(@"SEL$1")
      OUTLINE(@"SEL$7C83FF7B")
      COALESCE_SQ(@"SEL$2")
      OUTLINE(@"SEL$3")
      OUTLINE(@"SEL$2")
      FULL(@"SEL$1" "A"@"SEL$1")
      PQ_FILTER(@"SEL$1" SERIAL)
      FULL(@"SEL$ACD206C8" "B"@"SEL$2")
      END_OUTLINE_DATA
  */

The COALESCE_SQ(@”SEL$3″) and COALESCE_SQ(@”SEL$2″)represent the hints used to force a subquery coalescing transformation. If you want to cancel such a subquery coalescing transformation then the NO_COALESCE_SQ(qb_name) hint will be the one you should use. You can also change the hidden parameter _optimizer_coalesce_subqueries governing this transformation which defaults to true in 12cR1.

3. Subquery unnesting
        3.1. The 12cR1 effect

The any-to-exist conversion followed by the subquery coalescing should have normally paved the way for subquery unnesting transformation. Unfortunately, the real life query, which the actual model is a true representation of, the coalesced subqueries have not been unnested altering dramatically the performance of the original query as shown via its corresponding sql monitor report:

-- Get the report_id corresponding to the triplet (sql_id,sql_exec_id, sql_exec_start)
 select 
      report_id
    , key1 sql_id
    , key2 sql_exec_id
    , key3 sql_exec_start
  from
     dba_hist_reports
  where
     component_name ='sqlmonitor'
  and key1 = '97r22vwap7x9t';

--  Use this report_id to get the historical SQL monitor report
select
     dbms_auto_report.report_repository_detail
     (rid   => 8934
     ,type => 'text')
  from dual;

Global Information
---------------------  
 Status                :  DONE (ALL ROWS)       
 Instance ID           :  1                     
 Session               :  skskslsls (1343:42100) 
 SQL ID                :  97r22vwap7x9t         
 SQL Execution ID      :  16777216              
 Execution Started     :  06/27/2017 11:11:35   
 First Refresh Time    :  06/27/2017 11:11:41 
 Last Refresh Time     :  06/27/2017 12:02:57 
 Duration              :  3082s                    
 Module/Action         :  xxxxxxxxxxxx          
 Service               :  xxxxxxxxx          
 Program               :  xxxxxxxxxxx                  
 Fetch Calls           :  13             

Global Stats
=================================================
| Elapsed |   Cpu   |  Other   | Fetch | Buffer |
| Time(s) | Time(s) | Waits(s) | Calls |  Gets  |
=================================================
|    3083 |    3079 |     3.67 |    13 |  294M  |
=================================================

SQL Plan Monitoring Details (Plan Hash Value=4071256796)
========================================================================
| Id |               Operation |   Name   |  Rows   | Execs |   Rows   |
|    |                         |          | (Estim) |       | (Actual) |
========================================================================
|  0 | SELECT STATEMENT        |          |         |     1 |     1010 |
|  1 |   FILTER                |          |         |     1 |     1010 |
|  2 |    TABLE ACCESS FULL    | TABLE1   |   89457 |     1 |    89457 |
|  3 |    TABLE ACCESS FULL    | TABLE2   |       1 | 89457 |     1010 |
========================================================================

The FILTER at operation n°1 acts exactly as NESTED LOOPS join would do: for each row from table TABLE1 (A-Rows=89,457) looks for rows in table TABLE2 (Starts = 89,457) that satisfy the filter predicate n°3.

This is exactly what happens when a correlated subquery is not unnested: it will be evaluated as many times as there are rows in the outer query block.

       3.2. The 11gR2 effect

The on call DBA told me that this query was performing very well before the upgrade. My immediate reflex was then to run the same query under the previous 14 OFE(Optimizer Features Enable) . Useless to say that it completed very quickly using the following execution plan:

Plan hash value: 3372386476
-------------------------------------------------------------
| Id  | Operation             | Name     | Rows  | Time     |
-------------------------------------------------------------
|   0 | SELECT STATEMENT      |          |       |          |
|*  1 |  HASH JOIN            |          |  2202 | 00:00:01 |
|   2 |   VIEW                | VW_NSO_1 |  2202 | 00:00:01 |
|   3 |    HASH UNIQUE        |          |  2202 | 00:00:01 |
|   4 |     UNION-ALL         |          |       |          |
|*  5 |      TABLE ACCESS FULL| TABLE2   |  1101 | 00:00:01 |
|*  6 |      TABLE ACCESS FULL| TABLE2   |  9890 | 00:00:01 |
|   7 |   TABLE ACCESS FULL   | TABLE1   | 89457 | 00:00:01 |
-------------------------------------------------------------

The presence of the term VW_NSO_1 at operation id n°2 in the previous plan is a sign of Oracle having unnested the two correlated subqueries in order to eliminate the prolific FILTER effect of multi-starting the TABLE2outer full scan operation.

However, the presence of TABLE2 twice in the execution plan tends to suggest that Oracle has not used the 12cR1 technic to coalesce the two correlated subqueries. Indeed in contrast to 12cR1, in 11gR2 Oracle has merged the two subqueries into a single one but without eliminating the two select against TABLE2. In the corresponding 10053 trace file we can read the following lines:

Final query after transformations:******* UNPARSED QUERY IS *******

select
  a.id1 id1,
  a.n1 n1,
  a.start_date start_date
from
  (  (select c.id1 id1 from t2 c where c.status='ERR')
    union
     (select b.id id from t2 b where b.status='COM')
  ) vw_nso_1,
  t1 a
where
  a.id1= vw_nso_1.id1;

-- 11.2.0.4
*****************************
Cost-Based Subquery Unnesting
*****************************
SU: Unnesting query blocks in query block SEL$1 (#1) that are valid to unnest.
Subquery removal for query block SEL$3 (#2)
RSW: Not valid for subquery removal SEL$3 (#2)
Subquery unchanged.
Subquery Unnesting on query block SEL$1 (#1)SU: Performing unnesting that does not require costing.
SU: Considering subquery unnest on query block SEL$1 (#1).
SU:   Checking validity of unnesting subquery SEL$2 (#3)
SU:   Checking validity of unnesting subquery SEL$3 (#2)
../..
SU:   Checking validity of unnesting subquery SET$E74BECDC (#6)
SU:   Passed validity checks.
SU:   Transform an ANY subquery to semi-join or distinct.

With the OFE set to 11.2.0.4 Oracle has transformed the two disjunctive subqueries into a single subquery in which the dijsunction has been replaced by a union operation (instead of union all because a distinct operation has to be applied for the in-list elements). The result of this transformation has been transformed into a inline view before being unnested with the outer block query:

Registered qb: SET$7FD77EFD 0x1c710c88 (SUBQ INTO VIEW FOR COMPLEX UNNEST SET$E74BECDC)

Thanks to the unnesting of the inline view ( VW_NSO_1) a more peformant and suitable hash join (at least for the real life query) has been used instead of the mutli-executed filter operation.

4. Why the unnesting has not been possible in 12cR1?

As always the 10053 trace file reproduced below (reduced to the bare minimum) gives an answer to the posed question:

-- 12.1.0.1
*****************************
Cost-Based Subquery Unnesting
*****************************
SU: Unnesting query blocks in query block SEL$1 (#1) that are valid to unnest.
Subquery Unnesting on query block SEL$1 (#1)SU: Performing unnesting that does not require costing.
SU: Considering subquery unnest on query block SEL$1 (#1).
SU: Checking validity of unnesting subquery SEL$ACD206C8 (#2)
SU: SU bypassed: Invalid correlated predicates.
SU: Validity checks failed.
*******************************

It seems that Oracle has been unable to unnest the two subqueries it has coalesced because they are using two different correlated predicates as shown below:

where
  exists
  (
    select
      0 id
    from
      t2 b
    where
      b.status  = 'COM'
    and a.id1   = b.id  --1st correlated predicate
    or b.status ='ERR'
    and a.id1   = b.id1 -- 2nd correlated predicate
  );

Had those two correlated predicates been identical the unnesting of the coalesced disjunctive correlated subqueries would have been possible as demonstrated below:

explain plan for
select
    a.id1
   ,a.n1
   ,a.start_date
from 
    t1 a
where 
   (a.id1 in
         (select b.id1 –- spot the change here
          from t2 b
          where b.status = 'COM')
   or a.id1 in
         (select c.id1
          from t2 c
          where c.status = 'ERR')
        );

select * from table(dbms_xplan.display(format =>'+outline'));

------------------------------------------------------
| Id  | Operation                    | Name  | Rows  |
------------------------------------------------------
|   0 | SELECT STATEMENT             |       |    10 |
|   1 |  NESTED LOOPS                |       |    10 |
|   2 |   NESTED LOOPS               |       |    10 |
|   3 |    SORT UNIQUE               |       |  9890 |
|*  4 |     TABLE ACCESS FULL        | T2    |  9890 |
|*  5 |    INDEX UNIQUE SCAN         | T1_PK |     1 |
|   6 |   TABLE ACCESS BY INDEX ROWID| T1    |     1 |
------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      SEMI_TO_INNER(@"SEL$82F4A621" "B"@"SEL$2")
      NLJ_BATCHING(@"SEL$82F4A621" "A"@"SEL$1")
      USE_NL(@"SEL$82F4A621" "A"@"SEL$1")
      LEADING(@"SEL$82F4A621" "B"@"SEL$2" "A"@"SEL$1")
      INDEX(@"SEL$82F4A621" "A"@"SEL$1" ("T1"."ID1"))
      FULL(@"SEL$82F4A621" "B"@"SEL$2")
      OUTLINE(@"SEL$2")
      OUTLINE(@"SEL$3")
      COALESCE_SQ(@"SEL$2")
      OUTLINE(@"SEL$7C83FF7B")
      COALESCE_SQ(@"SEL$3")
      OUTLINE(@"SEL$ACD206C8")
      OUTLINE(@"SEL$1")
      UNNEST(@"SEL$ACD206C8")
      OUTLINE_LEAF(@"SEL$82F4A621")
      ALL_ROWS
      DB_VERSION('12.2.0.1')
      OPTIMIZER_FEATURES_ENABLE('12.2.0.1')
      IGNORE_OPTIM_EMBEDDED_HINTS
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------

   4 - filter("B"."STATUS"='COM' OR "B"."STATUS"='ERR')
   5 - access("A"."ID1"="B"."ID1")

Note
-----
   - this is an adaptive plan

5. Summary

The Oracle way of coalescing subqueries seems to be going back and forth between different versions. Two dijsunctive correlated subqueries have been differently coalesced in 11.2.0.4 and 12.1.0.1. While in the former they are classically transformed into a single subquery combining two branches of a union of two tables, in the later release they have been coalesced to form a single subquery accesing a single table. The 12cR1 and 12cR2 way of coalescing two correlated subqueries introduces a new requirement for their unnesting: they need to have the same correlated predicates. Thereby you should watch out carrefully your 12c queries having two (or more) correlated subqueries. If they are using different (invalid:-)) predicates they might be still coalesced but they will not be unnested. If you judge that the unnesting of your subqueries is better than using their coalesced version as a predicate of a FILTER operation then you have at your dispoable the NO_COALESCE_SQ(qb_name)or the _optimizer_coalesce_subqueries hidden parameter to cancel the subqueries coalescing transformation.

6. Bonus (more…)

August 5, 2017

Parallel index rebuild: Oracle versus MS-SQL Server

Filed under: Oracle — hourim @ 4:57 pm

A couple of months ago I have been prompted by this MS-SQL Server article about rebuilding indexes in parallel. The bottom line of this article is that MS-SQL Server engine will not use parallel run when rebuilding a unique index starting with a column having a single distinct value. This article will first demonstrate this claim using MS-SQL Server 2016 and then extend the same demonstration to Oracle in its latest version.

1. SQL Server

The followings will create a table and its unique three columns composite clustered index

select @@version
Microsoft SQL Server 2016 (RTM-GDR) (KB3210111) - 13.0.1728.2 (X64)  
Dec 13 2016 04:40:28   Copyright (c) Microsoft Corporation 
Developer Edition (64-bit) on Windows 8.1 6.3 <X64> (Build 9600: ) 

-- create table
create table t (c1 int, c2 int, c3 int);

set nocount on
begin tran
declare @i int = 0
while @i < 10000000
begin
    insert into t values (@i, @i, @i)
    set @i = @i + 1
end
commit tran

-- create a unique clustered index
create unique clustered index ix on t(c1,c2,c3)

Now that we are done with the model we are going to rebuild the above clustered index using parallel run

-- This will use parallel plan
ALTER INDEX ix on [dbo].t rebuild WITH (maxdop=10, online=on)

Before rebuilding the index I have activated the ”Include Actual Execution Plan” icon of SSMS (SQL Server Management Studio) so that I got the following “row source” plan where the two yellow arrows indicate that the rebuild has been, indeed, parallelized:

However has the first column c1 of the index been single valued the index would have not been rebuild parallely by the SQL Server engine as the followings demonstrate:

-- set the leading column of the clustered index to a single value
 update t set c1 = 1

-- update statistics with 100% sample
update statistics t with fullscan

–- This will use serial plan
ALTER INDEX ix on [dbo].t rebuild WITH (maxdop=10, online=on)

The execution plan of the second index rebuild show that this time MS-SQL refuses to obey the parallel hint and decides to go serially as shown below:

The MS-SQL documentation stipulates that when the leading key of the index of a non-partitioned table has a single distinct value it becomes useless to give a parallel thread a range of single value to process them. In other words the leading column should have multiple distinct values in order for parallelism to be worth a utilization.

Now that we have set the scene for MS-SQL let’s reproduce it for Oracle

2. Oracle

SQL> select banner from v$version where rownum=1;

BANNER
-----------------------------------------------------------------------------
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> create table t (c1 int, c2 int, c3 int);

SQL> begin
       for j in 1..10000000
     loop
        insert /*+ append */ into t values (j, j, j);
     end loop;
     commit;
     end;
     /

SQL> create unique index ix on t(c1,c2,c3);

SQL> exec dbms_stats.gather_table_stats(user, 't');

SQL> select 
         column_name
         ,num_distinct
         ,histogram
    from 
        all_tab_col_statistics
    where table_name = 'T'
    order by 2;

COLUMN_NAM NUM_DISTINCT HISTOGRAM
---------- ------------ ---------------
C1              9914368 NONE
C3              9914368 NONE
C2              9914368 NONE

It’s now time to use parallelism to rebuild the above index

SQL> alter index ix rebuild parallel 4;

select
    dfo_number, tq_id, server_type, instance, process, num_rows
from
   v$pq_tqstat
order by
    dfo_number, tq_id, server_type, instance, process;

DFO_NUMBER      TQ_ID SERVER_TYPE    INSTANCE PROCESS    NUM_ROWS
---------- ---------- ------------ ---------- -------- ----------
         1          0 Consumer              1 P000        2923571
         1          0 Consumer              1 P001        2623371
         1          0 Consumer              1 P002        2270919
         1          0 Consumer              1 P003        2182139
         1          0 Producer              1 P004        2638901
         1          0 Producer              1 P005        2637316
         1          0 Producer              1 P006        2611722
         1          0 Producer              1 P007        2114251
         1          0 Ranger                1 QC               12
         1          1 Consumer              1 QC                4
         1          1 Producer              1 P000              1
         1          1 Producer              1 P001              1
         1          1 Producer              1 P002              1
         1          1 Producer              1 P003              1


Parallel Execution Details (DOP=4 , Servers Allocated=8)
===================================================================
|      Name      | Type  | Server# | Elapsed |   Cpu   |    IO    |
|                |       |         | Time(s) | Time(s) | Waits(s) |
===================================================================
| PX Coordinator | QC    |         |    0.10 |    0.02 |     0.01 |
| p000           | Set 1 |       1 |    5.53 |    4.12 |     0.02 |
| p001           | Set 1 |       2 |    5.48 |    3.60 |     0.01 |
| p002           | Set 1 |       3 |    4.87 |    3.20 |     0.03 |
| p003           | Set 1 |       4 |    4.97 |    3.03 |     0.01 |
| p004           | Set 2 |       1 |    1.05 |    1.05 |          |
| p005           | Set 2 |       2 |    1.01 |    0.83 |          |
| p006           | Set 2 |       3 |    1.04 |    1.03 |          |
| p007           | Set 2 |       4 |    0.99 |    0.84 |          |
===================================================================

SQL Plan Monitoring Details (Plan Hash Value=3277881472)
===========================================================================
| Id |         Operation          |   Name   |  Rows   | Cost |   Time    |
|    |                            |          | (Estim) |      | Active(s) |
===========================================================================
|  0 | CREATE INDEX STATEMENT     |          |         |      |         1 |
|  1 |   PX COORDINATOR           |          |         |      |         1 |
|  2 |    PX SEND QC (ORDER)      | :TQ10001 |      82 |      |         1 |
|  3 |     INDEX BUILD UNIQUE     | IX       |         |      |         4 |
|  4 |      SORT CREATE INDEX     |          |      82 |      |         6 |
|  5 |       PX RECEIVE           |          |      82 |    2 |         4 |
|  6 |        PX SEND RANGE       | :TQ10000 |      82 |    2 |         2 |
|  7 |         PX BLOCK ITERATOR  |          |      82 |    2 |         1 |
|  8 |          TABLE ACCESS FULL | T        |      82 |    2 |         1 |
===========================================================================

As expected Oracle, in accordance with MS-SQL Server, has obeyed the instruction of rebuilding the index in parallel.

Let’s now unify the distinct values of the leading column c1 of the index and then try rebuilding it parallely:

 SQL> update t set c1 = 1;

SQL> exec dbms_stats.gather_table_stats(user, 't');

SQL> select 
         column_name
         ,num_distinct
         ,histogram
      from 
        all_tab_col_statistics
      where table_name = 'T'
      order by 2;

COLUMN_NAM NUM_DISTINCT HISTOGRAM
---------- ------------ ----------
C1                    1 NONE
C3              9914368 NONE
C2              9914368 NONE

SQL> alter index ix rebuild parallel 4;

select
    dfo_number, tq_id, server_type, instance, process, num_rows
from
   v$pq_tqstat
order by
    dfo_number, tq_id, server_type, instance, process;
DFO_NUMBER      TQ_ID SERVER_TYPE     INSTANCE PROCESS     NUM_ROWS
---------- ---------- ------------- ---------- --------- ----------
         1          0 Consumer               1 P000         2923571
         1          0 Consumer               1 P001         2623371
         1          0 Consumer               1 P002         2270919
         1          0 Consumer               1 P003         2182139
         1          0 Producer               1 P004         2694213
         1          0 Producer               1 P005         2542784
         1          0 Producer               1 P006         2814202
         1          0 Producer               1 P007         1950991
         1          0 Ranger                 1 QC                12
         1          1 Consumer               1 QC                 4
         1          1 Producer               1 P000               1
         1          1 Producer               1 P001               1
         1          1 Producer               1 P002               1
         1          1 Producer               1 P003               1

Parallel Execution Details (DOP=4 , Servers Allocated=8)
===================================================================
|      Name      | Type  | Server# | Elapsed |   Cpu   |    IO    |
|                |       |         | Time(s) | Time(s) | Waits(s) |
===================================================================
| PX Coordinator | QC    |         |    0.78 |    0.02 |     0.01 |
| p000           | Set 1 |       1 |    5.16 |    3.95 |     0.01 |
| p001           | Set 1 |       2 |    5.10 |    3.56 |     0.01 |
| p002           | Set 1 |       3 |    4.09 |    3.15 |     0.01 |
| p003           | Set 1 |       4 |    4.70 |    3.23 |     0.00 |
| p004           | Set 2 |       1 |    1.38 |    1.08 |     0.02 |
| p005           | Set 2 |       2 |    1.26 |    1.23 |     0.03 |
| p006           | Set 2 |       3 |    1.32 |    1.12 |     0.03 |
| p007           | Set 2 |       4 |    1.31 |    0.99 |     0.03 |
===================================================================

SQL Plan Monitoring Details (Plan Hash Value=3277881472)
===========================================================================
| Id |         Operation          |   Name   |  Rows   | Cost |   Time    |
|    |                            |          | (Estim) |      | Active(s) |
===========================================================================
|  0 | CREATE INDEX STATEMENT     |          |         |      |         2 |
|  1 |   PX COORDINATOR           |          |         |      |         7 |
|  2 |    PX SEND QC (ORDER)      | :TQ10001 |     10M |      |         1 |
|  3 |     INDEX BUILD UNIQUE     | IX       |         |      |         4 |
|  4 |      SORT CREATE INDEX     |          |     10M |      |         5 |
|  5 |       PX RECEIVE           |          |     10M | 2397 |         3 |
|  6 |        PX SEND RANGE       | :TQ10000 |     10M | 2397 |         2 |
|  7 |         PX BLOCK ITERATOR  |          |     10M | 2397 |         1 |
|  8 |          TABLE ACCESS FULL | T        |     10M | 2397 |         1 |
===========================================================================		

In contrast to MS-SQL server which refuses to use parallelism when rebuilding a unique clustered index starting with a column having a unique distinct value, Oracle has successfully rebuild the same kind of unique index in approximatively the same execution time.

3. SUMMARY

We’ve considered two instances of the same unique composite index:

  1. A unique composite index starting with a column having a unique distinct value
  2. A unique composite index starting with a more selective column

It is interesting to see that Oracle, in contrast to MS-SQL Server, is in the capacity of using parallelism when rebuilding both type of indexes. In addition Oracle rebuild execution time of the second type of indexes is as good as that of the first type.

June 24, 2017

ORA-54032 : column to be renamed is used in a virtual column expression

Filed under: Oracle,virtual column — hourim @ 3:45 pm

This is a simple note re-explaining what this 12cR1 ORA-54032 error is as it kicked in again a couple of days ago at one of my customer sites and showing that it has definitely been solved in 12cR2.

Here’s a how to reproduce it at will:

12cR1

SQL> select banner from v$version where rownum=1;

BANNER
----------------------------------------------------------------------------
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production

SQL> create table t1(x number, y number, z number);

Table created.

SQL> alter table t1 rename column x to x_bis;

Table altered.

As you can see we renamed column x to x_bis without any issue. However if we create a column group extension out of (x_bis, y) columns we will not be able to rename neither x_bis nor y column as shown below:

SQL> SELECT
          dbms_stats.create_extended_stats
           (ownname   => user
           ,tabname   => 't1'
          ,extension => '(x_bis,y)'
           ) ext
    FROM dual;

-------------------------------
SYS_STU3L40_B_DWBMIA8PMJRGR$S0

SQL> alter table t1 rename column x_bis to x;
alter table t1 rename column x_bis to x
                             *
ERROR at line 1:
ORA-54032: column to be renamed is used in a virtual column expression

That’s exactly what happened to my client with the sole difference that the client has never explicitly created any virtual column hence its great stupefaction. I had to explain him that this is because Oracle has created, behind the scenes, an extended column group, which is nothing else than a virtual column. This creation is very probably a response to a SQL Plan Directive request.

And indeed, the column group extension I have manually created above is a virtual column created out of a combination of x_bis and y column as clearly shown by the default value of this virtual column:

SQL> select 
         column_name
        ,data_default
        ,data_length
     from
       user_tab_cols
     where
       table_name = 'T1'
     and hidden_column = 'YES';

COLUMN_NAME                    DATA_DEFAULT                      DATA_LENGTH
------------------------------ --------------------------------- -----------
SYS_STU3L40_B_DWBMIA8PMJRGR$S0 SYS_OP_COMBINED_HASH("X_BIS","Y")   22

We are unable to rename the x_bis column to x because x_bis is used in the definition of the above virtual column. But that was before the arrival of 12cR2.

12cR2

Execute the same experiment in 12cR2 and you will realize that things have changed as the followings demonstrate:

SQL> select banner from v$version where rownum=1;
BANNER
--------------------------------------------------------------------------------
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> create table t1(x number, y number, z number);

Table T1 created.

SQL> alter table t1 rename column x to x_bis;

Table T1 altered.

  SELECT
      dbms_stats.create_extended_stats
       (ownname   => user
       ,tabname   => 't1'
       ,extension => '(x_bis,y)'
       ) ext
  FROM dual;

EXT
-------------------------------
SYS_STU3L40_B_DWBMIA8PMJRGR$S0

SQL> alter table t1 rename column x_bis to x;

Table T1 altered.

As of 12cR2 we are not anymore annoyed by the error when altering a table column used in a virtual column expression. This is because Oracle will take the alter column from the table level and extend it to the virtual column where it can be found as shown below:

SQL> select 
         column_name
        ,data_default
        ,data_length
     from
       user_tab_cols
     where
       table_name = 'T1'
     and hidden_column = 'YES';

COLUMN_NAME                    DATA_DEFAULT                   DATA_LENGTH
------------------------------ ------------------------------ -----------
SYS_STU3L40_B_DWBMIA8PMJRGR$S0 SYS_OP_COMBINED_HASH("X","Y")           22

Spot how, in the data_default value of the generated column group extension, the new modified name of the column has been changed as well.

SUMMARY

Again you know that an automatic extended column group extension, which might be created on behalf of you, generates a border corner where you are not anymore able to rename any of the columns that composed the extend virtual column group. Fortunately as of Oracle 12cR2 Oracle extends the column renaming directly to the extended virtual column and doesn’t raise the ORA-054023 anymore.

June 17, 2017

12cR2 OR-Expansion

Filed under: CBO — hourim @ 7:47 am

In contrast to what the title seems suggesting, the OR-Expansion is not a new 12cR2 transformation. It has existed before and has been nicely explained in this article by the Oracle optimizer group. However, let me invite you to observe with me the following simple demonstration:

create table t1 (n1 number, n2 number, c1 varchar2(10));
create index t1_idx1 on t1(n1, n2);
create index t1_idx2 on t1(n2);

explain plan for
select * 
from t1
where (n1 =1 or n2 = 42);

select * from table(dbms_xplan.display);

The above query has two different execution plans in 12cR1 and 12cR2 as shown below respectively:

-- 12cR1 : 12.1.0.2.0
---------------------------------------------------------------
| Id  | Operation                            | Name    | Rows  | 
----------------------------------------------------------------
|   0 | SELECT STATEMENT                     |         |     2 |
|   1 |  CONCATENATION                       |         |       | 
|   2 |   TABLE ACCESS BY INDEX ROWID BATCHED| T1      |     1 |
|*  3 |    INDEX RANGE SCAN                  | T1_IDX2 |     1 |
|   4 |   TABLE ACCESS BY INDEX ROWID BATCHED| T1      |     1 |
|*  5 |    INDEX RANGE SCAN                  | T1_IDX1 |     1 | 
----------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("N2"=42)
   5 - access("N1"=1)
       filter(LNNVL("N2"=42))
-- 12cR2 : 12.2.0.1.0
------------------------------------------------------------------------
| Id  | Operation                             | Name            | Rows  | 
-------------------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                 |     2 | 
|   1 |  VIEW                                 | VW_ORE_BA8ECEFB |     2 |   
|   2 |   UNION-ALL                           |                 |       |   
|   3 |    TABLE ACCESS BY INDEX ROWID BATCHED| T1              |     1 |   
|*  4 |     INDEX RANGE SCAN                  | T1_IDX1         |     1 | 
|*  5 |    TABLE ACCESS BY INDEX ROWID BATCHED| T1              |     1 | 
|*  6 |     INDEX RANGE SCAN                  | T1_IDX2         |     1 | 
-------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   4 - access("N1"=1)
   5 - filter(LNNVL("N1"=1))
   6 – access("N2"=42)

As you can see starting from Oracle 12cR2 Oracle has enhanced the OR expansion from a simple concatenation operator to a slightly more sophisticated union-all operation. The new OR-Expansion has by now been given a new name recognizable in the execution plan via the word: VW_ORE_###.

But what performance added value this transformation is supposed to bring?

Hopefully one of my real life queries gave me the answer to the above question: in contrast to its old implementation, the new enhanced 12cR2 OR-Expansion can be combined with other transformations opening, as such, a new path to the Oracle optimizer that is simply impossible to happen with the pre-12cR2 OR-Expansion.

A picture being worth a thousand of words here’s below a simple demonstration of how such a double transformation can be combined (the model comes from the previous article):

--12cR1
explain plan for
select 
   t1.*
  ,t2.*
from
   t1
left outer join
   t2
on
  (t1.id1 = t2.product_t1
  and
     (t2.start_date <= date'2012-06-12' and t2.end_date >= date'2012-06-07'
        or (t2.id1 > 100)
     )
  );
-------------------------------------------------------
| Id  | Operation           | Name            | Rows  |
-------------------------------------------------------
|   0 | SELECT STATEMENT    |                 | 14949 |
|*  1 |  HASH JOIN OUTER    |                 | 14949 | 
|   2 |   TABLE ACCESS FULL | T1              | 10000 |
|   3 |   VIEW              | VW_DCL_C83A7ED5 |  9950 |
|*  4 |    TABLE ACCESS FULL| T2              |  9950 |
-------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("T1"."ID1"="ITEM_2"(+))
   4 - filter("T2"."ID1">100 OR "T2"."START_DATE"<=TO_DATE(' 2012-06-12 
    00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "T2"."END_DATE">=TO_DATE(' 2012-06-07 
              00:00:00', 'syyyy-mm-dd hh24:mi:ss'))

If you have read my previous article you will recognize in the above execution plan the Decorrelated Lateral view transformation via its name at line 3 (VW_DCL_C83A7ED5). At my client site this transformation was systematically triggering a full table scan so that I was obliged to cancel it as explained in the corresponding article. I would have instead loved if Oracle had gone a step forward and or-expanded this VW_DCL_C83A7ED5 lateral view in order to eliminate the OR disjunctive predicate to open the index access path possibility. Unfortunately combining these two transformation seems to be impossible by then.

That was until the arrival of the 12cR2 and its new enhanced OR-Expansion.

-- 12cR2 : 12.2.0.1.0
explain plan for
select /*+ OR_EXPAND(@"SEL$6226B99A" (1) (2)) */
   t1.*
  ,t2.*
from
   t1
left outer join
   t2
on
  (t1.id1 = t2.product_t1
  and
     (t2.start_date <= date'2012-06-12' and t2.end_date >= date'2012-06-07'
        or (t2.id1 > 100)
     )
  );
---------------------------------------------------------
| Id  | Operation             | Name            | Rows  | 
---------------------------------------------------------
|   0 | SELECT STATEMENT      |                 | 14950 |
|*  1 |  HASH JOIN OUTER      |                 | 14950 | 
|   2 |   TABLE ACCESS FULL   | T1              | 10000 | 
|   3 |   VIEW                | VW_DCL_C83A7ED5 |  9951 |
|   4 |    VIEW               | VW_ORE_37EAC9F1 |  9951 | 
|   5 |     UNION-ALL         |                 |       | 
|*  6 |      TABLE ACCESS FULL| T2              |  9901 | 
|*  7 |      TABLE ACCESS FULL| T2              |    50 | 
---------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 1 - access("T1"."ID1"="ITEM_2"(+))
 6 - filter("T2"."ID1">100)
 7 - filter("T2"."START_DATE"<=TO_DATE(' 2012-06-12 00:00:00', 'syyyy-mm-dd 
    hh24:mi:ss') AND "T2"."END_DATE">=TO_DATE(' 2012-06-07 00:00:00', 'syyyy-mm-dd 
              hh24:mi:ss') AND LNNVL("T2"."ID1">100))

Spot now how the 12cR2 Or-Expansion(VW_ORE_37EAC9F1) has been combined with the 12cR1 Decorrelated Lateral view(VW_DCL_C83A7ED5) opening, as such, the possibility to visit t2 table via an index access path since the disjunctive or predicate has been removed from the refactored query (the index has not been used in my case but that’s only a question of cost).

Since the 12cR2 Or-Expansion is a cost based transformation (see below in the 10053 trace file) it happens that Oracle has decided to do not use it. This is why I used the hint OR_EXPAND in order to force it kicking in for the pedagogic purpose of this article.

In order to produce the above execution plan, Oracle has gone through the following SQL refactoring steps:

 --first it has OR-Expanded the t2 table query block and named it VW_ORE_37EAC9F1

(select 
   t2.id1        item_1,
   t2.product_t1 item_2,
   t2.start_date item_3,
   t2.end_date   item_4,
   t2.padding    item_5
from
   t2 t2
where t2.id1 >100
union all
select 
   t2.id1        item_1,
   t2.product_t1 item_2,
   t2.start_date item_3,
   t2.end_date   item_4,
   t2.padding    item_5
from
   t2 t2
where 
    t2.start_date <=to_date('2012-06-12 00:00:00', 'syyyy-mm-dd hh24:mi:ss')
and t2.end_date   >=to_date('2012-06-07 00:00:00', 'syyyy-mm-dd hh24:mi:ss')
and lnnvl(t2.id1 >100)  VW_ORE_37EAC9F1
 --it have then Decorrelated the OR-Expanded query block and named it VW_DCL_C83A7ED5 

((select 
   t2.id1        item_1,
   t2.product_t1 item_2,
   t2.start_date item_3,
   t2.end_date   item_4,
   t2.padding    item_5
from
   t2 t2
where t2.id1 >100
union all
select 
   t2.id1        item_1,
   t2.product_t1 item_2,
   t2.start_date item_3,
   t2.end_date   item_4,
   t2.padding    item_5
from
   t2 t2
where 
    t2.start_date <=to_date('2012-06-12 00:00:00', 'syyyy-mm-dd hh24:mi:ss')
and t2.end_date   >=to_date('2012-06-07 00:00:00', 'syyyy-mm-dd hh24:mi:ss')
and lnnvl(t2.id1 >100)
) VW_ORE_37EAC9F1
) VW_DCL_C83A7ED5

--and finally it joined the Decorrelated OR-Expanded t2 table query with t1 table
select
  t1.id1   id1,
  t1.flag1 flag1,
  t1.flag2 flag2,
  t1.n1    n1,
  t1.v1    v1,
  vw_dcl_c83a7Ed5.item_1_0 id1,
  vw_dcl_c83a7Ed5.item_2_1 product_t1,
  vw_dcl_c83a7Ed5.item_3_2 start_date,
  vw_dcl_c83a7Ed5.item_4_3 end_date,
  vw_dcl_c83a7Ed5.item_5_4 padding
from t1 t1,
     (select
        vw_ore_37eac9f1.item_1 item_1_0,
        vw_ore_37eac9f1.item_2 item_2_1,
        vw_ore_37eac9f1.item_3 item_3_2,
        vw_ore_37eac9f1.item_4 item_4_3,
        vw_ore_37eac9f1.item_4 item_5_4
      from
        (select 
           t2.id1        item_1,
           t2.product_t1 item_2,
           t2.start_date item_3,
           t2.end_date   item_4,
           t2.padding    item_5
         from
            t2 t2
         where t2.id1 >100
        union all
         select 
           t2.id1        item_1,
           t2.product_t1 item_2,
           t2.start_date item_3,
           t2.end_date   item_4,
           t2.padding    item_5
         from
           t2 t2
         where 
           t2.start_date <=to_date('2012-06-12 00:00:00', 'syyyy-mm-dd hh24:mi:ss')
       and t2.end_date   >=to_date('2012-06-07 00:00:00', 'syyyy-mm-dd hh24:mi:ss')
       and lnnvl(t2.id1 >100)
      )  vw_ore_37eac9f1
     ) vw_dcl_c83a7Ed5
where t1.id1 = vw_dcl_c83a7ed5.item_2_1(+);

The OR-Expansion appears first at the Legend section of the CBO 10053 trace file:

Legend
 The following abbreviations are used by optimizer trace
 CBQT  – cost-based query transformation
 ORE   - CBQT OR-Expansion
 

It can be disabled locally using the hint NO_OR_EXPAND:

explain plan for
select /*+ NO_OR_EXPAND(@"SEL$6226B99A" (1) (2)) */
   t1.*
  ,t2.*
from
   t1
left outer join
   t2
on
  (t1.id1 = t2.product_t1
  and
     (t2.start_date <= date'2012-06-12' and t2.end_date >= date'2012-06-07'
        or (t2.id1 > 100)
     )
  );

-------------------------------------------------------
| Id  | Operation           | Name            | Rows  |
-------------------------------------------------------
|   0 | SELECT STATEMENT    |                 | 14999 | 
|*  1 |  HASH JOIN OUTER    |                 | 14999 | 
|   2 |   TABLE ACCESS FULL | T1              | 10000 |  
|   3 |   VIEW              | VW_DCL_C83A7ED5 | 10000 | 
|*  4 |    TABLE ACCESS FULL| T2              | 10000 | 
-------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("T1"."ID1"="ITEM_2"(+))
   4 - filter("T2"."ID1">100 OR "T2"."START_DATE"<=TO_DATE(' 2012-06-12 
    00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "T2"."END_DATE">=TO_DATE(' 2012-06-07 
              00:00:00', 'syyyy-mm-dd hh24:mi:ss'))

And globally via the hidden param _optimizer_cbqt_or_expansion which defaults to on:

SQL> alter system set "_optimizer_cbqt_or_expansion"= off;

explain plan for
select /*+ OR_EXPAND(@"SEL$6226B99A" (1) (2)) */
   t1.*
  ,t2.*
from
   t1
left outer join
   t2
on
  (t1.id1 = t2.product_t1
  and
     (t2.start_date <= date'2012-06-12' and t2.end_date >= date'2012-06-07'
        or (t2.id1 > 100)
     )
  );
  
  select * from table(dbms_xplan.display);
 --------------------------------------------------------
| Id  | Operation             | Name            | Rows  | 
---------------------------------------------------------
|   0 | SELECT STATEMENT      |                 | 14999 |
|*  1 |  HASH JOIN RIGHT OUTER|                 | 14999 |
|   2 |   VIEW                | VW_DCL_C83A7ED5 | 10000 |
|*  3 |    TABLE ACCESS FULL  | T2              | 10000 |
|   4 |   TABLE ACCESS FULL   | T1              | 10000 |
---------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("T1"."ID1"="ITEM_2"(+))
   3 - filter("T2"."ID1">100 OR "T2"."START_DATE"<=TO_DATE(' 2012-06-12 
    00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "T2"."END_DATE">=TO_DATE(' 2012-06-07 
              00:00:00', 'syyyy-mm-dd hh24:mi:ss'))

SUMMARY

Starting from Oracle 12cR2 the OR-Expansion has been enhanced from the classical concatenation to a more flexible union-all operation. This article shows that one of the advantages brought by this enhancement resides in the possibility it offers to the CBO to combine the new OR-Expansion with other transformations like with the Decorrelated Lateral view. This might increase the chance to have optimal execution plan in queries using disjunctive predicates.

June 14, 2017

Optimiser non sharing reasons

Filed under: CBO,cursor sharing — hourim @ 4:28 pm

Whenever Oracle is prevented from sharing an execution plan of an already seen parent cursor, it hard parses a new plan and externalizes the corresponding non-sharing reason in the gv$sql_shared_cursor dedicated view. As of Oracle 12cR2 there are as many as 66 non-sharing reasons of which I have already explained 10 in the following series of articles I wrote for Toad World:

June 2, 2017

_optimizer_ansi_join_lateral_enhance

Filed under: CBO — hourim @ 1:51 pm

There was a question raised at my client site shorter after I troubleshooted a performance issue due to the new 12cR1 Decorrelated Lateral View transformation: Is there a way to disable this CBO transformation without sending back the optimizer to its previous version?”

It ‘s an interesting question which has triggered a brief discussion and this article.

My short answer was: I have to test and check

And the long answer is:

10053 to the rescue

As long as I was able to engineer a reproducible case where this CBO transformation is used, I decided to generate its corresponding 10053 trace file and look for the word “lateral” in this file. I found 14 hits of this word of which one seems to correlate very well with my aim:

 *************************************
  PARAMETERS WITH DEFAULT VALUES
  ******************************
Compilation Environment Dump
optimizer_mode_hinted               = false
optimizer_features_hinted           = 0.0.0
parallel_execution_enabled          = true
parallel_query_forced_dop           = 0
parallel_dml_forced_dop             = 0
parallel_ddl_forced_degree          = 0
parallel_ddl_forced_instances       = 0
_query_rewrite_fudge                = 90
optimizer_features_enable           = 12.1.0.1
_optimizer_ansi_join_lateral_enhance = true --> this one

Although there is nothing in the name of the above parameter suggesting a decorrelation of a lateral view, it is nevertheless, the unique CBO parameter whith a name including the word lateral. This suggests that setting it off will cancel this transformation. Let’s then implement this transformation and try cancelling it with setting this parameter off.

SQL> select banner from v$version where rownum =1;

BANNER
-----------------------------------------------------------------------------
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production

create table t1
as
 with generator as (
   select --+ materialize
      rownum id
   from dual
   connect by level <= 1000
)
select
    rownum id1,
    mod(rownum-1,2) flag1,
    mod(rownum-1,3) flag2,
    rownum          n1,
    lpad(rownum,30) v1
from
   generator v1,
   generator v2
where
   rownum <= 1e4;
    
alter table t1 add constraint t1_pk primary key (id1);
 
create table t2
as
 select
    level id1
   ,trunc((rownum+2)/2) product_t1
   ,date '2012-06-07' + mod((level-1)*5,10) + interval '5' minute start_date
   ,date '2012-06-08' + mod((level-1)*5,10) + interval '5' minute end_date
   ,rpad('xx',10) padding
from
   dual
connect by level <=1e4;
 
alter table t2 add constraint t2_pk primary key (id1);
alter table t2 add constraint t2_t1_fk foreign key (product_t1) references t1(id1);
 
-- creating an index covering the FK deadlock threat 
create index idx_t2_usr_1 on t2(product_t1, start_date);

explain plan for 
select 
   t1.*
  ,t2.*
from
   t1
left outer join
   t2
on
  (t1.id1 = t2.product_t1
  and
     (t2.start_date <= :1 and t2.end_date >= :2
        or (t2.id1 > 100)
     )
  );
select * from table(dbms_xplan.display(format=>'advanced'));

-------------------------------------------------------
| Id  | Operation           | Name            | Rows  |
-------------------------------------------------------
|   0 | SELECT STATEMENT    |                 | 14925 |
|*  1 |  HASH JOIN OUTER    |                 | 14925 |
|   2 |   TABLE ACCESS FULL | T1              | 10000 |
|   3 |   VIEW              | VW_DCL_1B0973D4 |  9926 |
|*  4 |    TABLE ACCESS FULL| T2              |  9926 |
-------------------------------------------------------

Query Block Name / Object Alias (identified by operation id):
-------------------------------------------------------------
   1 - SEL$A7519627
   2 - SEL$A7519627 / T1@SEL$1
   3 - SEL$6226B99A / VW_LAT_AE9E49E8@SEL$AE9E49E8
   4 - SEL$6226B99A / T2@SEL$1


Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      FULL(@"SEL$6226B99A" "T2"@"SEL$1")
      USE_HASH(@"SEL$A7519627" "VW_LAT_AE9E49E8"@"SEL$AE9E49E8")
      LEADING(@"SEL$A7519627" "T1"@"SEL$1" "VW_LAT_AE9E49E8"@"SEL$AE9E49E8")
      NO_ACCESS(@"SEL$A7519627" "VW_LAT_AE9E49E8"@"SEL$AE9E49E8")
      FULL(@"SEL$A7519627" "T1"@"SEL$1")
      OUTLINE(@"SEL$1")
      OUTLINE(@"SEL$9EC647DD")
      OUTLINE(@"SEL$2")
      OUTLINE(@"SEL$AE9E49E8")
      DECORRELATE(@"SEL$BCD4421C")
      OUTLINE(@"SEL$6226B99A")
      MERGE(@"SEL$9EC647DD")
      OUTLINE(@"SEL$0E991E55")
      OUTLINE(@"SEL$BCD4421C")
      DECORRELATE(@"SEL$6226B99A")
      OUTLINE_LEAF(@"SEL$A7519627")
      DECORRELATE(@"SEL$BCD4421C")
      OUTLINE_LEAF(@"SEL$6226B99A")
      ALL_ROWS
      DB_VERSION('12.1.0.1')
      OPTIMIZER_FEATURES_ENABLE('12.1.0.1')
      IGNORE_OPTIM_EMBEDDED_HINTS
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("T1"."ID1"="ITEM_2"(+))
   4 - filter("T2"."ID1">100 OR "T2"."START_DATE"<=:1 AND "T2"."END_DATE">=:2)

Notice the apparition of view operation (VW_DCL_1B0973D4) at line n°3 in the above execution plan. This is Oracle way to signal the presence of a decorrelated lateral view transformation. If you want to learn more about this new transformation then you can read this US patent

DECORRELATE and NO_DECORRELATE hint

Before we go on, I want to show you very quickly how we can locally disable this new transformation using the following hint:

NO_DECORRELATE(@"SEL$BCD4421C")
explain plan for  
select /*+ NO_DECORRELATE(@"SEL$BCD4421C") */
   t1.*
  ,t2.*
from
   t1
left outer join
   t2
on
  (t1.id1 = t2.product_t1
  and
     (t2.start_date <= :1 and t2.end_date >= :2
        or (t2.id1 > 100)
     )
  );

select * from table(dbms_xplan.display);

-------------------------------------------------------------------------
| Id  | Operation                             | Name            | Rows  |
-------------------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                 | 10000 |
|   1 |  NESTED LOOPS OUTER                   |                 | 10000 |
|   2 |   TABLE ACCESS FULL                   | T1              | 10000 |
|   3 |   VIEW                                | VW_LAT_1B0973D4 |     1 |
|*  4 |    TABLE ACCESS BY INDEX ROWID BATCHED| T2              |     1 |
|*  5 |     INDEX RANGE SCAN                  | IDX_T2_USR_1    |     1 |
-------------------------------------------------------------------------  

Predicate Information (identified by operation id):
---------------------------------------------------
   4 - filter("T2"."ID1">100 OR "T2"."START_DATE"<=:1 AND "T2"."END_DATE">=:2)
   5 - access("T1"."ID1"="T2"."PRODUCT_T1")

As you can see using the NO_DECORRELATE(@”SEL$BCD4421C”) hint we cancelled the VW_DCL_1B0973D4 decorrelated lateral view since we have now a simple VW_LAT_1B0973D4 lateral view

In passing it is important to note that the outline presented above contains three hints with the word DECORRELATE:

Outline Data
-------------
  /*+
      DECORRELATE(@"SEL$BCD4421C")
      DECORRELATE(@"SEL$6226B99A")
      DECORRELATE(@"SEL$BCD4421C")
  */

Of these three hints it is only the hint referencing the query block named @”SEL$BCD4421C” that will succeed to correlated/decorrelate the lateral view. I don’t know why this hint with that query block named is repeated two times. I don’t know also why using the other hint with the query block named @”SEL$6226B99A” doesn’t decorrelate the lateral view.

Looking back into the 10053 trace file I found the following lines for query blocks SEL$BCD4421C and SEL$6226B99A respectively:

DCL: Checking validity of lateral view decorrelation SEL$BCD4421C (#1)
DCL: Passed decorrelation validity for lateral view block SEL$BCD4421C (#1)
DCL: Decorrelation of lateral view query block SEL$BCD4421C (#1).
Registered qb: SEL$6226B99A 0x8e4a5b58 (VIEW DECORRELATED SEL$BCD4421C; SEL$BCD4421C)

This tends to suggest that SEL$6226B99A represents the query block name of the lateral view and that SEL$BCD4421C represents the query block name of the decorrelated lateral view. Simply put I think that when confronted to such a kind of situation where you want to locally cancel the lateral decorrelation view than use the hint NO_DECORRELATE with the query block name that exists in the outline and doesn’t exist in the query block name given by the dbms_xplan.display_cursor:

Query Block Name / Object Alias (identified by operation id):
-------------------------------------------------------------
   1 - SEL$A7519627
   2 - SEL$A7519627 / T1@SEL$1
   3 - SEL$6226B99A / VW_LAT_AE9E49E8@SEL$AE9E49E8
   4 - SEL$6226B99A / T2@SEL$1

Don’t ask me why not simply using the table aliases as they appear in the original query to cancel this transformation. All my attempts have been unsuccessful.

Hidden parameter

What exactly is the parameter driving this transformation?

alter session set "_optimizer_ansi_join_lateral_enhance"=false;

explain plan for 
select 
   t1.*
  ,t2.*
from
   t1
left outer join
   t2
on
  (t1.id1 = t2.product_t1
  and
     (t2.start_date <= :1 and t2.end_date >= :2
        or (t2.id1 > 100)
     )
  );

select * from table(dbms_xplan.display(format =>'+outline'));
-------------------------------------------------------------------------
| Id  | Operation                             | Name            | Rows  |
-------------------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                 | 10000 |
|   1 |  NESTED LOOPS OUTER                   |                 | 10000 |
|   2 |   TABLE ACCESS FULL                   | T1              | 10000 |
|   3 |   VIEW                                | VW_LAT_1B0973D4 |     1 |
|*  4 |    TABLE ACCESS BY INDEX ROWID BATCHED| T2              |     1 |
|*  5 |     INDEX RANGE SCAN                  | IDX_T2_USR_1    |     1 |
-------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      BATCH_TABLE_ACCESS_BY_ROWID(@"SEL$BCD4421C" "T2"@"SEL$1")
      INDEX_RS_ASC(@"SEL$BCD4421C" "T2"@"SEL$1" ("T2"."PRODUCT_T1" "T2"."START_DATE"))
      USE_NL(@"SEL$0E991E55" "VW_LAT_AE9E49E8"@"SEL$AE9E49E8")
      LEADING(@"SEL$0E991E55" "T1"@"SEL$1" "VW_LAT_AE9E49E8"@"SEL$AE9E49E8")
      NO_ACCESS(@"SEL$0E991E55" "VW_LAT_AE9E49E8"@"SEL$AE9E49E8")
      FULL(@"SEL$0E991E55" "T1"@"SEL$1")
      OUTLINE(@"SEL$1")
      OUTLINE(@"SEL$9EC647DD")
      OUTLINE(@"SEL$2")
      OUTLINE(@"SEL$AE9E49E8")
      MERGE(@"SEL$9EC647DD")
      OUTLINE_LEAF(@"SEL$0E991E55")
      OUTLINE_LEAF(@"SEL$BCD4421C")
      ALL_ROWS
      OPT_PARAM('_optimizer_ansi_join_lateral_enhance' 'false')
      DB_VERSION('12.1.0.1')
      OPTIMIZER_FEATURES_ENABLE('12.1.0.1')
      IGNORE_OPTIM_EMBEDDED_HINTS
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   4 - filter("T2"."ID1">100 OR "T2"."START_DATE"<=:1 AND "T2"."END_DATE">=:2)
   5 – access("T1"."ID1"="T2"."PRODUCT_T1")

Notice again how we have succeeded to cancel the decorrelated lateral view transformation at a global level by setting off the hidden parameter _optimizer_ansi_join_lateral_enhance.

SUMMARY
In summary, does a decorrelated lateral view causes always a performance issue? No, not necessarily always. However if you fill in need of disabling it then you can:

  • Use the corresponding no_decorrelate hint with the correct query block name for a local effect
  • Set off the _optimizer_ansi_join_lateral_enhance parameter for a global effect with the usual warning when changing a hidden parameter.

April 29, 2017

Index and mixed order by

Filed under: Index,Tuning — hourim @ 12:10 pm

The orderly arrangement of the index keys in the leaf blocks makes it possible for Oracle to traverse the index both backward and forward. It renders, as such, the CBO in the plausible capacity of eliminating a costly order by operation. This order by elimination is, however, possible provided the index is used and the query order is wanted in the exact direction of the index columns backward or forward.

Let’s clarify this with a simple example

The model

create table t1 
as select 
    rownum n1
   ,trunc((rownum-1)/3) n2
   ,mod(rownum,5) n3
   ,lpad('x',5) v1
from dual 
connect by level <=1e3;

create index t1_idx on t1(n1, n2, n3);

explain plan for 
select * from t1 
where n1 <=5 
order by 
      n1,n2, n3;

select * from table(dbms_xplan.display);

------------------------------------------------------
| Id  | Operation                   | Name   | Rows  | 
------------------------------------------------------
|   0 | SELECT STATEMENT            |        |     5 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1     |     5 |
|*  2 |   INDEX RANGE SCAN          | T1_IDX |     5 |
------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("N1"<=5)

You should note in the above expected execution plan the absence of a SORT ORDER BY operation. This has been possible because the order in which the query result set is wanted matches perfectly the order in which the columns have been arranged in the index. Oracle is also able to avoid a SORT operation if the order of the query result set is wanted in the exact reverse direction of the index columns as shown below:

explain plan for 
select * from t1 
where n1 <=5 
order by 
      n1 desc,n2 desc, n3 desc;

select * from table(dbms_xplan.display);

-------------------------------------------------------
| Id  | Operation                    | Name   | Rows  |
-------------------------------------------------------
|   0 | SELECT STATEMENT             |        |     5 |
|   1 |  TABLE ACCESS BY INDEX ROWID | T1     |     5 |
|*  2 |   INDEX RANGE SCAN DESCENDING| T1_IDX |     5 |
-------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("N1"<=5)

Again the SORT ORDER BY operation has been avoided thanks to the ability of Oracle to traverse the index in a descending order (backward). However, if the result set is wanted in an order that is neither in the index direction nor in its opposite one, then it will be impossible for Oracle to avoid a SORT operation even when the index is used as the following demonstrates:

explain plan for 
select * from t1 
where n1 <=5 
order by 
      n1 ,n2 desc, n3 ;

select * from table(dbms_xplan.display);

---------------------------------------------------------------
| Id  | Operation                            | Name   | Rows  |
---------------------------------------------------------------
|   0 | SELECT STATEMENT                     |        |     5 |
|   1 |  SORT ORDER BY                       |        |     5 |
|   2 |   TABLE ACCESS BY INDEX ROWID BATCHED| T1     |     5 |
|*  3 |    INDEX RANGE SCAN                  | T1_IDX |     5 |
---------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("N1"<=5)

As you can see, this time, in contrast to the first two cases, a SORT ORDER BY operation kicks in. This operation becomes mandatory since the result set has been desired in an order that doesn’t match the order of the index columns. Spot in passing that when Oracle avoids a SORT ORDER BY it doesn’t visit the T1 table in a BATCHED mode and vice versa. I have already wrote a note about this here.

If you are in a situation where you want absolutely to avoid a costly SORT ORDER BY operation then you need to create a new matching index like the following one:

alter index t1_idx invisible;

create index t1_idx2 on t1(n1,n2 desc, n3);

explain plan for 
select * from t1 
where n1 <=5 
order by 
      n1 ,n2 desc, n3 ;

select * from table(dbms_xplan.display);

---------------------------------------------------------------
| Id  | Operation                   | Name    | Rows  | Bytes | 
---------------------------------------------------------------
|   0 | SELECT STATEMENT            |         |     5 |    85 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1      |     5 |    85 |
|*  2 |   INDEX RANGE SCAN          | T1_IDX2 |     5 |       |
---------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("N1"<=5)

The same desired order could also be achieved without a SORT ORDER BY operation by creating the following index:

drop index t1_idx2;

create index t1_idx2 on t1(n1 desc, n2, n3 desc);

explain plan for 
select * from t1 
where n1 <=5 
order by 
      n1 ,n2 desc, n3 ;

select * from table(dbms_xplan.display);

--------------------------------------------------------
| Id  | Operation                    | Name    | Rows  | 
--------------------------------------------------------
|   0 | SELECT STATEMENT             |         |     5 | 
|   1 |  TABLE ACCESS BY INDEX ROWID | T1      |     5 |  
|*  2 |   INDEX RANGE SCAN DESCENDING| T1_IDX2 |     9 | 
--------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access(SYS_OP_DESCEND("N1")>=HEXTORAW('3EF9FF') AND SYS_OP_DESCEND("N1") 
              IS NOT NULL)
       filter(SYS_OP_UNDESCEND(SYS_OP_DESCEND("N1"))<=5)

Looking at the above exotic predicate I am not sure that this last case will always outperform the preceding one.

So What?

Since you know roughly how an index should be designed to make a sort operation avoidable, are you going to change the ORDER BY clause to suit an existing index or create a new index to match the ORDER BY clause? The answer to this question has been magnificently given by Peter Gulutzan and Trudy Pelzer in their SQL Performance Tuning book “In both cases, we would have to say No. If you did either one, you’d be crossing the boundary between “taking advantage of a side effect” and “depending on a side effect.”

Because indexes exist mainly to support WHERE clause rather than ORDER BY I would very probably suggest to not change the existing index. Particularly when real life experience confirms that it is easier to add an index than to drop an existing one. But when we know how a well designed index can precisely cover a query and eventually avoid a sort operation, we can either challenge the client for the order in which he wants his result set or, all things being equal, design an index to match the order by clause.

Summary

Watch out your critical queries necessitating a certain order. If you are going to design an index covering one of their predicates, and all things being equal, think about designing this index so that its columns will match the query order by clause either backward or forward. But you shouldn’t being always forcing the CBO to use an index merely because it will avoid a SORT operation.

PS: this post has been inspired by an excellent blog on PostgreSQL by Bruce Momjian

April 16, 2017

12cR2: unreproducible SPM and cursor bind awareness

Filed under: Sql Plan Managment — hourim @ 4:53 pm

In the previous article we knew that, as of Oracle 12cR2, a cursor can’t be kept bind aware when it is protected by a single enabled and accepted SPM plan baseline. In other words, the preceding article explained that if you want a cursor to be both bind aware and protected against any unexpected execution plan change, you need then to have at least two enabled and accepted SPM baselines for this cursor.

That’s said multiple enabled and accepted SPM plan doesn’t necessarily means multiple enabled, accepted and reproducible SPM plans. And, as this article will demonstrate it, this reproducible path seems to have been ignored by the Oracle optimizer group when disabling the bind awareness property explained in the previous article. The new 12cR2 internal code disabling and enabling ACS seems to be based only on the accepted and enabled property of the bind aware cursor SPM plan. Let’s demonstrate this with a simple example

Setting the Scenes

-- create table, index and gather statistics
create table t1
    (col1  number
    ,col2  varchar2(50)
    ,flag  varchar2(2));
 
 insert into t1
    select rownum
          ,lpad('X',50,'X')
          ,case when rownum = 1
            then 'Y1'
               when rownum = 2
            then 'Y2'
               when mod(rownum,2) = 0
            then 'N1'
            else 'N2'
           end
    from   dual
connect by rownum <= 100000;
 
create index i1 on t1(flag);

BEGIN 
  dbms_stats.gather_table_stats
      (user
     ,'t1'
     ,method_opt => 'for all columns size skewonly' 
     ,estimate_percent => dbms_stats.auto_sample_size );
END; 
/

-- generate a bind aware cursor
alter session set cursor_sharing=force;

select count(*), max(col2) from t1 where flag = 'Y1';
select count(*), max(col2) from t1 where flag = 'N1';
select count(*), max(col2) from t1 where flag = 'N1';
select count(*), max(col2) from t1 where flag = 'Y1';

Observation

The above script will generate a bind sensitive and bind aware cursor as shown below:

select
    sql_id
  ,child_number
  ,is_bind_sensitive 
  ,is_bind_aware
  ,executions
  ,invalidations
  ,object_status
from
   gv$sql 
where
   sql_id = '6fbvysnhkvugw'
and is_shareable = 'Y';

SQL_ID        CHILD_NUMBER I I EXECUTIONS INVALIDATIONS OBJECT_STATUS      
------------- ------------ - - ---------- ------------- -------------
6fbvysnhkvugw            1 Y Y          1             0 VALID              
6fbvysnhkvugw            2 Y Y          1             0 VALID 

Let’s load these two execution plans (child number 1 and 2) into a SPM baseline and make few verifications:

declare
   rs pls_integer;
begin
   rs := dbms_spm.load_plans_from_cursor_cache('6fbvysnhkvugw');
end;
/

col plan_name format a30
select
  plan_name ,origin, enabled
from dba_sql_plan_baselines;

PLAN_NAME                      ORIGIN                        ENA
------------------------------ ----------------------------- ---
SQL_PLAN_d9tch6banyzg9616acf47 MANUAL-LOAD-FROM-CURSOR-CACHE YES
SQL_PLAN_d9tch6banyzg98576eb1f MANUAL-LOAD-FROM-CURSOR-CACHE YES

select
    sql_id
  ,child_number
  ,is_bind_sensitive 
  ,is_bind_aware
  ,executions
  ,invalidations
  ,object_status
from
   gv$sql 
where
   sql_id = '6fbvysnhkvugw'
and is_shareable = 'Y';

SQL_ID        CHILD_NUMBER I I EXECUTIONS INVALIDATIONS OBJECT_STATUS      
------------- ------------ - - ---------- ------------- -------------------
6fbvysnhkvugw            1 Y Y          1             1 INVALID_UNAUTH     
6fbvysnhkvugw            2 Y Y          1             1 INVALID_UNAUTH

As you can see the creation of the two SPM plans has already invalidated the two underlying child cursors making them both not anymore shareable for the next execution(INVALID_UNAUTHORIZED).

Now that we have a bind aware cursor protected by two enabled and accepted SPM plans let’s launch it a couple of time in order see what will happen:

SQL> select count(*), max(col2) from t1 where flag = 'Y1';

  COUNT(*) MAX(COL2)                                         
---------- --------------------------------------------------
         1 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

SQL> select * from table(dbms_xplan.display_cursor);

SQL_ID  6fbvysnhkvugw, child number 0
-------------------------------------
 
Plan hash value: 3625400295
-----------------------------------------------------
| Id  | Operation                    | Name | Rows  | 
-----------------------------------------------------
|   0 | SELECT STATEMENT             |      |       | 
|   1 |  SORT AGGREGATE              |      |     1 | 
|   2 |   TABLE ACCESS BY INDEX ROWID| T1   |     1 | 
|*  3 |    INDEX RANGE SCAN          | I1   |     1 | 
-----------------------------------------------------

 
Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("FLAG"=:SYS_B_0)
 
Note
-----
   - SQL plan baseline SQL_PLAN_d9tch6banyzg98576eb1f used for this statement

As expected the apparition of the child number 0 above means that Oracle has effectively flushed out the previous couple of bind aware child cursor (1,2). We need now to go again through a warming up period for ACS to kick in as shown below:

select
    sql_id
  ,child_number
  ,is_bind_sensitive 
  ,is_bind_aware
  ,executions
  ,invalidations
  ,object_status
from
   gv$sql 
where
   sql_id = '6fbvysnhkvugw'
and is_shareable = 'Y';

SQL_ID        CHILD_NUMBER I I EXECUTIONS INVALIDATIONS OBJECT_STATUS      
------------- ------------ - - ---------- ------------- -------------
6fbvysnhkvugw            1 Y N          1             1 VALID     

select count(*), max(col2) from t1 where flag = 'N1';
select count(*), max(col2) from t1 where flag = 'N1';
select count(*), max(col2) from t1 where flag = 'Y1';

select
    sql_id
  ,child_number
  ,is_bind_sensitive 
  ,is_bind_aware
  ,executions
  ,invalidations
  ,object_status
from
   gv$sql 
where
   sql_id = '6fbvysnhkvugw'
and is_shareable = 'Y';

SQL_ID        CHILD_NUMBER I I EXECUTIONS INVALIDATIONS OBJECT_STATUS      
------------- ------------ - - ---------- ------------- -------------
6fbvysnhkvugw            2 Y Y          1             1 VALID              
6fbvysnhkvugw            3 Y Y          1             0 VALID    

We’ve, finally, succeeded building a situation with which we can proof that the Oracle optimizer group has ignored disabling ACS even when it is protected by a single reproducible SPM plan as the following will demonstrate:

-- disabling the i1 index so that the SPM plan SQL_PLAN_d9tch6banyzg98576eb1f 
-- will be irreproducible

alter index i1 invisible;

Setting the i1 index invisible, the index range scan SPM plan will inevitably be ignored by the CBO because it is not anymore reproducible. The Full table scan SPM plan will become the unique enforced execution plan whatever the bind variable selectivity will be as shown below:

SQL> select count(*), max(col2) from t1 where flag = 'Y1';

  COUNT(*) MAX(COL2)                                         
---------- --------------------------------------------------
         1 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

SQL> select * from table(dbms_xplan.display_cursor);

SQL_ID  6fbvysnhkvugw, child number 2
-------------------------------------
select count(*), max(col2) from t1 where flag = :"SYS_B_0"
 
Plan hash value: 3724264953
---------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes |
---------------------------------------------------
|   0 | SELECT STATEMENT   |      |       |       |
|   1 |  SORT AGGREGATE    |      |     1 |    54 |
|*  2 |   TABLE ACCESS FULL| T1   |     1 |    54 |
---------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("FLAG"=:SYS_B_0)
 
Note
-----
   - SQL plan baseline SQL_PLAN_d9tch6banyzg9616acf47 used for this statement

When Oracle has detected that the index range scan SPM plan is not anymore reproducible it did two things:

a. invalidated and flushed out both index scan (child number 3) and full table scan(child number 2) cursor
b. created a new bind sensitive and not yet bind aware cursor n°2 as proofed below:

select
   sql_id
  ,child_number
  ,is_bind_sensitive 
  ,is_bind_aware
  ,executions
  ,invalidations
  ,object_status
from
   gv$sql 
where
   sql_id = '6fbvysnhkvugw'
and is_shareable = 'Y';

SQL_ID        CHILD_NUMBER I I EXECUTIONS INVALIDATIONS OBJECT_STATUS      
------------- ------------ - - ---------- ------------- -------------
6fbvysnhkvugw            2 Y N          1             2 VALID  

While officially there are still two accepted and enabled SPM plan baselines, in fact, only one SPM plan can be effectively be used to guarantee unpredictable performance degradation. So why Oracle has not disabled the bind sensitiveness property of this cursor? Is Oracle not claiming that it is useless to keep ACS working in presence of a single SPM plan?

select
   child_number
  ,bucket_id
  ,count
from
   gv$sql_cs_histogram
where
   sql_id = '6fbvysnhkvugw'
and child_number = '2';

CHILD_NUMBER  BUCKET_ID      COUNT
------------ ---------- ----------
           2          0          1
           2          1          0
           2          2          0

Given the above situation we need only two extra executions of the same query at bucket_id n°1 or 2 and the cursor will become bind aware as shown below:

select count(*), max(col2) from t1 where flag = 'N1';
select count(*), max(col2) from t1 where flag = 'N1';

SQL> select
        sql_id
      ,child_number
      ,is_bind_sensitive 
      ,is_bind_aware
      ,executions
      ,invalidations
      ,object_status
    from
      gv$sql 
   where
     sql_id = '6fbvysnhkvugw'
   and is_shareable = 'Y';

SQL_ID        CHILD_NUMBER I I EXECUTIONS INVALIDATIONS OBJECT_STATUS      
------------- ------------ - - ---------- ------------- -------------
6fbvysnhkvugw            3 Y Y          1             1 VALID  

Summary

The developer at Oracle corporation has not considered stopping a cursor from being bind aware when one of its two SPM plans is indirectly rendered irreproducible. The internal code implemented by Oracle when deciding to stop ACS seems to be based only the Enabled and Accepted property of the corresponding SPM plan. Unfortunately an accepted and enabled SPM plan is not guaranteed to be reproducible. This is why we can still find, in 12cR2, a bind aware cursor protected by a single SPM plan.

March 25, 2017

De-Correlated Lateral view: VW_DCL_mmm

Filed under: Oracle — hourim @ 5:18 pm

Oracle Cost Based Optimizer has a number of query rewrites and transformations both simple and complex, old and new. Wether you know very few of them or only the most common ones, I believe it is worth keeping yourself well updated with the new ones as it might introduce strange performance issue like the one I have encountered very recently. This new 12c transformation I want to explain here is known as De-Correlated Lateral view. It appears in execution plans as VW_DCL_mmm. This article examines this transformation in the context of a real life production query.

A critical query, executed several hundred of thousands of times within a two hours batch job consumes 67K of Logical I/O per execution. It made the job systematically fail with a time out error. The average execution time of this query being less than one second, I ruled out the possibility of using SQL monitoring report to spot where most of these Logical I/O are coming from. Hopefully the high number of times this query is launched during the batch job makes its activity sampled into ASH. As such, via the following simple query I knew exactly at which operations of the corresponding execution plan this query is spending most of its time and resource:

select
   sql_plan_line_id
  ,count(1)
from
   gv$active_session_history
where
 sample_time between to_date('22032017 23:00:18', 'ddmmyyyy hh24:mi:ss')
             and     to_date('22032017 23:50:18', 'ddmmyyyy hh24:mi:ss')
and
 sql_id = '94jkz74mvwmbj'
group by
   sql_plan_line_id
order by 2 desc;

SQL_PLAN_LINE_ID   COUNT(1)
---------------- ----------
              48         41
              47         9
	          11         8
              30         1
SQL> select * from table(dbms_xplan.display_curosr('94jkz74mvwmbj'));                                                                              
----------------------------------------------------------------          
| Id  | Operation                     | Name            | Rows  |                
-----------------------------------------------------------------                                                                                   
|*  11|  HASH JOIN OUTER              |                 |      3|     
|   12|  NESTED LOOP OUTER            |                 |      3|                                           
|   33|   TABLE ACCESS BY INDEX ROWID | T1              |      1|  
|*  34|   INDEX UNIQUE SCAN           | T1_PK           |      1|           
|   47|  VIEW                         | VW_DCL_52812513 |  5340K|                                           
|*  48|    TABLE ACCESS FULL          | T2              |  5340K|                                           
-----------------------------------------------------------------         

Reduced to the bare minimum, the above execution plan is the result of a left outer join between table T1 and table T2 using a primary-foreign key relationship (predicate n°11) and a mix of an OR and an AND predicate applied on table T2 (operation n° 48). We will see later in this article that the two bolded words above represent the key words of a Lateral View.

As per regards to the real life query I was practically sure that a NESTED LOOP with table T1 as the outer row source probing the inner row source via an existing foreign key index will end up by filtering table T2 using the OR and the AND predicate but only on the rows that survived the primary-foreign key join.

But what the heck is this VW_DCL_52812513 view? It is useless to say that I didn’t find any related information both using Google and MyOracle Support.

After a couple of hour of investigation at the client site I ended up finding a work around and pushed an urgent fix into PRODUCTION. Notice with me how the fix has transformed the statistics of this query and its batch job:

SQL> @sqlstats.sql 
Enter value for sql_id:  94jkz74mvwmbj

CHILD_NUMBER SQL_PROFILE                   PLAN_HASH_VALUE  AVG_GETS  EXECS
----------- ------------------------------ --------------- ---------- ------
4                                           1521027974       66288     1976  → old execution plan
6           PROFILE_94jkz74mvwmbj_MANUAL     553415384          90    10092  → new execution plan

In order to definitely understand this new transformation and share it with you I’ve engineered the following model:

Setting the Scenes

create table t1
as
 with generator as (
   select --+ materialize
      rownum id
   from dual
   connect by level <= 1000
)
select
    rownum id1,
    mod(rownum-1,2) flag1,
    mod(rownum-1,3) flag2,
    rownum          n1,
    lpad(rownum,30) v1
from
   generator v1,
   generator v2
where
   rownum <= 1e4;
   
alter table t1 add constraint t1_pk primary key (id1);

create table t2
as 
 select
    level id1
   ,trunc((rownum+2)/2) product_t1
   ,date '2012-06-07' + mod((level-1)*5,10) + interval '5' minute start_date
   ,date '2012-06-08' + mod((level-1)*5,10) + interval '5' minute end_date
   ,rpad('xx',10) padding
from
   dual
connect by level <=1e4;

alter table t2 add constraint t2_pk primary key (id1);
alter table t2 add constraint t2_t1_fk foreign key (product_t1) references t1(id1);

-- creating an index covering the FK deadlock threat and other business requirements
create index idx_t2_usr_1 on t2(product_t1, start_date);

Here’s below the query (and its execution plan) with which I have reproduced the interesting part of the client’s real life query:

explain plan for
select
   t1.*
  ,t2.*
from
   t1
left outer join
   t2
on
  (t1.id1 = t2.product_t1
  and
     (t2.start_date <= :1 and t2.end_date >= :2
        or (t2.id1 > 100)
     )
  );

SQL> select * from table(dbms_xplan.display);                                                                                                  
-------------------------------------------------------                                           
| Id  | Operation           | Name            | Rows  |                                           
------------------------------------------------------                                           
|   0 | SELECT STATEMENT    |                 | 14925 |                                           
|*  1 |  HASH JOIN OUTER    |                 | 14925 |                                             
|   2 |   TABLE ACCESS FULL | T1              | 10000 |                                           
|   3 |   VIEW              | VW_DCL_C83A7ED5 |  9926 |                                           
|*  4 |    TABLE ACCESS FULL| T2              |  9926 |                                           
------------------------------------------------------                                           
                                                                                                                                  
Predicate Information (identified by operation id):                                                                               
---------------------------------------------------                                                                                                                                                                                         
   1 - access("T1"."ID1"="ITEM_2"(+))                                                                                             
   4 - filter("T2"."ID1">100 OR "T2"."START_DATE"<=:1 
         AND "T2"."END_DATE">=:2) 

Spot again how the predicate part can be so crucial to indicate the way you should pursue when troubleshooting query performance issues:

Predicate Information (identified by operation id):                                                                               
--------------------------------------                                                                                                                                                                                        
   1 - access("T1"."ID1"="ITEM_2"(+))                

While I recognize the (+) symbol as the Oracle way of re-architecting and ANSI outer join, I admit that I have no idea what the heck is this ITEM_2 in predicate n°1?

When all else fails then a 10053 trace file might help

alter session set tracefile_identifier='VW_DCL_MHO_XXXX';
alter session set events '10053 trace name context forever, level 1';
explain plan for
select
   t1.*
  ,t2.*
from
   t1
left outer join
   t2
on
  (t1.id1 = t2.product_t1
  and
     (t2.start_date <= :1 and t2.end_date >= :2
        or (t2.id1 > 100)
     )
  );
alter session set events '10053 trace name context off';           

Digging a little bit into the generated trace file I found that Oracle has transformed the original query into the following one:

SELECT 
  t1.id1 id1,
  t1.flag1 flag1,
  t1.flag2 flag2,
  t1.n1 n1,
  t1.v1 v1,
  vw_dcl_1b0973d4.item_1 id1,
  vw_dcl_1b0973d4.item_2 product_t1,
  vw_dcl_1b0973d4.item_3 start_date,
  vw_dcl_1b0973d4.item_4 end_date,
  vw_dcl_1b0973d4.item_5 padding
FROM c##mhouri.t1 t1,
  (SELECT 
    t2.id1 item_1_0,
    t2.product_t1 item_2_1,
    t2.start_date item_3_2,
    t2.end_date item_4_3,
    t2.padding item_5_4
  FROM c##mhouri.t2 t2
  WHERE t2.start_date<=:b1 AND t2.end_date >=:b2
  OR t2.id1           >100
  ) VW_DCL_1B0973D4
WHERE t1.id1=VW_DCL_1B0973D4.item_2(+) 

Simply put Oracle did two things:
1. Created a Lateral view
2. De-Correlated this lateral view by excluding the join predicate with table T1 from the Lateral view

This is clearly backed up by the following lines in the same trace file

DCL: Checking validity of lateral view decorrelation SEL$BCD4421C (#1)
DCL: Passed decorrelation validity for lateral view block SEL$BCD4421C (#1)
DCL: Decorrelation of lateral view query block SEL$BCD4421C (#1).
Registered qb: SEL$6226B99A 0x693ad4d0 (VIEW DECORRELATED SEL$BCD4421C; SEL$BCD4421C)

Since I have finally succeeded to understand what this transformation is I knew what else I have to do:

explain plan for
select /*+ optimizer_features_enable('11.2.0.4') */
   t1.*
  ,t2.*
from
   t1
left outer join
   t2
on
  (t1.id1 = t2.product_t1
  and
     (t2.start_date <= :1 and t2.end_date >= :2
        or (t2.id1 > 100)
     )
  );

SQL> select * from table(dbms_xplan.display);                                                                                                                                                                                                       
--------------------------------------------------------------                       
| Id  | Operation                     | Name         | Rows  |                                   
--------------------------------------------------------------                                   
|   0 | SELECT STATEMENT              |              | 10000 |                                   
|   1 |  NESTED LOOPS OUTER           |              | 10000 |                                   
|   2 |   TABLE ACCESS FULL           | T1           | 10000 |                                   
|   3 |   VIEW                        |              |     1 |                                   
|*  4 |    TABLE ACCESS BY INDEX ROWID| T2           |     1 |                                   
|*  5 |     INDEX RANGE SCAN          | IDX_T2_USR_1 |     1 |                                   
--------------------------------------------------------------                                   
                                                                                                                                  
Predicate Information (identified by operation id):                                                                               
---------------------------------------------------                                                                                                                                                                                                           
   4 - filter("T2"."ID1">100 OR "T2"."START_DATE"<=:1 AND "T2"."END_DATE">=:2)                                                    
   5 - access("T1"."ID1"="T2"."PRODUCT_T1")                   

This exactly what I was expecting at the beginning of my investigations. Annihilating the effect of the de-correlated lateral view will open a much better path at least for my real life query case.

Summary

As of now I hope that this article can help identifying what a VW_DCL_mmm transformation is. I hope as well that google will hist this article when asked about this particular transformation.

March 18, 2017

12cR2: SPM and cursor bind awareness property

Filed under: Oracle — hourim @ 7:24 am

In the previous article we knew that, as of Oracle 12cR2, it is now possible to capture SPM baselined plans from AWR historical tables. In this article we are going to see how in 12cR2 a cursor will immediately stop to be bind aware when it is protected by a single SPM plan. The basic idea driving this new implementation is that, since the Adaptive Cursor Sharing main goal is to generate multiple optimal execution plans, if you decide to constrain it with a single SPM plan, you are implicitly asking Oracle to stop generating multiple execution plans.

Let’s demonstrate this new ACS-SPM relationship using the same model as that of the preceding article

Setting the Scenes

create table t_acs(n1 number, n2 number);

BEGIN
 for j in 1..1200150 loop
  if j = 1 then
    insert into t_acs values (j, 1);
  elsif j>1 and j<=101 then
    insert into t_acs values(j, 100);
  elsif j>101 and j<=1101 then
    insert into t_acs values (j, 1000);
  elsif j>10001 and j<= 110001 then
    insert into t_acs values(j,10000); 
  else
    insert into t_acs values(j, 1000000); end if;
 end loop; 
commit;
END;
/

create index t_acs_i1 on t_acs(n2);

BEGIN 
  dbms_stats.gather_table_stats
     (user
     ,'t_acs'
     ,method_opt => 'for all columns size skewonly' 
     ,cascade => true
     ,estimate_percent => dbms_stats.auto_sample_size );
END; 
/

var ln2 number;
exec :ln2 := 100
select count(1) from t_acs where n2 = :ln2;

exec :ln2 := 1e6
select count(1) from t_acs where n2 = :ln2;
select count(1) from t_acs where n2 = :ln2;

exec :ln2 := 100
select count(1) from t_acs where n2 = :ln2;


Observation

The execution of the above script will end up by producing two bind sensitive, bind aware and shareable cursors as shown below:

select 
    sql_id
  ,child_number
  ,is_bind_sensitive 
  ,is_bind_aware
  ,executions
from
   v$sql 
where
   sql_id = 'f2pmwazy1rnfd'
and is_shareable = 'Y';

SQL_ID        CHILD_NUMBER I I EXECUTIONS
------------- ------------ - - ----------
f2pmwazy1rnfd            1 Y Y          1 → full scan plan
f2pmwazy1rnfd            2 Y Y          1 → index range scan plan

So far so good.
What if I decide to create a SPM execution plan in order to constrain the above query to use always the index range scan plan?

SQL> alter session set optimizer_capture_sql_plan_baselines = TRUE;

SQL> select count(1) from t_acs where n2 = :ln2;

  COUNT(1)
----------
       100

SQL> select count(1) from t_acs where n2 = :ln2;

  COUNT(1)
----------
       100

SQL> alter session set optimizer_capture_sql_plan_baselines = FALSE;

SQL> col plan_name format a40
SQL> select 
        plan_name ,origin, accepted, enabled
     from dba_sql_plan_baselines;

PLAN_NAME                        ORIGIN          ACC ENA
------------------------------- ---------------- --- ---
SQL_PLAN_fn4mhg52jx5z125348c47  AUTO-CAPTURE     YES YES

As such the next time I will run this query it will use the index range scan SPM baselined plan as shown below:

SQL> select count(1) from t_acs where n2 = :ln2;

SQL> select * from table(dbms_xplan.display_cursor);

SQL_ID  f2pmwazy1rnfd, child number 1                                                                                                                                              
-------------------------------------                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
Plan hash value: 1882749816                                                                                                                                                                                                                                                                                                                                    
----------------------------------------------                                                                                                    
| Id  | Operation         | Name     | Rows  |                                                                                                   
----------------------------------------------                                                                                                    
|   0 | SELECT STATEMENT  |          |       |                                                                                                    
|   1 |  SORT AGGREGATE   |          |     1 |                                                                                                     
|*  2 |   INDEX RANGE SCAN| T_ACS_I1 |   100 |                                                                                                  
----------------------------------------------                                                                                                    
                                                                                                                                                                                   
Predicate Information (identified by operation id):                                                                                                                                
---------------------------------------------------                                                                                                                                                                                                                                                                                              
   2 - access("N2"=:LN2)                                                                                                                                                           
                                                                                                                                                                                   
Note                                                                                                                                                                               
-----                                                                                                                                                                              
- SQL plan baseline SQL_PLAN_fn4mhg52jx5z125348c47 used for this statement  

But what you might ignore is that, following the creation of this SPM baselined plan, Oracle did something behind the scene as shown via the following cursor new situation:

select 
    sql_id
  ,child_number
  ,is_bind_sensitive 
  ,is_bind_aware
  ,executions
from
   v$sql 
where
   sql_id = 'f2pmwazy1rnfd'
and is_shareable = 'Y';

SQL_ID        CHILD_NUMBER I I EXECUTIONS
------------- ------------ - - ----------
f2pmwazy1rnfd            1 N N          1

If you analyze carefully the new situation compared to the old one you will realize that the following actions have been done behind the scenes

1. The previous child cursors n°1 and n°2 has been flushed out.
2. A new no bind sensitive and no bind aware child cursor n°1 has been created

It is the number of execution (only 1) of child cursor n°1 which clearly explains that the previous two bind aware cursors have been flushed out from memory due to the new SPM baselined plan. The child cursor n°0 is still in the shared pool but is in a non shareable status.

SQL> select 
        sql_id
      ,child_number
      ,is_bind_sensitive 
      ,is_bind_aware
      ,is_shareable
      ,executions
    from
       v$sql 
   where
      sql_id = 'f2pmwazy1rnfd';

SQL_ID        CHILD_NUMBER I I I EXECUTIONS
------------- ------------ - - - ----------
f2pmwazy1rnfd            0 Y N N          2 → non shareable
f2pmwazy1rnfd            1 N N Y          1

That’s the new 12cR2 implementation managing the relationship between ACS and SPM. ACS is disabled when it is constrained by a single accepted and enabled SPM plan.

If we disable the previous SPM baseline the cursor becomes bind aware after a warm up period as usual

SQL> @disSPM.sql 
Enter value for plan_name: SQL_PLAN_fn4mhg52jx5z125348c47

SQL> select 
          plan_name ,origin, enabled
      from dba_sql_plan_baselines;

PLAN_NAME                       ORIGIN       ENA
------------------------------- ------------ ---
SQL_PLAN_fn4mhg52jx5z125348c47  AUTO-CAPTURE NO 

exec :ln2 :=1e6
select count(1) from t_acs where n2 = :ln2;

exec :ln2 := 100
select count(1) from t_acs where n2 = :ln2;

etc…./...

SQL> select 
        sql_id
      ,child_number
      ,is_bind_sensitive 
      ,is_bind_aware
      ,is_shareable
      ,executions
    from
       v$sql 
   where
      sql_id = 'f2pmwazy1rnfd'
  ;

SQL_ID        CHILD_NUMBER I I I EXECUTIONS
------------- ------------ - - - ----------
f2pmwazy1rnfd            0 Y N N          2
f2pmwazy1rnfd            1 Y N N          3
f2pmwazy1rnfd            2 Y Y Y          1 → bind aware full scan
f2pmwazy1rnfd            3 Y Y Y          1 → bind aware range scan

Summary

As of Oracle 12cR2 when an ACS cursor having multiple optimal execution plans is constrained by a single enabled and accepted SPM plan, Oracle will age it out from memory and cancel its bind sensitive and bind aware properties. This will remain intact until the SPM is dropped or disabled. In the next article we will see what happens to an ACS cursor when one of its multiple accepted and enabled SPM plan is rendered unreproducible.

Next Page »

Blog at WordPress.com.

Tony's Oracle Tips

Tony Hasler's light hearted approach to learning about Oracle

Richard Foote's Oracle Blog

Focusing Specifically On Oracle Indexes, Database Administration and Some Great Music

Hatem Mahmoud Oracle's blog

Just another Oracle blog : Database topics and techniques

Mohamed Houri’s Oracle Notes

Qui se conçoit bien s’énonce clairement

Oracle Diagnostician

Performance troubleshooting as exact science

Raheel's Blog

Things I have learnt as Oracle DBA

Coskan's Approach to Oracle

What I learned about Oracle

So Many Oracle Manuals, So Little Time

“Books to the ceiling, Books to the sky, My pile of books is a mile high. How I love them! How I need them! I'll have a long beard by the time I read them”—Lobel, Arnold. Whiskers and Rhymes. William Morrow & Co, 1988.

EU Careers info

Your career in the European Union

Carlos Sierra's Tools and Tips

Tools and Tips for Oracle Performance and SQL Tuning

Oracle Scratchpad

Just another Oracle weblog

OraStory

Dominic Brooks on Oracle Performance, Tuning, Data Quality & Sensible Design ... (Now with added Sets Appeal)