Mohamed Houri’s Oracle Notes

March 17, 2018

Library cache lock

Filed under: Oracle — hourim @ 7:55 am

This is a very simple note, as much for my own documentation as anything else, showing how I have extremely rapidly identified what was causing a database application to wait on a library cache lock wait event.

The ASH of my customer case was showing the following list of predominant wait events:

select event, count(1)
from gv$active_session_history
where
    sample_time between to_date('09032018 00:46:00', 'ddmmyyyy hh24:mi:ss')
                and     to_date('09032018 10:44:00', 'ddmmyyyy hh24:mi:ss')
group by event
order by 2 desc;

EVENT                                  COUNT(1)
------------------------------------ ----------
library cache lock                       350174
library cache: mutex X                    54474
cursor: pin S wait on X                   10896
                                           2964

Naturally I wanted to know what sql_id is responsible of these library cache wait events first via ASH

select sql_id, count(1)
from gv$active_session_history
where
    sample_time between to_date('09032018 00:46:00', 'ddmmyyyy hh24:mi:ss')
                and     to_date('09032018 10:44:00', 'ddmmyyyy hh24:mi:ss')
and event in 
 ('library cache lock','library cache: mutex X','cursor: pin S wait on X')
group by sql_id
order by 2 desc;

SQL_ID            COUNT(1)
----------------- --------
6tcs65pchhp71       147902
9mqzzppbpa64j         7373
                         3	

And then via classical dbms_xplan to get the corresponding execution plan and v$sql to get the SQL text respectively:

SQL> select * from table(dbms_xplan.display_cursor('6tcs65pchhp71',null));

PLAN_TABLE_OUTPUT
----------------------------------------------------------------------------
SQL_ID: 6tcs65pchhp71 cannot be found
SQL> select sql_fulltext, executions, end_of_fetch_count
    from gv$sql
    where sql_id = '6tcs65pchhp71';

no rows selected

Let’s summarize: there is a sql_id which is responsible of a dramatic library cache wait event that I am monitoring at real time basis and which

  • has no execution plan in memory
  • and is not present in gv$sql

The above two points manifestly are symptoms of a sql query which hasn’t gone beyond the parse phase. In other words it might be a query which Oracle is not able to soft parse and thereby it has never reached the hard parse phase nor the execution step. Hopefully ASH can clearly show this:

select 
     in_parse
    ,in_hard_parse
    ,in_sql_execution
    ,count(1)
from gv$active_session_history
where
    sample_time between to_date('09032018 00:46:00', 'ddmmyyyy hh24:mi:ss')
                and     to_date('09032018 10:44:00', 'ddmmyyyy hh24:mi:ss')
and
   sql_id = '6tcs65pchhp71'
group by 
     in_parse
    ,in_hard_parse
    ,in_sql_execution
order by 4 desc;

I I I   COUNT(1)
- - - ----------
Y N N     162758
Y Y N        385

Indeed this query is almost always in the “parse” phase. So the initial question of what is causing this dramatic library cache lock turned to be : why this query is sticking at the parse phase?

I don’t know why I decided to look at the dba_hist_sqltext but it made my day:

SQL> select sql_text from dba_hist_sqltext where sql_id = '6tcs65pchhp71';

SQL_TEXT
--------------------------------------------------------------------------------
select col1, col2,...,coln from t1;

I was again lucky that this query has no where clause and no bind variable value to use so that I tried to run it as is and here what I found:

select col1, col2,...,coln 
from t1;
     *
ERROR at line 15:
ORA-00600 : internal error code, arguments: [qksvcGetGuardCol:2], [50574], [0], [],
[],[],[],[],[],[],[],[],[],

The multi-user concurrent system was repeating the same query several times and since this query was always failing during the parse phase we observed this library cache lock at the top of the wait events.

Advertisements

March 11, 2018

Which execution plan will be used?

Filed under: Sql Plan Managment — hourim @ 4:08 pm

Following a question raised by Franck Pachot on Twitter, I was surprised that 43% of the answers were false. The question was terribly simple : an SQL Plan Baseline contains two execution plans:

  • Plan 1: is enabled but not accepted
  • Plan 2: is not enabled but accepted

If the Cost Based Optimiser comes up with Plan 1, which execution plan will the query finally be allowed to use?

There are very simple rules to know when it comes to the interaction between the CBO and the presence of SPM baselined plans:

  • Not enabled SPM plans will not be used regardless of their acceptance status
  • Enabled SPM plans will not be used if they are not accepted

From Franck’s question we understand that SPM Plan 2 is not enabled; so we will rule it out from the correct answer possibilities. We can see as well that SPM Plan 1 is not accepted which makes it unusable by the CBO. Since both SPM execution plans are not usable because of their enabled or accepted properties the CBO will use the plan it will come up with : Plan 1 in this case.

Here below the demonstration (the model can be found here):

alter session set cursor_sharing= force;
alter session set optimizer_capture_sql_plan_baselines=true;

select count(1) from t_acs where n2 = 1e6;
select count(1) from t_acs where n2 = 1e6;

alter session set optimizer_capture_sql_plan_baselines=false;

select  sql_handle, plan_name, enabled, accepted
from dba_sql_plan_baselines
where signature = '7542536418437113777';

SQL_HANDLE             PLAN_NAME                      ENA ACC
---------------------- ------------------------------ --- ---
SQL_68ac78e9166427b1   SQL_PLAN_6jb3sx4b689xj3069e6f9 YES YES

DECLARE
   l_p  PLS_INTEGER;
BEGIN
  l_p := DBMS_SPM.alter_sql_plan_baseline(
            sql_handle      => 'SQL_68ac78e9166427b1',
            plan_name       => 'SQL_PLAN_6jb3sx4b689xj3069e6f9',
            attribute_name  => 'enabled',
            attribute_value => 'NO');
END;
/

select  sql_handle, plan_name, enabled, accepted
from dba_sql_plan_baselines
where signature = '7542536418437113777';

SQL_HANDLE                     PLAN_NAME                      ENA ACC
------------------------------ ------------------------------ --- ---
SQL_68ac78e9166427b1           SQL_PLAN_6jb3sx4b689xj3069e6f9 NO  YES

We have our Plan 2: Accepted but not Enabled.

Next I will generate Plan 1: Enabled but not Accepted

select count(1) from t_acs where n2 = 1;

SQL> select count(1) from t_acs where n2 = 1;

  COUNT(1)
----------
         1

SQL> select * from table(dbms_xplan.display_cursor)


SQL_ID  7ck8k47bnqpnv, child number 1

An uncaught error happened in prepare_sql_statement : ORA-01403: no data found

NOTE: cannot fetch plan for SQL_ID: 7ck8k47bnqpnv, CHILD_NUMBER: 1
      Please verify value of SQL_ID and CHILD_NUMBER;
      It could also be that the plan is no longer in cursor cache (check v$sql_plan)

Whenever I see this error I know that it has been raised because the CBO comes up with an execution plan that doesn’t match one of the SPM baselined plans:

select  sql_handle, plan_name, enabled, accepted
from dba_sql_plan_baselines
where signature = '7542536418437113777';

SQL_HANDLE                     PLAN_NAME                      ENA ACC
------------------------------ ------------------------------ --- ---
SQL_68ac78e9166427b1           SQL_PLAN_6jb3sx4b689xj3069e6f9 NO  YES
SQL_68ac78e9166427b1           SQL_PLAN_6jb3sx4b689xjf5f8c88e YES NO

Here we go: we have an SPM baseline with two execution plans exactly as asked by Franck

select * from table(dbms_xplan.display_sql_plan_baseline(sql_handle => 'SQL_68ac78e9166427b1'));
 
--------------------------------------------------------------------------------
SQL handle: SQL_68ac78e9166427b1
SQL text: select count(:"SYS_B_0") from t_acs where n2 = :"SYS_B_1"
--------------------------------------------------------------------------------
 
--------------------------------------------------------------------------------
Plan name: SQL_PLAN_6jb3sx4b689xjf5f8c88e         Plan id: 4126722190
Enabled: YES     Fixed: NO      Accepted: NO      Origin: AUTO-CAPTURE
Plan rows: From dictionary
--------------------------------------------------------------------------------
 
Plan hash value: 1687207741
--------------------------------------------------------------------------------
| Id  | Operation         | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |            |     1 |     3 |     3   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE   |            |     1 |     3 |            |          |
|*  2 |   INDEX RANGE SCAN| T_ACS_IDX1 |     1 |     3 |     3   (0)| 00:00:01 |
--------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   2 - access("N2"=:SYS_B_1)
 
--------------------------------------------------------------------------------
Plan name: SQL_PLAN_6jb3sx4b689xj3069e6f9         Plan id: 812246777
Enabled: NO      Fixed: NO      Accepted: YES     Origin: AUTO-CAPTURE
Plan rows: From dictionary
--------------------------------------------------------------------------------
 
Plan hash value: 535703726
----------------------------------------------------------------------------
| Id  | Operation          | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |       |     1 |     3 |   628   (3)| 00:00:01 |
|   1 |  SORT AGGREGATE    |       |     1 |     3 |            |          |
|*  2 |   TABLE ACCESS FULL| T_ACS |  1099K|  3219K|   628   (3)| 00:00:01 |
----------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("N2"=:SYS_B_1)

Next I will execute the above query with a bind variable value that will favor the index range scan plan which is present in the SPM baseline, is enabled but not accepted:

SQL> @53on
alter session set events '10053 trace name context forever, level 1';
alter session set "_optimizer_trace"=all;

select count(1) from t_acs where n2 = 100;

SQL_ID  7ck8k47bnqpnv, child number 1

Plan hash value: 1687207741
--------------------------------------------------------
| Id  | Operation         | Name       | Rows  | Bytes |
--------------------------------------------------------
|   0 | SELECT STATEMENT  |            |       |       |
|   1 |  SORT AGGREGATE   |            |     1 |     3 |
|*  2 |   INDEX RANGE SCAN| T_ACS_IDX1 |   100 |   300 |
--------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("N2"=:SYS_B_1)

SQL> @53off
alter session set events '10053 trace name context off';

The absence of the Note at the bottom of the above execution plan indicates that the CBO didn’t use any execution plan from the SPM baseline but it uses rather the execution plan it comes up with during the compilation phase. In the corresponding 10053 trace file we can read the following lines:

SPM: statement found in SMB

SPM: finding a match for the generated plan, planId = 4126722190
SPM: plan baseline non-existant or empty (no accepted plans) so using cost-based plan, planId = 4126722190

The developer at Oracle could not be clearer than this : non-existant or empty (no accepted plans) so using cost-based plan

March 5, 2018

SEMI JOIN in modern relational databases

Filed under: Oracle — hourim @ 8:17 pm

A semi join returns rows only from the first table provided at least one matching row is found in the second table. The difference between a conventional join and a semi join is that rows from the first table are returned once at most even when the join column of the second table contains duplicate rows. There is no ANSI SQL way to invoke a semi join. The following syntax doesn’t exist:

  SQL>  select 
          t1.* 
        from t1 
          semi join t2 
        on t1.n1= t2.n1

In almost all modern relational databases, the semi join logical operation is triggered via the predicate IN, ANY, SOME, EXISTS and DISTINCT. MS-SQL Server has an extra particularity which makes it using a semi join during an INTERSECT operation as we will see later in this article.

The following queries illustrate these diferent semi-join cases in a 12.2.0.1 Oracle release:

SQL> create table t1
     as
        select
            rownum n1,
            mod(rownum,5) n2,
            trunc( (rownum - 1 / 3) ) n3
        from
            dual
        connect by
            level <= 1e4; SQL> create table t2
     as
        select
            rownum n1,
            mod(rownum,3) n2,
            trunc( (rownum - 1 / 5) ) n3
        from
            dual
        connect by
        level <= 5;

1. Using IN, ANY and SOME

SQL> select 
       t1.n1
     from
       t1
     where
       t1.n1 in (select t2.n1 from t2);

SQL> select 
       t1.n1
     from
       t1
     where
      t1.n1 = any (select t2.n1 from t2);

SQL> select 
       t1.n1
     from
       t1
     where
    t1.n1 = some (select t2.n1 from t2);
   
Plan hash value: 1275841967
----------------------------------------------------------------
| Id  | Operation            | Name | Starts | E-Rows | A-Rows |
----------------------------------------------------------------
|   0 | SELECT STATEMENT     |      |      1 |        |      5 |
|*  1 |  HASH JOIN RIGHT SEMI|      |      1 |      5 |      5 |
|   2 |   TABLE ACCESS FULL  | T2   |      1 |      5 |      5 |
|   3 |   TABLE ACCESS FULL  | T1   |      1 |  10000 |  10000 |
----------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("T1"."N1"="T2"."N1")

2. Using EXISTS

select 
   t1.n1
 from
   t1
 where
   exists (select null from t2 where t2.n1 = t1.n1);

Plan hash value: 1275841967
----------------------------------------------------------------
| Id  | Operation            | Name | Starts | E-Rows | A-Rows |
----------------------------------------------------------------
|   0 | SELECT STATEMENT     |      |      1 |        |      5 |
|*  1 |  HASH JOIN RIGHT SEMI|      |      1 |      5 |      5 |
|   2 |   TABLE ACCESS FULL  | T2   |      1 |      5 |      5 |
|   3 |   TABLE ACCESS FULL  | T1   |      1 |  10000 |  10000 |
----------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("T2"."N1"="T1"."N1")

3. Using DISTINCT in INNER join and the effect of the 12cR1 Partial join

select 
  distinct
   t1.n1
 from
   t1
 inner join t2
 on (t1.n1 = t2.n1);

Plan hash value: 3703458891
---------------------------------------------------------------
| Id  | Operation           | Name | Starts | E-Rows | A-Rows |
---------------------------------------------------------------
|   0 | SELECT STATEMENT    |      |      1 |        |      5 |
|   1 |  HASH UNIQUE        |      |      1 |      5 |      5 |
|*  2 |   HASH JOIN SEMI    |      |      1 |      5 |      5 |
|   3 |    TABLE ACCESS FULL| T2   |      1 |      5 |      5 |
|   4 |    TABLE ACCESS FULL| T1   |      1 |  10000 |    256 | –- only 256 rows
---------------------------------------------------------------

Outline Data
-------------

  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('12.2.0.1')
      DB_VERSION('12.2.0.1')
      ALL_ROWS
      OUTLINE_LEAF(@"SEL$58A6D7F6")
      MERGE(@"SEL$1" >"SEL$2")
      OUTLINE(@"SEL$2")
      OUTLINE(@"SEL$1")
      FULL(@"SEL$58A6D7F6" "T2"@"SEL$1")
      FULL(@"SEL$58A6D7F6" "T1"@"SEL$1")
      LEADING(@"SEL$58A6D7F6" "T2"@"SEL$1" "T1"@"SEL$1")
      USE_HASH(@"SEL$58A6D7F6" "T1"@"SEL$1")
      USE_HASH_AGGREGATION(@"SEL$58A6D7F6")
      PARTIAL_JOIN(@"SEL$58A6D7F6" "T1"@"SEL$1")
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   2 – access("T1"."N1"="T2"."N1")

The apparition of the semi join in the above execution plan is the result of the partial join transformation introduced by Oracle as from 12cR1. Earlier versions were not able to use this partial join transformation scanning, as such, the entire content of t1 table is scanned instead of tiny 256 rows as shown below:

select 
   /*+ NO_PARTIAL_JOIN(@"SEL$58A6D7F6" "T1"@"SEL$1") */
  distinct
   t1.n1
 from
   t1
 inner join t2
 on (t1.n1 = t2.n1);

Plan hash value: 975306333

---------------------------------------------------------------
| Id  | Operation           | Name | Starts | E-Rows | A-Rows |
---------------------------------------------------------------
|   0 | SELECT STATEMENT    |      |      1 |        |      5 |
|   1 |  HASH UNIQUE        |      |      1 |      5 |      5 |
|*  2 |   HASH JOIN         |      |      1 |      5 |      5 |
|   3 |    TABLE ACCESS FULL| T2   |      1 |      5 |      5 |
|   4 |    TABLE ACCESS FULL| T1   |      1 |  10000 |  10000 | -- 10,000 rows
---------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("T1"."N1"="T2"."N1")

So far all the semi join logical operations have been executed via a HASH JOIN physical operation. This doesn’t mean that a semi joins can’t be used in a NESTED LOOPS physical operation. Here’s below the demonstratation:

SQL> create index t2_idx on t2(n1);
SQL> select 
       t1.n1
     from
       t1
     where
      t1.n1 in (select t2.n1 from t2);

----------------------------------------------------------------
| Id  | Operation          | Name   | Starts | E-Rows | A-Rows |
----------------------------------------------------------------
|   0 | SELECT STATEMENT   |        |      1 |        |      5 |
|   1 |  NESTED LOOPS SEMI |        |      1 |      5 |      5 |
|   2 |   TABLE ACCESS FULL| T1     |      1 |  10000 |  10000 |
|*  3 |   INDEX RANGE SCAN | T2_IDX |  10000 |      1 |      5 |
----------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("T1"."N1"="T2"."N1")

4. SEMI join and uniqueness
I started this article by specifying that a semi join guaranties at most one row will be returned (per join column) from the left table even if there are duplicate rows in the join column of the second table. An example being worth a thousand words so here we go:

SQL> -- insert first a duplicate row in the rigth table
SQL> insert into t2 values(5,42,42);
SQL> commit;

-- first we will start using a conventional join
SQL> select 
   t1.n1
 from
   t1
 inner join t2
 on (t1.n1 = t2.n1);
        N1
----------
         1
         2
         3
         4
         5
         5
6 rows selected

----------------------------------------------------------------
| Id  | Operation          | Name   | Starts | E-Rows | A-Rows |
----------------------------------------------------------------
|   0 | SELECT STATEMENT   |        |      1 |        |      6 |
|   1 |  NESTED LOOPS      |        |      1 |      5 |      6 |
|   2 |   TABLE ACCESS FULL| T1     |      1 |  10000 |  10000 |
|*  3 |   INDEX RANGE SCAN | T2_IDX |  10000 |      1 |      6 |
----------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 – access("T1"."N1"="T2"."N1")

-- and now a semi join
SQL> select 
       t1.n1
     from
       t1
     where
      t1.n1 in (select t2.n1 from t2);

        N1
----------
         1
         2
         3
         4
         5

5 rows selected
----------------------------------------------------------------
| Id  | Operation          | Name   | Starts | E-Rows | A-Rows |
----------------------------------------------------------------
|   0 | SELECT STATEMENT   |        |      1 |        |      5 |
|   1 |  NESTED LOOPS SEMI |        |      1 |      5 |      5 |
|   2 |   TABLE ACCESS FULL| T1     |      1 |  10000 |  10000 |
|*  3 |   INDEX RANGE SCAN | T2_IDX |  10000 |      1 |      5 |
----------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("T1"."N1"="T2"."N1")

It is for this reason that the semi join has been implemented: do not duplicate rows from the outer table when they multiple join with rows from the inner table.

Since now we know the semi join main goal we can infer that, as long as, Oracle is sure that the right table has no duplicate values in the join column, it will fall back to a simple inner join. Let’s check:

SQL> drop index t2_idx;
SQL> create unique index t2_uq_idx on t2(n1);
SQL> select
       t1.n1
     from
       t1
     where
       exists (select null from t2 where t2.n1 = t1.n1);

        N1
     ------
         1
         2
         3
         4
         5
-------------------------------------------------------------------
| Id  | Operation          | Name      | Starts | E-Rows | A-Rows |
-------------------------------------------------------------------
|   0 | SELECT STATEMENT   |           |      1 |        |      5 |
|   1 |  NESTED LOOPS      |           |      1 |      5 |      5 |
|   2 |   TABLE ACCESS FULL| T1        |      1 |  10000 |  10000 |
|*  3 |   INDEX UNIQUE SCAN| T2_UQ_IDX |  10000 |      1 |      5 |
-------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("T2"."N1"="T1"."N1")

Notice that we have no semi join anymore. Thanks to the unique index on t2(n1) table, Oracle knows that there are no duplicate rows in the join column of the outer table. It can then reliably use an inner join because rows from the outer table will at most join exactly once with rows from the inner table.

There exists another way to make Oracle falling to an inner join instead of a semi join even when there are duplicate rows in the join column of the outer table. In fact whenever Oracle uses an operation that eliminates duplicate rows from the outer table (for example sort unique or group by) it can confidently use an inner join in place of a semi join. Here’s below one case demontrated:

SQL> drop index t2_uq_idx;
SQL> insert into t2 values(5,42,42);

SQL> create unique index t1_uq_idx on t1(n1);

SQL> select
       t1.n1
     from
       t1
     where
       exists (select null from t2 where t2.n1 = t1.n1);

        N1
     ------
         1
         2
         3
         4
         5
--------------------------------------------------------------------
| Id  | Operation           | Name      | Starts | E-Rows | A-Rows |
--------------------------------------------------------------------
|   0 | SELECT STATEMENT    |           |      1 |        |      5 |
|   1 |  NESTED LOOPS       |           |      1 |      5 |      5 |
|   2 |   SORT UNIQUE       |           |      1 |      5 |      5 |
|   3 |    TABLE ACCESS FULL| T2        |      1 |      5 |      6 |
|*  4 |   INDEX UNIQUE SCAN | T1_UQ_IDX |      5 |      1 |      5 | -- 5 rows
--------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   4 - access("T2"."N1"="T1"."N1")

As you can see Oracle didn’t use a semi join because it has scanned all rows from the outer table, ordered them and eliminated duplicate rows via the SORT UNIQUE operation. As such when joining with the t1 table Oracle knows that it can join t2 rows at most exactly once. Notice in passing that while the semi join has full scanned the t1 table (10,000 rows) the inner join touched only 5 distinct index keys.

5. SEMI join and foreign key integrity

Suppose now that we want to select rows from a child table matching their parent rows in the parent table.

–- drop and recreate table t1 and t2
SQL> create index t1_idx on t1(n1);
SQL> alter table t1 add constraint t1_uq unique (n1);
SQL> alter table t2 add constraint t2_t1_fk foreign key (n1) references t1(n1);

The setup is now done. But let’s disable first the foreign key, the unique constraint and try a semi join query:

SQL> alter table t1 disable constraint t1_uq;
SQL> alter table t2 disable constraint t2_t1_fk;

SQL> select
        t2.n1
     from
        t2
     where
       exists (select null from t1 where t1.n1 = t2.n1);

-----------------------------------------------------------------------
|   Id  | Operation               | Name   | Starts | E-Rows | A-Rows |
-----------------------------------------------------------------------
|     0 | SELECT STATEMENT        |        |      1 |        |      5 |
|- *  1 |  HASH JOIN SEMI         |        |      1 |      5 |      5 |
|     2 |   NESTED LOOPS SEMI     |        |      1 |      5 |      5 |
|-    3 |    STATISTICS COLLECTOR |        |      1 |        |      5 |
|     4 |     TABLE ACCESS FULL   | T2     |      1 |      5 |      5 |
|  *  5 |    INDEX RANGE SCAN     | T1_IDX |      5 |  10000 |      5 |
|-    6 |   INDEX FAST FULL SCAN  | T1_IDX |      0 |  10000 |      0 |
-----------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("T1"."N1"="T2"."N1")
   5 - access("T1"."N1"="T2"."N1")

Note
-----
   - this is an adaptive plan (rows marked '-' are inactive)

As expected a classical semi join operation.

Let’s now enable the unique key of the parent table t1;

SQL> alter table t1 enable constraint t1_uq;

SQL> select
        t2.n1
     from
        t2
     where
       exists (select null from t1 where t1.n1 = t2.n1);

----------------------------------------------------------------
| Id  | Operation          | Name   | Starts | E-Rows | A-Rows |
----------------------------------------------------------------
|   0 | SELECT STATEMENT   |        |      1 |        |      5 |
|   1 |  NESTED LOOPS      |        |      1 |      5 |      5 |
|   2 |   TABLE ACCESS FULL| T2     |      1 |      5 |      5 |
|*  3 |   INDEX RANGE SCAN | T1_IDX |      5 |      1 |      5 |
----------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("T1"."N1"="T2"."N1")

The semi join has gone because Oracle knows that there is a unique constraint on the join column from the outer table t1, it can then safely use an inner join.

Finally let’s enable the foreign key and re-execute the semi join query:

SQL> alter table t2 enable constraint t2_t1_fk;

SQL> select
        t2.n1
     from
        t2
     where
       exists (select null from t1 where t1.n1 = t2.n1);

-------------------------------------------------------------
| Id  | Operation         | Name | Starts | E-Rows | A-Rows |
-------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |      1 |        |      5 |
|*  1 |  TABLE ACCESS FULL| T2   |      1 |      5 |      5 |
-------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("T2"."N1" IS NOT NULL)

Oracle can use a trusted foreign key relationship to remove a logical semi-join and accesses only the t2 table.

5. SEMI join and INTERSECT

In SQL Server an intersect set operation can also be done via a semi-join as the following proves:

create table t1(n1 int, n2 int,n3 int) ;
create table t2(n1 int, n2 int, n3 int) ;

insert into t1(n1,n2,n3)
select top 10000
  row_number() over (order by a.name)
 ,row_number() over (order by a.name)%5
 ,(row_number() over (order by a.name) -1) /3
from sys.all_objects a cross join sys.all_objects;

insert into t2(n1,n2,n3)
select top 5
  row_number() over (order by a.name )
 ,row_number() over (order by a.name)%3
 ,(row_number() over (order by a.name)-1)/5
from sys.all_objects a cross join sys.all_objects;

select t2.n1 from t2 intersect select t1.n1 from t1;


But neither Oracle nor PostgreSQL use a semi join when evaluating an intersect set operation as the followings prove respectively:

–- Oracle 12cR2
SQL> select t2.n1 from t2 intersect select t1.n1 from t1;

---------------------------------------------------------------
| Id  | Operation           | Name | Starts | E-Rows | A-Rows |
---------------------------------------------------------------
|   0 | SELECT STATEMENT    |      |      1 |        |      5 |
|   1 |  INTERSECTION       |      |      1 |        |      5 |
|   2 |   SORT UNIQUE       |      |      1 |      6 |      6 |
|   3 |    TABLE ACCESS FULL| T2   |      1 |      6 |      6 |
|   4 |   SORT UNIQUE       |      |      1 |  10000 |  10000 |
|   5 |    TABLE ACCESS FULL| T1   |      1 |  10000 |  10000 |
---------------------------------------------------------------
-- PostgreSQL 10.1
create table t1 as
  select 
     generate_series(1, 1e4) n1
	,generate_series(1, 1e4)%5 n2
	,(generate_series(1, 1e4)-1)/3 n3;
	
create table t2 as
  select 
     generate_series(1, 5) n1
	,generate_series(1, 5)%3 n2
	,(generate_series(1, 5)-1)/5 n3;
	
select t2.n1 from t2 intersect select t1.n1 from t1;


postgres=# explain analyze select t2.n1 from t2 intersect select t1.n1 from t1;
                                                           QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
 HashSetOp Intersect  (cost=0.00..347.00 rows=200 width=36) (actual time=13.186..13.189 rows=5 loops=1)
   ->  Append  (cost=0.00..316.90 rows=12040 width=36) (actual time=0.363..9.571 rows=10005 loops=1)
         ->  Subquery Scan on "*SELECT* 1"  (cost=0.00..55.90 rows=2040 width=36) (actual time=0.362..0.369 rows=5 loops=1)
               ->  Seq Scan on t2  (cost=0.00..30.40 rows=2040 width=4) (actual time=0.050..0.053 rows=5 loops=1)
         ->  Subquery Scan on "*SELECT* 2"  (cost=0.00..261.00 rows=10000 width=9) (actual time=0.031..7.743 rows=10000 loops=1)
               ->  Seq Scan on t1  (cost=0.00..161.00 rows=10000 width=5) (actual time=0.029..4.478 rows=10000 loops=1)
 Planning time: 0.555 ms
 Execution time: 13.911 ms

6. SUMMARY

We have seen in this article how a semi join logical operation is triggered when using IN, ANY, SOME, EXISTS and DISTINCT in several modern relational databases. We’ve outlined as well that, in contrast to Oracle and PostgreSQL, SQL-Server can use a semi join during an INTERSECT set operation. We’ve demonstrated that while a regular joins can duplicate rows, semi joins do not produce any duplicates. They are defined to return (or not) rows only from one table of the join. We’ve also shown that, as long as the CBO is sure that the right table has no duplicate values in the join column, it will fall back to a simple inner join where extra transformations become possible and where the left table might not be fully scanned.

March 1, 2018

Diagnosing the past : CPU bound

Filed under: Oracle — hourim @ 6:24 pm

Few weeks ago I have been confronted to a weird real life system performance issue which, I believe is worth a sharing blog post. It started by a customer complaint about a dramatic slow in inserts of commands and products so that the application in its entirety appeared frozen. A very quick look-up over ASH shows no particular wait events or any new TOP SQL that I am not aware of. I know this application to such an extent that I can recognize its top queries via their sql_id. This is why my first impression was that there is no particular unusual wait event to point out except a high CPU consumption.

But the Load Profile of the corresponding one hour AWR report looks very odd:

Load Profile                    Per Second   Per Transaction  Per Exec  Per Call
~~~~~~~~~~~~~~~            ---------------   --------------- --------- ---------
             DB Time(s):              95.2               4.7      0.02     0.02
              DB CPU(s):               7.7               0.1      0.00     0.00
      Background CPU(s):               0.2               0.0      0.00     0.00
      Redo size (bytes):         439,216.3          18,666.4
  Logical read (blocks):       1,774,174.9          75,401,1
          Block changes:           2,465.0             104.8
 Physical read (blocks):           3,609.2             153.4
Physical write (blocks):             135.2               5.8
       Read IO requests:           2,436.8             103.6
      Write IO requests:              53.6               2.3
           Read IO (MB):              28.2               1.2
          Write IO (MB):               1.1               0.0
           IM scan rows:               0.0               0.0
Session Logical Read IM:               
             User calls:           5,423.4             230.5
           Parses (SQL):             244.4              10.4
      Hard parses (SQL):               0.5               0.0
     SQL Work Area (MB):              28.3               1.2
                 Logons:               0.7               0.0
         Executes (SQL):           4,470.6             190.0
              Rollbacks:               0.2               0.0
           Transactions:              23.5

Have you ever seen such a dramatic Load Profile? The foreground sessions spend collectively 95 seconds of time for each wall clock second. All those sessions were either actively working or waiting. They burned 7.7 of CPU per wall clock second. For a machine of 24 cores this seems to be a CPU constrained application. The overall workload is generating 1,77 million of logical blocks read per second which represent a rate of approximatively 6GB of logical blocks read per hour. And only a tiny percentage of those block reads are changed (2,465/1,774,174 = 0,13%).

I was really intrigued by this abnormal situation since I didn’t spot any unusual wait event or SQL query. While I was drilling down the different sections of the AWR report, cross-checking properly the performance key indicators they deliver, something in the TOP 10 Foreground Events captured my attention:

Top 10 Foreground Events by Total Wait Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                           Total Wait      Avg   % DB    Wait
Event                                Waits Time (sec)    Wait(ms)  time Class
------------------------------ ----------- ---------- --------- ------ --------
db file sequential read          2,813,888        32K     11.37    9.3 User I/O
DB CPU                                          27,9K              8.1
db file parallel read              278,494      10,8K     38.67    3.1 User I/O
read by other session              444,473       4785     10.77    1.4 User I/O
direct path read                   311,471     4718,6     15.15    1.4 User I/O
gc cr block 2-way                  577,081     2366,4      4.10     .7 Cluster
log file sync                       89,040     2254,7     25.32     .7 Commit
db file scattered read              138,798    2136,8     15.40     .6 User I/O
gc current block 2-way              368,907    1994,1      5.41     .6 Cluster
gc buffer busy acquire              329,900    1776,8      5.39     .5 Cluster
  • The % DB time column doesn’t sum up to 100% (or near this number) totalling only 26,4%
  • The disk is very slow servicing the physical I/O and background events with a dramatic average time wait.

When the %DB time column doesn’t add up to nearly 100% it generally indicates a symptom of a CPU bounded application. Indeed this is confirmed a little bit upward in the same AWR report via the HOST CPU section:

Host CPU
~~~~~~~~                  Load Average
 CPUs Cores Sockets     Begin       End     %User   %System  %WIO  %Idle
----- ----- ------- --------- --------- --------- --------- ------ -----
   48    24       4     55.57   219.13       92.0       2.3    2.3   5.0

The application was already over CPU bounded at the beginning (55) and completely frozen at the end of the load (219 CPU). For a machine with 24 cores, the DB CPU load is indeed very high

 DB CPU Load = DB CPU time/Elapsed Time
            = 27900/3600 = 7.7

And so is the DB time load


              Snap Id      Snap Time      Sessions Curs/Sess  Instance
            --------- ------------------- -------- --------- ---------
Begin Snap:     28812 06-Feb-18 13:00:01       365      12.5         2
  End Snap:     28813 06-Feb-18 14:00:15       421      12.5         2
   Elapsed:               60.23 (mins)
   DB Time:            5,734.18 (mins)

DB time Load = DB time/Elapsed Time
              = 5734.1/60.23 = 95.2

So what else?

I asked the storage team to check the performance of the underlying disk. The confirmation of the dramatic disk latency has been obviously confirmed by the storage team. But the fundamental and legitimate question is what has changed per regards to the yesterday normal functioning of the application that might explain this performance deterioration?

While I was trying to figure out a solution I asked a help from a DBA who is the primary owner of another critical application. To my surprise this DBA told me that one of his applications is also suffering from similar performance deterioration and that this application is hosted by the same server as the one hosting the application I am trying to troubleshoot. What a coincidence.

A couple of minutes later we found that a critical batch has been triggered in the other DBA application. This batch was extensively executing a parallel query using a very high degree of parallelism (DOP) which was consuming the whole SAN bandwidth. Needless to say that when this batch job has been stopped the initial problem I was trying to fix vanishes by itself and the database returned to its normal functioning

Bottom Line

When you are troubleshooting an application which seems to be completely frozen and where the disk is very badly servicing the database and where the DB time % column of the AWR Top 10 foreground Events section is not summing up to 100% then you are probably CPU bounded. In such case, don’t forget to consider, among the panoply of your check list, whether others applications hosted in the same server are not stressing the disk with very aggressive parallel queries.

January 27, 2018

Adaptive Cursor Sharing and parallel execution plan

Filed under: Oracle — hourim @ 7:33 am

I have shown and demonstrated at different places that there exist three pre-requisites for a cursor, using natural or forced bind variable, to be bind sensitive:

  • it uses a range (inequality) predicate
  • it uses an equality predicate and histogram
  • it uses a partition key in its predicate

I think that from now on we can add a fourth pre-requisite which is:

  • the very first execution of this cursor should not be run via a parallel execution plan.

Let’s demonstrate this point (the model should be taken from this post):

alter system flush shared_pool;
alter session set cursor_sharing= force;
alter table t_acs parallel;

select count(1) from t_acs where n2 = 1;

SQL_ID  7ck8k47bnqpnv, child number 0
-------------------------------------
------------------------------------------------
| Id  | Operation         | Name       | Rows  |
------------------------------------------------
|   0 | SELECT STATEMENT  |            |       |
|   1 |  SORT AGGREGATE   |            |     1 |
|*  2 |   INDEX RANGE SCAN| T_ACS_IDX1 |     1 |
------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("N2"=:SYS_B_1)

select
    child_number
   ,is_bind_sensitive
   ,is_bind_aware
 from 
   gv$sql
 where
    sql_id ='7ck8k47bnqpnv';

CHILD_NUMBER I I
------------ - -
           0 Y N

The very first execution of the above cursor used a serial execution plan and an equality predicate on a skewed column having Frequency histogram. This is why it has been marked bind sensitive.

But what would have happened to the cursor if its very first execution has been ran parallely?

alter system flush shared_pool;
select count(1) from t_acs where n2 = 1e6;
-----------------------------------------------------------
| Id  | Operation              | Name     | Rows  | Bytes |
-----------------------------------------------------------
|   0 | SELECT STATEMENT       |          |       |       |
|   1 |  SORT AGGREGATE        |          |     1 |     3 |
|   2 |   PX COORDINATOR       |          |       |       |
|   3 |    PX SEND QC (RANDOM) | :TQ10000 |     1 |     3 |
|   4 |     SORT AGGREGATE     |          |     1 |     3 |
|   5 |      PX BLOCK ITERATOR |          |  1099K|  3220K|
|*  6 |       TABLE ACCESS FULL| T_ACS    |  1099K|  3220K|
-----------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   6 - access(:Z>=:Z AND :Z<=:Z)
       filter("N2"=:SYS_B_1)

Note
-----
   - Degree of Parallelism is 8 because of table property

select
    child_number
   ,is_bind_sensitive
   ,is_bind_aware
 from 
   gv$sql
 where
    sql_id ='7ck8k47bnqpnv';

CHILD_NUMBER I I
------------ - -
           0 N N

Since the very first execution of the cursor uses a parallel execution plan Oracle refuses to set its bind sensitive property. And from this stage and on, until a cursor is flushed out from the library cache, all cursor executions will share the same parallel execution plan.

But what would have happened if the very first cursor execution would have used a serial plan?

alter system flush shared_pool;

select count(1) from t_acs where n2 = 100;
select count(1) from t_acs where n2 = 100;
select count(1) from t_acs where n2 = 100;
------------------------------------------------
| Id  | Operation         | Name       | Rows  |
------------------------------------------------
|   0 | SELECT STATEMENT  |            |       |
|   1 |  SORT AGGREGATE   |            |     1 |
|*  2 |   INDEX RANGE SCAN| T_ACS_IDX1 |   100 |
------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 – access("N2"=:SYS_B_1)

select
    child_number
   ,is_bind_sensitive
   ,is_bind_aware
   ,executions
 from 
   gv$sql
 where
    sql_id ='7ck8k47bnqpnv';

CHILD_NUMBER I I EXECUTIONS
------------ - - ----------
           0 Y N          3

As expected the cursor is bind sensitive. Let’s now make it bind aware :

select count(1) from t_acs where n2 = 1000;
select count(1) from t_acs where n2 = 1000;
select count(1) from t_acs where n2 = 1000;
select count(1) from t_acs where n2 = 1000;

select
    child_number
   ,is_bind_sensitive
   ,is_bind_aware
   ,executions
 from 
   gv$sql
 where
    sql_id ='7ck8k47bnqpnv';

CHILD_NUMBER I I EXECUTIONS
------------ - - ----------
           0 Y N          6
           1 Y Y          1

And here where the serious stuff starts. I will show you how Oracle will unset the bind sensitive and bind awareness property of the above cursor whenever the execution plan triggered by the ECS layer code and produced by CBO is a parallel plan:

select count(1) from t_acs where n2 = 1e6;

SQL_ID  7ck8k47bnqpnv, child number 2
-------------------------------------
-----------------------------------------------------------
| Id  | Operation              | Name     | Rows  | Bytes |
-----------------------------------------------------------
|   0 | SELECT STATEMENT       |          |       |       |
|   1 |  SORT AGGREGATE        |          |     1 |     3 |
|   2 |   PX COORDINATOR       |          |       |       |
|   3 |    PX SEND QC (RANDOM) | :TQ10000 |     1 |     3 |
|   4 |     SORT AGGREGATE     |          |     1 |     3 |
|   5 |      PX BLOCK ITERATOR |          |  1099K|  3219K|
|*  6 |       TABLE ACCESS FULL| T_ACS    |  1099K|  3219K|
-----------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   6 - access(:Z>=:Z AND :Z<=:Z)
       filter("N2"=:SYS_B_1)

Note
-----
   - Degree of Parallelism is 8 because of table property

select
    child_number
   ,is_bind_sensitive
   ,is_bind_aware
   ,executions
 from 
   gv$sql
 where
    sql_id ='7ck8k47bnqpnv';

CHILD_NUMBER I I EXECUTIONS
------------ - - ----------
           0 Y N          6
           1 Y Y          1
           2 N N          1

The new child number 2 is not anymore bind sensistive nor bind aware. That’s the most important conclusion of this article : Oracle doesn’t allow ACS to work with parallelism.

Summary

ACS has been implemented for very frequently used queries having different set of bind variables values each of which generating different amount of I/O. In this context, Oracle decided to cancel ACS whenever a parallel plan is triggred by ECS. Before 12cR2 there was a bug identified by Bug 16226575 in which ACS was disabled for query having decorated parallel object irrespective of the plan being chosen serial or parallel. As we saw in this article this has been fixed in 12cR2. ACS will be disabled only if its produced execution plan is parallel.

January 20, 2018

NESTED LOOP and full/right outer join in modern RDBMS

Filed under: CBO,explain plan — hourim @ 11:36 am

The Oracle Cost Based Optimizer, the MS-SQL Server optimizer and the PostgreSQL query planner cannot use a NESTED LOOP physical operation to execute FULL OUTER and RIGHT OUTER joins logical operations. They all address the RIGHT OUTER join limitation by switching the inner and the outer row source so that a LEFT OUTER JOIN can be used. While the first two optimizer turn the FULL OUTER join into a LEFT OUTER join concatenated with an ANTI-join, PostgreSQL query planner will always use a HASH/MERGE JOIN to do a FULL OUTER join.

Let’s make this less confusing by starting with the basics. The algorithm of a NESTED LOOP physical operation is:

for each row ro in the outer row source 
loop
   for each row ri in the inner row source
   loop
     if ro joins ri then return current row
   end loop
end loop

1.Oracle 12cR2

A simple execution of the above algorithm can be schematically represented via the following Oracle execution plan:

select /*+ use_nl(t1 t2) */ 
      t1.*
from t1 inner join t2
on t1.n1 = t2.n1;

----------------------------------------------------------------
| Id  | Operation          | Name   | Starts | E-Rows | A-Rows |
----------------------------------------------------------------
|   0 | SELECT STATEMENT   |        |      1 |        |      3 |
|   1 |  NESTED LOOPS      |        |      1 |      4 |      3 |
|   2 |   TABLE ACCESS FULL| T1     |      1 |      3 |      3 |
|*  3 |   INDEX RANGE SCAN | T2_IDX |      3 |      1 |      3 |
----------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 – access("T1"."N1"="T2"."N1")

As you can see, in accordance with the algorithm, for each row in T1 table (A-Rows=3 operation id n°2) we scanned 3 times (Starts = 3 operation id n°3) the T2_IDX index.

Let’s now try a FULL OUTER join but without any hint:

select  
      t1.*
from t1 full outer join t2
on t1.n1 = t2.n1;

---------------------------------------------------
| Id  | Operation             | Name     | Rows  |
---------------------------------------------------
|   0 | SELECT STATEMENT      |          |       |
|   1 |  VIEW                 | VW_FOJ_0 |     4 |
|*  2 |   HASH JOIN FULL OUTER|          |     4 |
|   3 |    TABLE ACCESS FULL  | T1       |     3 |
|   4 |    TABLE ACCESS FULL  | T2       |     4 |
---------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("T1"."N1"="T2"."N1")

So far so good. A HASH JOIN FULL OUTER to honor a full outer join between two tables.

But what if I want to use a NESTED LOOP FULL OUTER instead of HASH JOIN FULL OUTER join ?

select  /*+ use_nl(t1 t2) */
      t1.*
from t1 FULL outer join t2
on t1.n1 = t2.n1;

-------------------------------------------------------
| Id  | Operation            | Name   | Rows  | Bytes |
-------------------------------------------------------
|   0 | SELECT STATEMENT     |        |       |       |
|   1 |  VIEW                |        |     6 |   120 |
|   2 |   UNION-ALL          |        |       |       |
|   3 |    NESTED LOOPS OUTER|        |     4 |    40 |
|   4 |     TABLE ACCESS FULL| T1     |     3 |    21 |
|*  5 |     INDEX RANGE SCAN | T2_IDX |     1 |     3 |
|   6 |    NESTED LOOPS ANTI |        |     2 |    12 |
|   7 |     TABLE ACCESS FULL| T2     |     4 |    12 |
|*  8 |     TABLE ACCESS FULL| T1     |     2 |     6 |
-------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   5 - access("T1"."N1"="T2"."N1")
   8 - filter("T1"."N1"="T2"."N1")

What the heck is this execution plan of 8 operations?

Instead of having a simple NESTED LOOP FULL OUTER I got a concatenation of NESTED LOOPS OUTER and a NESTED LOOPS ANTI join.That’s an interesting transformation operated by the CBO.

Should I have tried to reverse engineer the query that sits behind the above execution plan I would have very probably obtained the following query:

select  
      t1.*
from 
      t1 
     ,t2 
where t1.n1 = t2.n1(+)
union all
select 
     t2.*
from t2
where not exists (select /*+ use_nl(t2 t1) */ 
                      null 
                  from t1 
                  where t1.n1 = t2.n1);

------------------------------------------------------
| Id  | Operation           | Name   | Rows  | Bytes |
------------------------------------------------------
|   0 | SELECT STATEMENT    |        |       |       |
|   1 |  UNION-ALL          |        |       |       |
|   2 |   NESTED LOOPS OUTER|        |     4 |    40 |
|   3 |    TABLE ACCESS FULL| T1     |     3 |    21 |
|*  4 |    INDEX RANGE SCAN | T2_IDX |     1 |     3 |
|   5 |   NESTED LOOPS ANTI |        |     2 |    20 |
|   6 |    TABLE ACCESS FULL| T2     |     4 |    28 |
|*  7 |    TABLE ACCESS FULL| T1     |     2 |     6 |
------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   4 - access("T1"."N1"="T2"."N1")
   7 - filter("T1"."N1"="T2"."N1")

In fact when I have directed Oracle to use a NESTED LOOP to FULL OUTER JOIN T1 and T2 it has turned out my instruction into:

 T1 LEFT OUTER JOIN T2 UNION ALL T2 ANTI JOIN T1

Which is nothing else than :

  • select all rows from T1 and T2 provided they join
  • add to these rows, rows from T1 that don’t join (LEFT OUTER)
  • add to these rows, all rows from T2 that don’t join (ANTI) with rows from T1

Do you know why Oracle did all this somehow complicated gymnastic?

It did it because I asked it to do an impossible operation: NESTED LOOP doesn’t support FULL OUTER join.

It doesn’t support RIGHT OUTER join as well as shown below:

select  /*+ use_nl(t1 t2) */
      t1.*
from t1 
RIGHT outer join t2
on t1.n1 = t2.n1;

---------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes |
---------------------------------------------------
|   0 | SELECT STATEMENT   |      |       |       |
|   1 |  NESTED LOOPS OUTER|      |     4 |    40 |
|   2 |   TABLE ACCESS FULL| T2   |     4 |    12 |
|*  3 |   TABLE ACCESS FULL| T1   |     1 |     7 |
---------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - filter("T1"."N1"="T2"."N1")

Don’t be confused here. The above RIGHT OUTER JOIN has been turned into a LEFT OUTER JOIN by switching the inner and the outer table. As such, T2 being placed in the left side of the join Oracle is able to use a NESTED LOOP to operate a LEFT OUTER JOIN. You will see this clearly explained in the corresponding SQL Server execution plan I will show later in this article.

2. PostgreSQL 10.1

Since there are no hints in PostgreSQL to make a join using a NESTED LOOP I will start by cancelling hash and merge join operations as shown below:

postgres=# set enable_mergejoin=false;
SET
postgres=# set enable_hashjoin=false;
SET

And now I am ready to show you how the PostgreSQL query planner turns a right outer join into a left outer join when a NESTED LOOP operation is used:

postgres=# explain
postgres-# select
postgres-#       t1.*
postgres-# from t1 right outer join t2
postgres-# on t1.n1 = t2.n1;

                            QUERY PLAN
-------------------------------------------------------------------
 Nested Loop Left Join  (cost=0.00..95.14 rows=23 width=42)
   Join Filter: (t1.n1 = t2.n1)
   ->  Seq Scan on t2  (cost=0.00..1.04 rows=4 width=4)
   ->  Materialize  (cost=0.00..27.40 rows=1160 width=42)
         ->  Seq Scan on t1  (cost=0.00..21.60 rows=1160 width=42)
(5 lignes)

However, in contrast to Oracle and MS-SQL Server, PostgreSQL query planner is unable to transform a full outer join into a combination of an NESTED LOOP LEFT OUTER join and an ANTI-join as the following demonstrates:

explain 
select
      t1.*
from t1 full outer join t2
on t1.n1 = t2.n1;
                                QUERY PLAN
--------------------------------------------------------------------------
 Hash Full Join  (cost=10000000001.09..10000000027.27 rows=1160 width=42)
   Hash Cond: (t1.n1 = t2.n1)
   ->  Seq Scan on t1  (cost=0.00..21.60 rows=1160 width=42)
   ->  Hash  (cost=1.04..1.04 rows=4 width=4)
         ->  Seq Scan on t2  (cost=0.00..1.04 rows=4 width=4)

Spot in passing how disabling the hash join option (set enable_hashjoin=false) is not an irreversible action. Whenever the query planner is unable to find another way to accomplish its work it will use all option available even those being explicitely disabled.

3. MS-SQL Server 2016



4. Summary
In several if not all modern Relational DataBase Management Systems, NESTED LOOP operation doesn’t support right outer and full outer join. Oracle, MS-SQL Server and PostgreSQL turn “T1 right outer join T2” into “T2 left outer join T1” by switching the inner and the outer row source. Oracle and SQL Server turn a full outer join between T1 and T2 into a T1 left outer join T2 union-all T2 anti-join T1. PostgreSQL will always use a hash/merge to full outer join T1 and T2.

Model

--@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
-- NESTED LOOP and full/right outer join : Oracle
--@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
drop table t1;
drop table t2;
create table t1(t1_id int , t1_vc varchar2(10));

insert into t1 values (1, 't1x');
insert into t1 values (2, 't1y');
insert into t1 values (3, 't1z');
 
create index t1_idx on t1(t1_id);

create table t2 (t2_id int, t2_vc varchar(10));

insert into t2 values (2, 't2x');
insert into t2 values (3, 't2y');
insert into t2 values (3, 't2yy');
insert into t2 values (4, 't2z');

create index t2_idx on t2(t2_id);

December 28, 2017

Index statistics

Filed under: Statistics — hourim @ 8:06 pm

There are so many situations in the Oracle database world that I would have probably never encountered if there hadn’t been so many different real life applications with different problems that needed solving. A couple of days ago, a critical query of one of such applications, experienced severe performance deterioration. When I started looking at this query I’ve first decided to print out the statistics of its historical executions shown below:

SQL> @HistStats
Enter value for sql_id: gqj0qgtduh1k7
Enter value for from_date: 01122017

SNAP_BEGIN                PLAN_HASH_VALUE      EXECS  AVG_ETIME    AVG_PIO    AVG_LIO   AVG_ROWS
------------------------- --------------- ---------- ---------- ---------- ---------- ----------
01/12/17 12:00:44.897 AM        179157465        178          4        357     244223        543
01/12/17 01:00:50.192 AM        179157465        40           6       3976     246934        400
01/12/17 04:00:02.121 AM        179157465         2           2        171     246203        400
../..
03/12/17 05:00:22.064 AM        179157465        10           18      4231     245802        400
../..
04/12/17 09:00:19.716 AM        179157465        13           1        4       253433        482

The most important characteristic of this query is that it consumes 250K logical I/O per execution and that it can sometime last up to 18 seconds. This is a query generated by Hibernate. Unfortunately, due to the Hibernate obsession of changing table aliases, this query has got a new sql_id when it has been deployed in production via the December release. This is why we lost the track of the query pre-release statistics that would have allowed us to get the previous acceptable execution plan.

Starting from the end
Let’s, exceptionally, start this article by showing the statistics of this query after I have fixed it

SQL> @HistStats
Enter value for sql_id: gqj0qgtduh1k7
Enter value for from_date: 01122017

SNAP_BEGIN                PLAN_HASH_VALUE      EXECS  AVG_ETIME    AVG_PIO    AVG_LIO   AVG_ROWS
------------------------- --------------- ---------- ---------- ---------- ---------- ----------
01/12/17 12:00:44.897 AM        179157465        178          4        357     244223        543
01/12/17 01:00:50.192 AM        179157465        40           6       3976     246934        400
01/12/17 04:00:02.121 AM        179157465         2           2        171     246203        400
../..
03/12/17 05:00:22.064 AM        179157465        10           18      4231     245802        400
../..
04/12/17 09:00:19.716 AM        179157465        13           1        4       253433        482
../..
11/12/17 02:00:04.453 PM        1584349102        4           0        14       17825        482 –- fixed
12/12/17 12:00:38.622 AM        1584349102       67           1        661      13840        522 
12/12/17 01:00:42.740 AM        1584349102       40           0        103       7605        852 
13/12/17 05:00:27.270 AM        1584349102       10           1        845      16832        400 

Notice how, thanks to the fix I will explain in the next section, I have got a remarkable drop of the number of Logical I/O consumption from 244K to 17K per execution on average.

How this has been achieved?
The real time sql monitoring report of the degraded query confirms the bad response time of 12 seconds:

Global Information
------------------------------
 Status              :  DONE (ALL ROWS)                 
 Instance ID         :  2                    
 Session             :  xxxx (324:25737) 
 SQL ID              :  gqj0qgtduh1k7        
 SQL Execution ID    :  16777216             
 Execution Started   :  12/05/2017 06:01:20  
 First Refresh Time  :  12/05/2017 06:01:29  
 Last Refresh Time   :  12/05/2017 06:01:32  
 Duration            :  12s                  
 Module/Action       :  JDBC Thin Client/-           
 Service             :  xxxxx            
 Program             :  JDBC Thin Client/
 Fetch Calls         :  10

This report presents an oddity in the following part of its 54 execution plan operations:

---------------------------------------------------------------------------------------
| Id  | Operation                           | Name         | Starts | E-Rows | A-Rows |
---------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                    |              |      1 |        |    398 |
|* 39 |  TABLE ACCESS BY INDEX ROWID BATCHED| T1           |      1 |     26 |    398 |
|* 40 |   INDEX RANGE SCAN                  | IDX_PRD_TYPE |      1 |     2M |     3M |
---------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
  39 - filter(("PRODUCT_ID"=:1 OR "PRODUCT_ID"=:2 OR "PRODUCT_ID"=:3 OR 
        "PRODUCT_ID"=:4 ../..  OR "PRODUCT_ID"=:398))
  40- access("PRODUCT_TYPE"='M'

This oddity resides in the fact that we went from 3M of rowids supplied by the index at operation 40 to only 398 rows at the table level. Such a situation can have one of the following explanations:

  • The CBO has made a very bad choice preferring the IDX_PRD_TYPE index over another existing index with a much better index effective selectivity.
  • IDX_PRD_TYPE is the sole index with which Oracle can cover the PRODUCT_ID and PRODUCT_TYPE predicates. And, in this case, it represents an imprecise index that should be re-designed if we want to efficiently cover the above query

The answer is straightforward If you remember that a column having an _ID suffix or prefix in its name refers generally to a primary key. The T1 table has indeed a primary key index defined as follows:

 T1_PK (PRODUCT_ID) 

In addition the 398 rows returned at the table level are nothing else than the 398 elements constituting the in-list of the corresponding query reproduced here below:

SELECT 
  *
FROM
   t1
WHERE 
   PRODUCT_ID 
       in (:1,:2,:3,:4,:5,....,:398)
AND PRODUCT_TYPE = 'M';

Table altered.

While the bad selected index has the following definition:

 IDX_PRD_TYPE(PRODUCT_TYPE,COLX)

And the initial question turned to : why the primary key index has not been used in this critical part of the executio plan?

At this stage of the investigations it becomes crystal clear for me that using the primary key index is, to a very large extent, better that using the index on the PRODUCT_TYPE column. This is simply because, if used by the CBO, the primary key index will supply its parent table operation by at most one rowid per element in the in-list predicate. That is for an in-list of 398 elements Oracle will only filter 398 rows at the table level using the PRODUCT_TYPE = ‘M’ filter. That’s largely better than filtering 3M of rows at the table level using the 3M of index rowids supplied by the IDX_PRD_TYPE(PRODUCT_TYPE,COLX) index.

Bear in mind as well that all parameters participating in the index desirability are at their default values. And that the PRODUCT_TYPE column has a remarkable frequency histogram showing perfectly the not evenly distribution of its 13 distinct values.

select 
   column_name
  ,histogram 
from 
  user_tab_col_statistics 
where 
  table_name = 'T1'
and
  column_name = 'PRODUCT_TYPE';

COLUMN_NAME  HISTOGRAM
------------ ---------------
PRODUCT_TYPE FREQUENCY

The explain plan for command applied when the primary key index is forced and when not used shows that the combined table/index access cost is cheaper when the IDX_PRD_TYPE index is used as the following proves:

-------------------------------------------------------------------
| Id  | Operation                    | Name  | Rows  | Cost (%CPU)|
-------------------------------------------------------------------
|   0 | SELECT STATEMENT             |       |    27 |   519   (1)|
|   1 |  INLIST ITERATOR             |       |       |            |
|*  2 |   TABLE ACCESS BY INDEX ROWID| T1    |    27 |   519   (1)|
|*  3 |    INDEX UNIQUE SCAN         | T1_PK |   398 |   399   (0)|
-------------------------------------------------------------------

--------------------------------------------------------------------------------
| Id  | Operation                           | Name         | Rows  |Cost (%CPU)|
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                    |              |    27 |  58   (0)|
|*  1 |  TABLE ACCESS BY INDEX ROWID BATCHED| T1           |    27 |  58   (0)|
|*  2 |   INDEX RANGE SCAN                  | IDX_PRD_TYPE |  2183K|   3   (0)|
--------------------------------------------------------------------------------

And the initial question turned to : why the cost of the inapropriate index is cheaper?

After few minutes of reflection I decided to get the statistics of these two indexes:

select
   index_name
  ,distinct_keys
  ,leaf_blocks
from
   user_indexes
where
  table_name = 'T1';
 
INDEX_NAME       DISTINCT_KEYS LEAF_BLOCKS
---------------- ------------- -----------
T1_PK                 32633730    111089
IDX_PRD_TYPE

And finally the initial question turned to be: why the IDX_PRD_TYPE index has no statistics?

The answer is:

  • The statistics of T1 table have been voluntarily locked since several months ago
  • The IDX_PRD_TYPE index has been very recently created and implemented in Production

The combination of the above two points resulted in a newly created index without any statistics making this index appearing very cheap in the CBO eyes and consequently leading to a very bad choice and to a noticeable performance issue.

Obviously gathering the statistics of this index makes it less desirable in favour of the primary key index as the following proves:

exec dbms_stats.unlock_table_stats(user, 't1');
begin
 dbms_stats.gather_index_stats(user, 'IDX_PRD_TYPE', no_invalidate =>false);
end;
/
exec dbms_stats.unlock_table_stats(user, 't1');

-------------------------------------------------------------------
| Id  | Operation                    | Name  | Rows  | Cost (%CPU)|
-------------------------------------------------------------------
|   0 | SELECT STATEMENT             |       |    27 |   519   (1)|
|   1 |  INLIST ITERATOR             |       |       |            |
|*  2 |   TABLE ACCESS BY INDEX ROWID| T1    |    27 |   519   (1)|
|*  3 |    INDEX UNIQUE SCAN         | T1_PK |   398 |   399   (0)|
-------------------------------------------------------------------

--------------------------------------------------------------------------------
| Id  | Operation                           | Name         | Rows  |Cost (%CPU)|
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                    |              |    27 |  137K   (2|
|*  1 |  TABLE ACCESS BY INDEX ROWID BATCHED| T1           |    27 |  137K   (2)|
|*  2 |   INDEX RANGE SCAN                  | IDX_PRD_TYPE |  2183K| 5678    (1)|
--------------------------------------------------------------------------------

Bottom Line

All things being equal when you decide to lock the statistics of one your tables then bear in mind that any newly created index on this table will not have any statistics. This index will the be, naturally, much more desirable than any other index created before the lock of the table statistics resulting, fairlly likely, into wrong index choice and bad execution plans. If you are in a similar situation you had then better to manually gather statistics of your new indexes as long as the statistics of your table are kept locked.

December 22, 2017

Null-Accepting Semi-Join

Filed under: CBO — hourim @ 11:15 am

Introduction

Null-Accepting semi-join is a new enhancement brought to the CBO by the 12cR1 release. It extends the semi-join algorithm to qualify rows from the table in the left hand side that have a null value in the join column.It kicks in for queries like the following one:

SELECT  
   count(1)
FROM t1
   WHERE(t1.n1 is null
         OR exists (SELECT null FROM t2 where t2.n1 = t1.n1)
		);

It is recognisable in execution plans via its acronym NA (Null-Accepting)

----------------------------------------------------
| Id  | Operation           | Name | Rows  | Bytes |
----------------------------------------------------
|   0 | SELECT STATEMENT    |      |       |       |
|   1 |  SORT AGGREGATE     |      |     1 |     6 |
|*  2 |   HASH JOIN SEMI NA |      |     7 |    42 |
|   3 |    TABLE ACCESS FULL| T1   |    10 |    30 |
|   4 |    TABLE ACCESS FULL| T2   |    10 |    30 |
----------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("T2"."N1"="T1"."N1")

It appears in the CBO 10053 trace file as (for its HASH JOIN version):

 Best:: JoinMethod: HashNullAcceptingSemi

Don’t get confused by the NA acronym that appears in the ANTI-JOIN operation where it refers to Null-Aware rather than to Null-Accepting as shown in the following execution plan and 10053 trace file respectively:

SELECT  
   count(1)
FROM t1
   WHERE t1.n1 NOT IN (select n1 from t2);

----------------------------------------------------
| Id  | Operation           | Name | Rows  | Bytes |
----------------------------------------------------
|   0 | SELECT STATEMENT    |      |       |       |
|   1 |  SORT AGGREGATE     |      |     1 |     6 |
|*  2 |   HASH JOIN ANTI NA |      |     1 |     6 |
|   3 |    TABLE ACCESS FULL| T1   |    10 |    30 |
|   4 |    TABLE ACCESS FULL| T2   |    10 |    30 |
----------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 – access("T1"."N1"="N1")

 Best:: JoinMethod: HashNullAwareAnti

If you want to clear this confusion then remember that the Null-Accepting transformation occurs for rows that (SEMI) join while Null-Aware transformation is for rows that would not join (ANTI).

The semi-join Null-Accepting logical operation can also be serviced by the NESTED LOOP physical operation as the following demonstrates:

SELECT  
   count(1)
FROM t1
   WHERE(t1.n1 is null
         OR exists (SELECT /*+ NL_SJ */ null FROM t2 where t2.n1 = t1.n1)
		); 

------------------------------------------------------
| Id  | Operation             | Name | Rows  | Bytes |
------------------------------------------------------
|   0 | SELECT STATEMENT      |      |       |       |
|   1 |  SORT AGGREGATE       |      |     1 |     6 |
|   2 |   NESTED LOOPS SEMI NA|      |     7 |    42 |
|   3 |    TABLE ACCESS FULL  | T1   |    10 |    30 |
|*  4 |    TABLE ACCESS FULL  | T2   |     7 |    21 |
------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   4 – filter("T2"."N1"="T1"."N1")

Best:: JoinMethod: NestedLoopNullAcceptingSemi

It is the ability, acquired by the CBO as from 12cR1, to unnest the above kind of disjunctive subquery that makes the Null-Accepting transformation possible as shown in the corresponding10053 trace file:

*****************************
Cost-Based Subquery Unnesting
*****************************
SU: Unnesting query blocks in query block SEL$1 (#1) that are valid to unnest.
Subquery Unnesting on query block SEL$1 (#1)
SU: Performing unnesting that does not require costing.
SU: Considering subquery unnest on query block SEL$1 (#1).
SU: Checking validity of unnesting subquery SEL$2 (#2)
SU: Transforming EXISTS subquery to a join.
Registered qb: SEL$5DA710D3 0x1d225e60 (SUBQUERY UNNEST SEL$1; SEL$2)

Prior to 12cR1 it was not possible to automatically unnest the above subquery to join it with its parent block leading to the below execution plan where the inner table T2 is scanned mutliple times:

SELECT /*+ gather_plan_statistics */ 
   count(1)
FROM t1
   WHERE(t1.n1 is null
         OR exists (SELECT /*+ no_unnest */ null FROM t2 where t2.n1 = t1.n1)
        );  
---------------------------------------------------------------
| Id  | Operation           | Name | Starts | E-Rows | A-Rows |
---------------------------------------------------------------
|   0 | SELECT STATEMENT    |      |      1 |        |      1 |
|   1 |  SORT AGGREGATE     |      |      1 |      1 |      1 |
|*  2 |   FILTER            |      |      1 |        |     10 |
|   3 |    TABLE ACCESS FULL| T1   |      1 |     10 |     10 |
|*  4 |    TABLE ACCESS FULL| T2   |      7 |      1 |      7 |
---------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter(("T1"."N1" IS NULL OR  IS NOT NULL))
   4 - filter("T2"."N1"=:B1)

The Bug

The Null-Accepting semi-join transformation comes, unfortunately, with a bug already identified in MyOracle Support via number 21201446. Here’s below how it can be reproduced:


SQL> select banner from v$version where rownum = 1;

BANNER
--------------------------------------------------------------------------------
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

create table t1 as select rownum n1, trunc((rownum -1/5)) n2 from dual connect by level <= 10; 

create table t2 as select rownum n1, trunc((rownum -1/3)) n2 from dual connect by level <= 10;

update t1 set n1 = null where n1 in (5,6,7);

exec dbms_stats.gather_table_stats(user, 't1');

exec dbms_stats.gather_table_stats(user, 't2');

SQL> SELECT  
         count(1)
    FROM t1
      WHERE(t1.n1 is null
         OR exists 
           (SELECT null FROM t2 where nvl(t2.n1,42) = nvl(t1.n1,42))
	    );

  COUNT(1)
----------
         7

----------------------------------------------------
| Id  | Operation           | Name | Rows  | Bytes |
----------------------------------------------------
|   0 | SELECT STATEMENT    |      |       |       |
|   1 |  SORT AGGREGATE     |      |     1 |     6 |
|*  2 |   HASH JOIN SEMI NA |      |     7 |    42 |
|   3 |    TABLE ACCESS FULL| T1   |    10 |    30 |
|   4 |    TABLE ACCESS FULL| T2   |    10 |    30 |
----------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 2 - filter(NVL("T2"."N1",42)=NVL("T1"."N1",42))

Using the HASH JOIN physical operation the query returns 7 rows. But it returns 10 rows when it uses the NESTED LOOP operation a shown below:

SELECT  
   count(1)
FROM t1
   WHERE(t1.n1 is null
          OR exists 
           (SELECT /*+ NL_SJ */ null FROM t2 where nvl(t2.n1,42) = nvl(t1.n1,42))
	   );

  COUNT(1)
----------
        10
		
------------------------------------------------------
| Id  | Operation             | Name | Rows  | Bytes |
------------------------------------------------------
|   0 | SELECT STATEMENT      |      |       |       |
|   1 |  SORT AGGREGATE       |      |     1 |     6 |
|   2 |   NESTED LOOPS SEMI NA|      |     7 |    42 |
|   3 |    TABLE ACCESS FULL  | T1   |    10 |    30 |
|*  4 |    TABLE ACCESS FULL  | T2   |     7 |    21 |
------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   4 – filter(NVL("T2"."N1",42)=NVL("T1"."N1",42))	

The Null-Accepting semi-join transformation is driven by the following hidden parameter which , if cancelled, will workarround this bug as shown below:

SQL> alter session set "_optimizer_null_accepting_semijoin"=false;

SELECT  
   count(1)
FROM t1
   WHERE(t1.n1 is null
          OR exists 
           (SELECT null FROM t2 where nvl(t2.n1,42) = nvl(t1.n1,42))
	   );

  COUNT(1)
----------
        10

----------------------------------------------------
| Id  | Operation           | Name | Rows  | Bytes |
----------------------------------------------------
|   0 | SELECT STATEMENT    |      |       |       |
|   1 |  SORT AGGREGATE     |      |     1 |     3 |
|*  2 |   FILTER            |      |       |       |
|   3 |    TABLE ACCESS FULL| T1   |    10 |    30 |
|*  4 |    TABLE ACCESS FULL| T2   |     1 |     3 |
----------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter(("T1"."N1" IS NULL OR  IS NOT NULL))
   4 - filter(NVL("T2"."N1",42)=NVL(:B1,42))

How this has been fixed in 12cR2?

Because of the bug n° 21201446 It seems that Oracle has completely cancelled the Null-Accepting semi-join transformation in 12cR2 for both NESTED LOOP and HASH JOIN physical operations when the NVL function is applied on the join column. Here’s below why I am thinking so:

SQL> select banner from v$version where rownum=1;

BANNER
-----------------------------------------------------------------------------
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SELECT  
   count(1)
FROM t1
   WHERE(t1.n1 is null
          OR exists 
           (SELECT null FROM t2 where nvl(t2.n1,42) = nvl(t1.n1,42))
	   );

  COUNT(1)
----------
        10

----------------------------------------------------
| Id  | Operation           | Name | Rows  | Bytes |
----------------------------------------------------
|   0 | SELECT STATEMENT    |      |       |       |
|   1 |  SORT AGGREGATE     |      |     1 |     3 |
|*  2 |   FILTER            |      |       |       |
|   3 |    TABLE ACCESS FULL| T1   |    10 |    30 |
|*  4 |    TABLE ACCESS FULL| T2   |     1 |     3 |
----------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter(("T1"."N1" IS NULL OR  IS NOT NULL))
   4 – filter(NVL("T2"."N1",42)=NVL(:B1,42))

In 12cR1, as we’ve shown above, the NESTED LOOP was not concerned by the bug. As from 12cR2 the NESTED LOOP SEMI JOIN is not anymore allowed to occur if a NVL function is applied on the join column as the following proves:

SELECT  
   count(1)
FROM t1
   WHERE(t1.n1 is null
          OR exists 
           (SELECT /*+ NL_SJ */ null FROM t2 where nvl(t2.n1,42) = nvl(t1.n1,42))
	   );

  COUNT(1)
----------
        10

----------------------------------------------------
| Id  | Operation           | Name | Rows  | Bytes |
----------------------------------------------------
|   0 | SELECT STATEMENT    |      |       |       |
|   1 |  SORT AGGREGATE     |      |     1 |     3 |
|*  2 |   FILTER            |      |       |       |
|   3 |    TABLE ACCESS FULL| T1   |    10 |    30 |
|*  4 |    TABLE ACCESS FULL| T2   |     1 |     3 |
----------------------------------------------------

Query Block Name / Object Alias (identified by operation id):
-------------------------------------------------------------
   1 - SEL$1
   3 - SEL$1 / T1@SEL$1
   4 - SEL$2 / T2@SEL$2

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('12.1.0.1')
      DB_VERSION('12.2.0.1')
      OPT_PARAM('_optimizer_null_accepting_semijoin' 'false')
      ALL_ROWS
      OUTLINE_LEAF(@"SEL$2")
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "T1"@"SEL$1")
      PQ_FILTER(@"SEL$1" SERIAL)
      FULL(@"SEL$2" "T2"@"SEL$2")
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter(("T1"."N1" IS NULL OR  IS NOT NULL))
   4 - filter(NVL("T2"."N1",42)=NVL(:B1,42))

December 10, 2017

Compress enable novalidate

Filed under: Oracle — hourim @ 7:47 pm

We knew here that we can’t DDL optimise a BASIC compressed table. So I uncompressed my table in a 12.2.0.1.0 and DDL optimise it as in the following:

SQL> select 
        pct_free
	,compression
	,compress_for
    from 
	 user_tables
    where 
	   table_name = 'T1';

  PCT_FREE COMPRESS COMPRESS_FOR
---------- -------- -------------
         0 DISABLED

SQL> alter table t1 add c_ddl number default 42 not null;

Table altered.

But I changed my mind and finally decided to revert back and drop this newly added column:

SQL> alter table t1 drop column c_ddl;
alter table t1 drop column c_ddl
                           *
ERROR at line 1:
ORA-39726: unsupported add/drop column operation on compressed tables

Damned database!!!

What’s going on here? My table is not compressed. Is it?

In fact I didn’t show you how I have uncompressed the t1 table:

SQL> alter table t1 nocompress;

Table altered.

Compressing or un-compressing a table without moving it will do nothing to its data. When the COMPRESSION column of a table is DISABLED this doesn’t necessarily mean that this table doesn’t contain compressed data . And this is probably the reason why the above bizarre error pops up out from nowhere.
In fact in order to have a significant meaning, the COMPRESSION column should normally always be considered with the PCT_FREE column (notice in passing that this is why I printed above the pct_free value):

SQL> select
         table_name
        ,compression
        ,pct_free
        ,compress_for
     from 
     user_tables
    where table_name = 'T1';

TABLE_NAME COMPRESS   PCT_FREE COMPRESS_FOR
---------- -------- ---------- --------------
T1         DISABLED          0

When PCT_FREE equals zero this is generally a hint that Oracle is considering BASIC compression since it thinks that the table is not considered for updates anymore. The definition given by Oracle to the COMPRESSION column goes in the same direction as well:

COMPRESSION	 	Indicates whether table compression 
                    is enabled (ENABLED) or not (DISABLED)

This is crystal clear: COMPRESSION referes to the status of the current compression. It doesn’t tell anything about the compression history of the table if any.

Compressing/uncompressing a table without moving it is nothing else than implementing or desimplementing compression but only for newly inserted rows. This is why I think that Oracle could have avoided the confusion by using the same command as when dealing with table constraints. Instead of this:

SQL> alter table t1 nocompress;

We would have this :

SQL> alter table t1 nocompress enable novalidate;

And of course the COMPRESSION column should have had a status indicating whether it is validated or not.

In the 10046 file of a compressed table with a move of its data and without a move we can see this respectively:

alter table t1 move compress basic

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.01       0.02          0          0          1           0
Execute      1      1.43       1.44          3       1865       2548     1000000
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        2      1.45       1.46          3       1865       2549     1000000

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 106  
Number of plan statistics captured: 1

Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
         0          0          0  LOAD AS SELECT  T1 (cr=1905 pr=1 pw=1382 time=1436345 us
   1000000    1000000    1000000   TABLE ACCESS FULL T1 (cr=1797 pr=0 pw=0 time=104378 

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  PGA memory operation                         1140        0.00          0.01
  Disk file operations I/O                        1        0.00          0.00
  db file sequential read                         3        0.00          0.00
  direct path write                               1        0.00          0.00
  reliable message                                4        0.00          0.00
  enq: RO - fast object reuse                     2        0.00          0.00
  enq: CR - block range reuse ckpt                2        0.00          0.00
  SQL*Net message to client                       1        0.00          0.00
  SQL*Net message from client                     1        5.93          5.93
********************************************************************************
alter table t1 nocompress


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          2           0
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        2      0.00       0.01          0          0          2           0

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 106  

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  Compression analysis                            8        0.00          0.00
  SQL*Net message to client                       1        0.00          0.00
  SQL*Net message from client                     1        6.65          6.65
********************************************************************************

Spot that whileOracle managed to rework a million rows of in the first case it didn’t touch any in the second case proving that nocompress/compress without a move doesn’t concern existing data.

Since it concerns only new data then an inserted row should be compressed/non-compressed. Let’s see:

SQL> create table t1 as select
      rownum n1
     ,mod(rownum,10) n2
     from dual
     connect by level <=1e6;

Table created.

SQL> @sizeBysegNameMB
Enter value for segment_name: t1
Enter value for owner: c##mhouri


SEGMENT_TYPE  TABLESPACE_NAME  SEGMENT_NAME PARTITION_NAME          MB
------------- ---------------- ------------ --------------- ----------
TABLE         USERS            T1                                   15
                                                            ----------
Total Segment Size                                                  15

Let’s compress this table without moving its data and get its new size:

SQL> alter table t1 compress;

Table altered.

SQL> @sizeBysegNameMB
Enter value for segment_name: t1
Enter value for owner: c##mhouri


SEGMENT_TYPE  TABLESPACE_NAME  SEGMENT_NAME PARTITION_NAME          MB
------------- ---------------- ------------ --------------- ----------
TABLE         USERS            T1                                   15
                                                            ----------
Total Segment Size                                                  15

As you can see the size of the table remains intact indicating again that compressing without moving does nothing to the existing data.

Let’s now direct path load a single row, dump the corresponding block and analyse the dump to see if compression has been activated or not:

SQL> select rowid, n1,n2 from t1 where n1=1;

ROWID                      N1         N2
------------------ ---------- ----------
AAATYNAAHAAAdpTAAA          1          1

SQL> insert /*+ append */ into t1 select 1,42 from dual;

1 row created.

SQL> select rowid, n1,n2 from t1 where n1=1;
select rowid, n1,n2 from t1 where n1=1
                         *
ERROR at line 1:
ORA-12838: cannot read/modify an object after modifying it in parallel


SQL> commit;

Commit complete.

SQL> select rowid, n1,n2 from t1 where n1=1;

ROWID                      N1         N2
------------------ ---------- ----------
AAATYNAAHAAAdpTAAA          1          1
AAATYNAAHAADOgcAAA          1         42

SQL> select
  2    dbms_rowid.rowid_relative_fno('AAATYNAAHAADOgcAAA') file_no
  3  , dbms_rowid.rowid_block_number('AAATYNAAHAADOgcAAA') block_no
  4  from dual;

   FILE_NO   BLOCK_NO
---------- ----------
         7     845852


SQL> alter session set tracefile_identifier='DumpingDirectLoadRowInsert';

Session altered.

SQL> alter system dump datafile 7 block 845852;

System altered.

Unfortunately I couldn’t spot anything interesting in the dump file which indicates that the inserted row has been compressed. I will probably come back to this article in a near future.

December 7, 2017

Optimizer statistics gathering

Filed under: direct path — hourim @ 8:24 pm

As from Oralce 12cR1, the Optimizer can automatically collect statistics on empty tables provided they are, notably, direct path loaded. This is recognisable in execution plans through the new row source operation named OPTIMIZER STATISTICS GATHERING

-----------------------------------------------------------
| Id  | Operation                        | Name   | Rows  |
-----------------------------------------------------------
|   0 | INSERT STATEMENT                 |        |  1000K|
|   1 |  LOAD AS SELECT                  | T1_TAR |       |
|   2 |   OPTIMIZER STATISTICS GATHERING |        |  1000K|
|   3 |    TABLE ACCESS FULL             | T1     |  1000K|
-----------------------------------------------------------

This is, in principle, a good initiative. But look at what I have been asked to trouble shoot a couple of days ago:

Global Information
------------------------------
 Status              :  EXECUTING                 
 Instance ID         :  1                    
 Session             :  xxxx (15:32901) 
 SQL ID              :  am6923m45s203        
 SQL Execution ID    :  16777216             
 Execution Started   :  12/02/2017 08:01:17  
 First Refresh Time  :  12/02/2017 08:01:28  
 Last Refresh Time   :  12/05/2017 11:55:51  
 Duration            :  272276s                  
 Module/Action       :  SQL*Plus/-           
 Service             :  xxxxx            
 Program             :  sqlplus@xxxx

===================================================================================================
| Id |             Operation               |  Name  | Activity | Activity Detail                  |
|    |                                     |        |   (%)    |   (# samples)                    |
===================================================================================================
| -> 0 | INSERT STATEMENT                  |        |   0.00   | Data file init write             |   
| -> 1 |   LOAD AS SELECT                  | T1_TAR |   0.06   | Cpu(28)                          |
|                                          |        |          | direct path write(2)             |
| -> 2 |    OPTIMIZER STATISTICS GATHERING |        |  99.93   | Cpu(52956)                       |
| -> 3 |     TABLE ACCESS FULL             | T1     |          |  SQL*Net more data from dblink(1)|               
==================================================================================================    

You have to believe me when I say that this insert had been running for 3 days when the above SQL report was taken. As you can see 99% of the direct path load execution time was spent gathering statistics at operation in line n°2. This is how a new feature designed to help starts disturbing you.

The insert/select statement selects a CLOB from a remote database. I was curious to see whether getting rid of this CLOB from the insert operation would make things better:

SQL>set timing on
 
    INSERT
   /*+ append */
 INTO T1_TAR
    (col1
   -- ,clob_col2
	,col3
	,col4
	,col5
	,col6
	,col7
	,col8
	)
 SELECT
    t1.col1 
   --,t1.clob_col2
   ,t1.col3
   ,t1.col4
   ,t1.col5
   ,t1.col6
   ,t1.col7
   ,t1.col8
  FROM
    t1@dblink t1;

338481182 rows created.

Elapsed: 00:11:38.85

------------------------------------------------------------
| Id  | Operation          | Name   | Rows  | Bytes |IN-OUT|
-----------------------------------------------------------
|   0 | INSERT STATEMENT   |        |  318M|    28GB|      |   
|   1 |  LOAD AS SELECT    | T1_TAR |      |        |      |
|   2 |   REMOTE           | T1     |  318M|    28GB|R->S  |
------------------------------------------------------------

That’s really interesting.

Why would the presence or the absence of a CLOB column in the insert/select statement allow or prevent the CBO from gathering statistics online?

In fact the CLOB column has nothing to do with the online statistics gathering decision. Should I have commented any other column I would have had the same behaviour as the following prooves:

INSERT
   /*+ append */
 INTO T1_TAR
    (col1
    ,clob_col2
	,col3
	,col4
	,col5
	,col6
	--,col7
	,col8
	)
 SELECT
    t1.col1 
   ,t1.clob_col2
   ,t1.col3
   ,t1.col4
   ,t1.col5
   ,t1.col6
   --,t1.col7
   ,t1.col8
  FROM
    t1@dblink t1;
------------------------------------------------------------
| Id  | Operation          | Name   | Rows  | Bytes |IN-OUT|
-----------------------------------------------------------
|   0 | INSERT STATEMENT   |        |  318M|    28GB|      |   
|   1 |  LOAD AS SELECT    | T1_TAR |      |        |      |
|   2 |   REMOTE           | T1     |  318M|    28GB|R->S  |
------------------------------------------------------------

The OPTIMIZER STATISTICS GATHERING operation requires the presence of the totality of the table columns in order to kick in during direct path load operations. This is probably because Oracle doesn’t want to collect statistics for a bunch of columns and keep others without statistics. This being said, this is another non-really documented restriction that comes with this new 12c feature.

Since I was not going to challenge my client getting rid of one column from the insert/select I finally opted for locally cancelling the online statistics gathering using the corresponding hint as shown below:

INSERT
   /*+ append 
       no_gather_optimizer_statistics
   */
 INTO T1_TAR
    (col1
    ,clob_col2
    ,col3
    ,col4
    ,col5
    ,col6
    ,col7
    ,col8
    )
 SELECT
    t1.col1 
   ,t1.clob_col2
   ,t1.col3
   ,t1.col4
   ,t1.col5
   ,t1.col6
   ,t1.col7
   ,t1.col8
  FROM
    t1@dblink t1;

Global Information
------------------------------
 Status              :  DONE                 
 Instance ID         :  1                    
 Session             :  xxxx (132:63271) 
 SQL ID              :  Od221zf0srs6q        
 SQL Execution ID    :  16777216             
 Execution Started   :  12/02/2017 15:10:16  
 First Refresh Time  :  12/02/2017 15:10:22  
 Last Refresh Time   :  12/05/2017 15:32:56  
 Duration            :  1360s                  
 Module/Action       :  SQL*Plus/-           
 Service             :  xxxxx            
 Program             :  sqlplus@xxxx

SQL Plan Monitoring Details (Plan Hash Value=2098243032)
================================================================
| Id |             Operation   |  Name  |   Time    |   Rows   |
|    |                         |        | Active(s) | (Actual) |
================================================================
|  0 | INSERT STATEMENT        |        |      1355 |        2 |
|  1 |   LOAD AS SELECT        | T1_TAR |      1355 |        2 |
|  2 |     REMOTE              | T1     |      1355 |     338M |
================================================================    

That’s how we fixed this issue.

Watch out your direct path load over empty tables selecting from either local or distant databases; the new 12c online table and columns statics gathering might considerably delay your treatement.

« Previous PageNext Page »

Create a free website or blog at WordPress.com.

Tony's Oracle Tips

Tony Hasler's light hearted approach to learning about Oracle

Richard Foote's Oracle Blog

Focusing Specifically On Oracle Indexes, Database Administration and Some Great Music

Hatem Mahmoud Oracle's blog

Just another Oracle blog : Database topics and techniques

Mohamed Houri’s Oracle Notes

Qui se conçoit bien s’énonce clairement

Oracle Diagnostician

Performance troubleshooting as exact science

Raheel's Blog

Things I have learnt as Oracle DBA

Coskan's Approach to Oracle

What I learned about Oracle

So Many Oracle Manuals, So Little Time

“Books to the ceiling, Books to the sky, My pile of books is a mile high. How I love them! How I need them! I'll have a long beard by the time I read them”—Lobel, Arnold. Whiskers and Rhymes. William Morrow & Co, 1988.

EU Careers info

Your career in the European Union

Carlos Sierra's Tools and Tips

Tools and Tips for Oracle Performance and SQL Tuning

Oracle Scratchpad

Just another Oracle weblog

OraStory

Dominic Brooks on Oracle Performance, Tuning, Data Quality & Sensible Design ... (Now with added Sets Appeal)