Amit Bansal

Plan Stability using Sql Profiles and SQL Plan Management

PLAN STABILITY

How many times you have noticed a query using Index X when you wanted it to use index Y or query performing Nested Loop join when Hash Join would have completed the query much faster.Or take a scenario when the application suddenly starts using wrong plan after database restart. To solve all these issues oracle provides feature called Plan stability. As per docs

“Plan stability prevents certain database environment changes from affecting the performance characteristics of applications. Such changes include changes in optimizer statistics, changes to the optimizer mode settings, and changes to parameters affecting the sizes of memory structures, such as SORT_AREA_SIZE and BITMAP_MERGE_AREA_SIZE. “

Oracle provides following ways to ensure it

1) Stored outlines
2) Sql Profiles
3)Sql Plan Management

Stored outlines

A stored outline is a collection of hints associated with a specific SQL statement that allows a standard execution plan to be maintained, regardless of changes in the system environment or associated statistics.
You can read about outlines at ORACLE-BASE

Sql Profiles

As per Jonathan Lewis post,

SQL Profile consists of a stored SQL statement, and a set of hints that will be brought into play when that SQL has to be optimised. Unlike Stored Outlines, the hints for SQL Profiles do not attempt to dictate execution mechanisms directly. Instead they supply arithmetical correction factors to the optimizer as it does its arithemetic as, even with a 100% sample size, it is still possible for the optimizer to misinterpret your statistics and produce an unsuitable execution path.

In principle, if the data distributions do not change, then a stored profile will ensure that the optimizer “understands” your data and does the right thing – even when the data volume changes.

Difference between stored outline and stored profiles

Quoting Tom kyte post 

Stored outlines are a set of hints that say “use this index, do this table first, do that next, use this access path, perform this filter then that filter”….

Sql profiles are more like extended statistics – they are the result of “analyzing a query”, the information provided to the optimizer is not HOW to perform the query – but rather better information about how many rows will flow out of a step in the plan, how selective something is.

Say you have a query, you generate a stored outline for it. You now add an index to the underlying table. This index would be GREAT for the query. A stored outline won’t use it, the stored outline says “use this index – query that table – then do this”. Since the GREAT index did not exist when the outline was generated – it won’t be used.

Say you have a query, you generate a profile for it. You now add an index to the underlying table. This index would be GREAT for the query. A profile can use it – since the profile is just more information for the CBO – better cardinality estimates.

So Stored outlines will ensure that whatever plan you fix, it will be used whereas a Sql profile might use different plan if we see some change in data distribution or new indexes.

There are two ways of generating sql profiles

1)Using Sql Tuning Advisor – A lot has been written about this, so I won’t discuss this.
2)Manually – This might be a surprising for lot of people as they might not be aware that we can generate a sql profile manually.I will be discussing this in detail below

Oracle provides a script coe_xfr_sql_profile.sql as part of Note 215187.1 – SQLT (SQLTXPLAIN) – Tool That Helps To Diagnose SQL Statements Performing Poorly which can be used to manually create Sql profiles.

It’s usage is pretty simple.

a) You need to execute the script and it will prompt you to pass sql_id for statement.
b)Then it will scan AWR repository and return Plan hash value for all possible execution plans for query along with average elapsed time.
c)You need to select the plan hash value which you think corresponds to good plan.
d)This will generate a script in current working directory which can be run as sys user and it will create the sql profile

This script is also useful if we wish to move the good plan from Non production environment to Production environment. To give a demo, I will create a table PLAN_STABILITY and index IDX_PLAN_STABI_OWNER on owner column.

SQL> create table plan_stability as select * from dba_objects;
Table created.

SQL> insert into plan_stability select * from dba_objects;
75048 rows created.

SQL> insert into plan_stability select * from plan_stability;
300192 rows created.
..
..
SQL> insert into plan_stability select * from plan_stability;
1200768 rows created.
SQL> commit;

22:15:05 SQL> create index IDX_PLAN_STABI_OWNER on plan_stability(owner);

Index created.
SQL> select count(*) from plan_stability;

COUNT(*)
----------
2401536

Let’s query the table for count of objects owned by SYS

SQL> select count(*) from plan_stability where owner='SYS';

COUNT(*)
----------
1020384

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------
SQL_ID g3wubsadyrt37, child number 0
-------------------------------------
select count(*) from plan_stability where owner='SYS'

Plan hash value: 3082746383

------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 57 (100)| |
| 1 | SORT AGGREGATE | | 1 | 6 | | |
|* 2 | INDEX RANGE SCAN| IDX_PLAN_STABI_OWNER | 23092 | 135K| 57 (0)| 00:00:01 |
------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

2 - access("OWNER"='SYS')

We see that we are using index IDX_PLAN_STABI_OWNER. Since data is less , Index access is best approach.But for this demo I want to force a Full table scan . Let’s generate a plan with FTS for plan_stability. To do this we can use  Invisible indexes (11g feature)  so that optimizer ignores index.

SQL> alter index IDX_PLAN_STABI_OWNER invisible;

Index altered.
SQL> select count(*) from plan_stability where owner='SYS';

COUNT(*)
----------
1020384

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------
SQL_ID g3wubsadyrt37, child number 0
-------------------------------------
select count(*) from plan_stability where owner='SYS'

Plan hash value: 363261562

-------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2476 (100)| |
| 1 | SORT AGGREGATE | | 1 | 6 | | |
|* 2 | TABLE ACCESS FULL| PLAN_STABILITY | 23092 | 135K| 2476 (1)| 00:00:30 |
-------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

2 - filter("OWNER"='SYS')
19 rows selected.
--Let's make it index visible again

SQL> alter index IDX_PLAN_STABI_OWNER visible;

Index altered.

We will have to take manual AWR snapshots before and end of the test, so that sql plans goes into AWR repository . This is necessary as coe_xfr_profile.sql will look at AWR data for plan history for a sql. Now we run coe_xfr_profile.sql and it will prompt us for sql_id. On passing the sql_id, it reports that there are two plans. We choose plan 363261562 (having higher elapsed time) as we wish to force a Full table scan

SQL>@coe_xfr_profile.sql

Parameter 1:
SQL_ID (required)

Enter value for 1: g3wubsadyrt37
PLAN_HASH_VALUE AVG_ET_SECS
--------------- -----------
3082746383 .064
363261562 .184

Parameter 2:
PLAN_HASH_VALUE (required)

Enter value for 2: 363261562

Values passed:
~~~~~~~~~~~~~
SQL_ID : "g3wubsadyrt37"
PLAN_HASH_VALUE: "363261562"
Execute coe_xfr_sql_profile_g3wubsadyrt37_363261562.sql
on TARGET system in order to create a custom SQL Profile
with plan 363261562 linked to adjusted sql_text.
COE_XFR_SQL_PROFILE completed.

This has generated a file coe_xfr_sql_profile_g3wubsadyrt37_363261562.sql in same directory which can be run to create the sql profile.Note that script has force_match => FALSE which means that if sql varies in literal, sql_profile will not work. To force it ,we need to set the paramter to force.

SQL>@coe_xfr_sql_profile_g3wubsadyrt37_363261562
SQL>REM
SQL>REM $Header: 215187.1 coe_xfr_sql_profile_g3wubsadyrt37_363261562.sql 11.4.1.4 2011/12/18 csierra $
SQL>REM
SQL>REM Copyright (c) 2000-2010, Oracle Corporation. All rights reserved.
SQL>REM
SQL>REM coe_xfr_sql_profile_g3wubsadyrt37_363261562.sql
SQL>REM
SQL>REM DESCRIPTION
SQL>REM This script is generated by coe_xfr_sql_profile.sql
SQL>REM It contains the SQL*Plus commands to create a custom
SQL>REM SQL Profile for SQL_ID g3wubsadyrt37 based on plan hash
SQL>REM value 363261562.
SQL>REM The custom SQL Profile to be created by this script
SQL>REM will affect plans for SQL commands with signature
SQL>REM matching the one for SQL Text below.
SQL>REM Review SQL Text and adjust accordingly.
SQL>REM
SQL>REM PARAMETERS
SQL>REM None.
SQL>REM
SQL>REM EXAMPLE
SQL>REM SQL> START coe_xfr_sql_profile_g3wubsadyrt37_363261562.sql;
SQL>REM
SQL>REM NOTES
SQL>REM 1. Should be run as SYSTEM or SYSDBA.
SQL>REM 2. User must have CREATE ANY SQL PROFILE privilege.
SQL>REM 3. SOURCE and TARGET systems can be the same or similar.
SQL>REM 4. To drop this custom SQL Profile after it has been created:
SQL>REM EXEC DBMS_SQLTUNE.DROP_SQL_PROFILE('coe_g3wubsadyrt37_363261562');
SQL>REM 5. Be aware that using DBMS_SQLTUNE requires a license
SQL>REM for the Oracle Tuning Pack.
SQL>REM
SQL>WHENEVER SQLERROR EXIT SQL.SQLCODE;
SQL>REM
SQL>VAR signature NUMBER;
SQL>REM
SQL>DECLARE
2 sql_txt CLOB;
3 h SYS.SQLPROF_ATTR;
4 BEGIN
5 sql_txt := q'[
6 select count(*) from plan_stability where owner='SYS'
7 ]';
8 h := SYS.SQLPROF_ATTR(
9 q'[BEGIN_OUTLINE_DATA]',
10 q'[IGNORE_OPTIM_EMBEDDED_HINTS]',
11 q'[OPTIMIZER_FEATURES_ENABLE('11.2.0.3')]',
12 q'[DB_VERSION('11.2.0.3')]',
13 q'[OPT_PARAM('_b_tree_bitmap_plans' 'false')]',
14 q'[OPT_PARAM('optimizer_dynamic_sampling' 0)]',
15 q'[ALL_ROWS]',
16 q'[OUTLINE_LEAF(@"SEL$1")]',
17 q'[FULL(@"SEL$1" "PLAN_STABILITY"@"SEL$1")]',
18 q'[END_OUTLINE_DATA]');
19 :signature := DBMS_SQLTUNE.SQLTEXT_TO_SIGNATURE(sql_txt);
20 DBMS_SQLTUNE.IMPORT_SQL_PROFILE (
21 sql_text => sql_txt,
22 profile => h,
23 name => 'coe_g3wubsadyrt37_363261562',
24 description => 'coe g3wubsadyrt37 363261562 '||:signature||'',
25 category => 'DEFAULT',
26 validate => TRUE,
27 replace => TRUE,
28 force_match => FALSE /* TRUE:FORCE (match even when different literals in SQL). FALSE:EXACT (similar to CURSOR_SHARING) */ );
29 END;
30 /

PL/SQL procedure successfully completed.

SQL>WHENEVER SQLERROR CONTINUE
SQL>SET ECHO OFF;

SIGNATURE
---------------------
4782703451567619555
... manual custom SQL Profile has been created
COE_XFR_SQL_PROFILE_g3wubsadyrt37_363261562 completed

Let’s verify the plan again now

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------
SQL_ID g3wubsadyrt37, child number 1
-------------------------------------
select count(*) from plan_stability where owner='SYS'

Plan hash value: 363261562

-------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2476 (100)| |
| 1 | SORT AGGREGATE | | 1 | 6 | | |
|* 2 | TABLE ACCESS FULL| PLAN_STABILITY | 23092 | 135K| 2476 (1)| 00:00:30 |
-------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

2 - filter("OWNER"='SYS')

Note
-----
- SQL profile coe_g3wubsadyrt37_363261562 used for this statement

In the Note section we can see that Sql Profile “coe_g3wubsadyrt37_363261562” has been used for this statement.

Suppose we now have to create a sql profile, when we don’t have good plan in AWR, then we will have to do a hack. There are two ways to do it

a)You can use coe_xfr_profile.sql to do same. You will have to run the script twice, one for original statement and secondly for hinted statement. Once done you need to copy the values corresponding to h := SYS.SQLPROF_ATTR from hinted sql and replace it in original script.e.g

I am creating a sql profile for following hinted statement

SQL> select /*+ full(plan_stability) */ count(*) from plan_stability where owner='SYS';

  COUNT(*)
----------
   1020384

 SQL> select /*+ full(plan_stability) */ count(*) from plan_stability where owner='SYS';

  COUNT(*)
----------
   1020384

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------
SQL_ID	9p07a64q8fgrn, child number 0
-------------------------------------
select /*+ full(plan_stability) */ count(*) from plan_stability where
owner='SYS'

Plan hash value: 363261562

-------------------------------------------------------------------------------------
| Id  | Operation	   | Name	    | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |		    |	    |	    |  2476 (100)|	    |
|   1 |  SORT AGGREGATE    |		    |	  1 |	  6 |		 |	    |
|*  2 |   TABLE ACCESS FULL| PLAN_STABILITY | 23092 |	135K|  2476   (1)| 00:00:30 |
-------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter("OWNER"='SYS')

--Creating Sql profile now

SQL>@coe_xfr_profile.sql 9p07a64q8fgrn 363261562

Parameter 1:
SQL_ID (required)

PLAN_HASH_VALUE AVG_ET_SECS
--------------- -----------
      363261562        .193

Parameter 2:
PLAN_HASH_VALUE (required)

Values passed:
~~~~~~~~~~~~~
SQL_ID	       : "9p07a64q8fgrn"
PLAN_HASH_VALUE: "363261562"

Execute coe_xfr_sql_profile_9p07a64q8fgrn_363261562.sql
on TARGET system in order to create a custom SQL Profile
with plan 363261562 linked to adjusted sql_text.

COE_XFR_SQL_PROFILE completed.

Now we create sql profile script for original statement using plan_hash_value corresponding to index access

20:47:52 SQL> @coe_xfr_profile.sql g3wubsadyrt37 3082746383

Parameter 1:
SQL_ID (required)

PLAN_HASH_VALUE AVG_ET_SECS
--------------- -----------
     3082746383        .084
      363261562        .184

Parameter 2:
PLAN_HASH_VALUE (required)

Values passed:
~~~~~~~~~~~~~
SQL_ID	       : "g3wubsadyrt37"
PLAN_HASH_VALUE: "3082746383"

Execute coe_xfr_sql_profile_g3wubsadyrt37_3082746383.sql
on TARGET system in order to create a custom SQL Profile
with plan 3082746383 linked to adjusted sql_text.

COE_XFR_SQL_PROFILE completed.

Now copy following section from coe_xfr_sql_profile_9p07a64q8fgrn_363261562.sql and replace in the coe_xfr_sql_profile_g3wubsadyrt37_3082746383.sql. Also change the name from coe_g3wubsadyrt37_3082746383 to coe_g3wubsadyrt37_363261562

h := SYS.SQLPROF_ATTR(
q'[BEGIN_OUTLINE_DATA]',
q'[IGNORE_OPTIM_EMBEDDED_HINTS]',
q'[OPTIMIZER_FEATURES_ENABLE('11.2.0.3')]',
q'[DB_VERSION('11.2.0.3')]',
q'[ALL_ROWS]',
q'[OUTLINE_LEAF(@"SEL$1")]',
q'[FULL(@"SEL$1" "PLAN_STABILITY"@"SEL$1")]',
q'[END_OUTLINE_DATA]');

This has again created a sql profile with FTS plan.

SQL> select name,SIGNATURE,SQL_TEXT,FORCE_MATCHING,STATUS from dba_sql_profiles;

NAME						    SIGNATURE SQL_TEXT								     FOR STATUS
------------------------------ ------------------------------ ---------------------------------------------------------------------- --- --------
coe_g3wubsadyrt37_363261562		  4782703451567619555 select count(*) from plan_stability where owner='SYS'		     NO  ENABLED

b) You can use scripts provided by Kerry Osborne in following article
http://kerryosborne.oracle-guy.com/2009/10/how-to-attach-a-sql-profile-to-a-different-statement-take-2/

SQL Plan Management

Let’s use the 11g feature SQL Plan Management to implement plan stability.SQL plan management is a preventative mechanism that records and evaluates the execution plans of SQL statements over time, and builds SQL plan baselines composed of a set of existing plans known to be efficient.
The SQL plan baselines are then used to preserve performance of corresponding SQL statements, regardless of changes occurring in the system. We can capture plans

a)Automatically – We can use OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=true and capture the sql baseline plans. This is pretty useful if you are upgrading database as you can capture good plans and then upgrade the database without worrying about plan change. e.g Post 11g upgrade from 10.2.0.4, set optimizer_features_enable=10.2.0.4 and capture all the plans.Once done you can set the OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=false and also set optimizer_features_enable back to 11.1/11.2. You can then enable/disable the plans or mark them Fixed. Fixed plans get preference over non-fixed plans.

Note that only plans for repeatable statements are stored in the SPM

b)Manually – We can use DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE and DBMS_SPM.LOAD_PLANS_FROM_SQLSET to load plans from cursor cache and Sqlset respectively.

Loading plan from Cursor Cache

We will discuss fucntion DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE to load plans from the cursor cache.In our test,we need to fix plan 363261562 for sql_id g3wubsadyrt37.Since the plan is available in cursor cache,  we will use following syntax

DECLARE
my_plans pls_integer;
BEGIN
my_plans := DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE(
sql_id => 'g3wubsadyrt37',
PLAN_HASH_VALUE =>363261562,
FIXED =>'YES');
dbms_output.put_line('Value is '||my_plans);
END;
/

It will return number of plans loaded.Let’s verify baseline by querying dba_sql_plan_baselines

SQL>select sql_handle, plan_name, enabled, accepted, fixed from dba_sql_plan_baselines;

SQL_HANDLE PLAN_NAME ENA ACC FIX
------------------------------ ------------------------------ --- --- ---
SQL_425f937308b8fde3 SQL_PLAN_44rwmfc4bjzg3621540b0 YES YES YES

We can plan corresponding to this baseline

SQL> select * from table(
dbms_xplan.display_sql_plan_baseline(
sql_handle=>'SQL_425f937308b8fde3',
format=>'basic')); 

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------

SQL handle: SQL_425f937308b8fde3
SQL text: select count(*) from plan_stability where owner='SYS'
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Plan name: SQL_PLAN_44rwmfc4bjzg3621540b0	  Plan id: 1645559984
Enabled: YES	 Fixed: NO	Accepted: YES	  Origin: MANUAL-LOAD
--------------------------------------------------------------------------------

Plan hash value: 363261562

---------------------------------------------
| Id  | Operation	   | Name	    |
---------------------------------------------
|   0 | SELECT STATEMENT   |		    |
|   1 |  SORT AGGREGATE    |		    |
|   2 |   TABLE ACCESS FULL| PLAN_STABILITY |
---------------------------------------------

20 rows selected.

We can check if our original query is using  the plan

SQL> select count(*) from plan_stability where owner='SYS';

COUNT(*)
----------
1020384

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------
SQL_ID g3wubsadyrt37, child number 1
-------------------------------------
select count(*) from plan_stability where owner='SYS'

Plan hash value: 363261562

-------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2476 (100)| |
| 1 | SORT AGGREGATE | | 1 | 6 | | |
|* 2 | TABLE ACCESS FULL| PLAN_STABILITY | 23092 | 135K| 2476 (1)| 00:00:30 |
-------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

2 - filter("OWNER"='SYS')

Note
-----
- SQL plan baseline SQL_PLAN_44rwmfc4bjzg3621540b0 used for this statement

We see from Note section  that SQL plan baseline SQL_PLAN_44rwmfc4bjzg3621540b0 has been used for the statement.

Loading plan from AWR/SQL Tuning set

If you know that good plan of the query is available in AWR, then you can use dbms_spm.load_plans_from_sqlset. To do this we need to first create a sql tuning set. This can be done by using DBMS_SQLTUNE

BEGIN
DBMS_SQLTUNE.CREATE_SQLSET(
sqlset_name => 'sqlset_g3wubsadyrt37',
description => 'Example to show dbms_spm.load_plans_from_sqlset');
END;
/

Let’s load the sql_id g3wubsadyrt37 in this Sql tuning set from AWR. We need to pass begin_snap and end_snap for SQL. We can also use basic_filter clause to restrict this sqlset to only one particular sql_id.

DECLARE
baseline_cursor DBMS_SQLTUNE.SQLSET_CURSOR;
BEGIN
OPEN baseline_cursor FOR
SELECT VALUE(p)
FROM TABLE (DBMS_SQLTUNE.SELECT_WORKLOAD_REPOSITORY(
BEGIN_SNAP => 641,
END_SNAP => 667,
BASIC_FILTER => 'sql_id = ''g3wubsadyrt37''',
ATTRIBUTE_LIST =>'ALL')) p;

DBMS_SQLTUNE.LOAD_SQLSET(
sqlset_name => 'sqlset_g3wubsadyrt37',
populate_cursor => baseline_cursor);
END;
/

Lets verify that both plans are loaded in the sqlset

SELECT SQL_ID,PLAN_HASH_VALUE FROM TABLE(DBMS_SQLTUNE.SELECT_SQLSET(
'sqlset_g3wubsadyrt37'));

SQL_ID PLAN_HASH_VALUE
------------- ---------------
g3wubsadyrt37 363261562
g3wubsadyrt37 3082746383

Next we create sql plan baseline using DBMS_SPM.LOAD_PLANS_FROM_SQLSET

DECLARE
my_plans pls_integer;
BEGIN
my_plans := DBMS_SPM.LOAD_PLANS_FROM_SQLSET(
SQLSET_NAME => 'sqlset_g3wubsadyrt37'
);
dbms_output.put_line('Value is '||my_plans);
END;
/

We now see that two plans are loaded

SQL> select sql_handle, plan_name, enabled, accepted, fixed from dba_sql_plan_baselines;

SQL_HANDLE PLAN_NAME ENA ACC FIX
------------------------------ ------------------------------ --- --- ---
SQL_425f937308b8fde3 SQL_PLAN_44rwmfc4bjzg3621540b0 YES YES NO
SQL_425f937308b8fde3 SQL_PLAN_44rwmfc4bjzg3ec110d89 YES YES NO

We can disable the SQL_PLAN_44rwmfc4bjzg3ec110d89 as we only want FTS plan

DECLARE
my_plans pls_integer;
BEGIN
my_plans := DBMS_SPM.ALTER_SQL_PLAN_BASELINE(
sql_handle =>'SQL_425f937308b8fde3',
PLAN_NAME =>'SQL_PLAN_44rwmfc4bjzg3ec110d89',
attribute_name =>'enabled',
attribute_value=>'NO');
dbms_output.put_line('Value is '||my_plans);
END;
/

Above statement will mark the index plan as disabled and our query will use only FTS plan.

To drop the baselines, we can use following syntax

DECLARE
my_plans pls_integer;
BEGIN
my_plans := DBMS_SPM.DROP_SQL_PLAN_BASELINE(
sql_handle => 'SQL_425f937308b8fde3'
);
dbms_output.put_line('Value is '||my_plans);
END;
/

Till now we have  discussed  scenarios when we have good plan in cursor cache/AWR. Suppose there is no good plan available but we can generate a good plan using hints. With SPM we can easily transfer profile from hinted sql to our original statement. Let’s see it in action

First we need to create a baseline with existing plan for query. This is required as it would generate a sql_handle which can be used to assign a plan. We are using PLAN_HASH_VALUE =>3082746383 i.e Indexed access path to create the sql baseline

DECLARE
my_plans pls_integer;
BEGIN
my_plans := DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE(
sql_id => 'g3wubsadyrt37',
PLAN_HASH_VALUE =>3082746383
);
dbms_output.put_line('Value is '||my_plans);
END;
/

SQL>select sql_handle, plan_name, enabled, accepted, fixed,sql_text from dba_sql_plan_baselines;

SQL_HANDLE PLAN_NAME ENA ACC FIX SQL_TEXT
------------------------------ ------------------------------ --- --- --- --------------------------------------------------------------------------------
SQL_425f937308b8fde3 SQL_PLAN_44rwmfc4bjzg3ec110d89 YES YES NO select count(*) from plan_stability where owner='SYS'

Querying dba_sql_plan_baselines we get sql_handle as SQL_425f937308b8fde3.Now let’s create a plan for query using hints.

SQL> select /*+ full(plan_stability) */ count(*) from plan_stability where owner='SYS';

COUNT(*)
----------
1020384

SQL> select /*+ full(plan_stability) */ count(*) from plan_stability where owner='SYS';

COUNT(*)
----------
1020384

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------
SQL_ID 9p07a64q8fgrn, child number 0
-------------------------------------
select /*+ full(plan_stability) */ count(*) from plan_stability where
owner='SYS'

Plan hash value: 363261562

-------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2476 (100)| |
| 1 | SORT AGGREGATE | | 1 | 6 | | |
|* 2 | TABLE ACCESS FULL| PLAN_STABILITY | 23092 | 135K| 2476 (1)| 00:00:30 |
-------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

2 - filter("OWNER"='SYS')

We need to now copy plan corresponding to sql_id=9p07a64q8fgrn and plan_hash_value=363261562. This can be done by using sql_handle for original statement and using sql_id/plan_hash_value of hinted statement

DECLARE
my_plans pls_integer;
BEGIN
my_plans := DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE(
sql_id => '9p07a64q8fgrn',
sql_handle =>'SQL_425f937308b8fde3',
PLAN_HASH_VALUE =>363261562,
FIXED =>'YES'
);
dbms_output.put_line('Value is '||my_plans);
END;
/

Note that I have used FIXED=>YES argument which would mark this plan as Fixed.A fixed plan takes precedence over a non-fixed plan. We can drop the old baseline plan now by passing sql_handle and plan_name. If you use only sql_handle, it will drop both the plans

DECLARE
my_plans pls_integer;
BEGIN
my_plans := DBMS_SPM.DROP_SQL_PLAN_BASELINE(
sql_handle => 'SQL_425f937308b8fde3',
PLAN_NAME =>'SQL_PLAN_44rwmfc4bjzg3ec110d89');
dbms_output.put_line('Value is '||my_plans);
END;
/

Instead of dropping plan, you can also disable the above plan using DBMS_SPM.ALTER_SQL_PLAN_BASELINE

DECLARE
my_plans pls_integer;
BEGIN
my_plans := DBMS_SPM.ALTER_SQL_PLAN_BASELINE(
sql_handle =>'SQL_425f937308b8fde3',
PLAN_NAME =>'SQL_PLAN_44rwmfc4bjzg3ec110d89',
attribute_name =>'enabled',
attribute_value=>'NO');
dbms_output.put_line('Value is '||my_plans);
END;
/

We now have two plans in baseline but only enabled plan SQL_PLAN_44rwmfc4bjzg3621540b0 will be used by the query.

Note: Whenever we create/drop sql baseline plan, sql is purged from cursor cache and we perform a hard parse.

While testing this , I found that if I create a alias for table , SQL Plan baseline is not working

SQL> select /*+ full(a) */ count(*) from plan_stability a where owner='SYS';

COUNT(*)
----------
1020384

Elapsed: 00:00:00.18
select /*+ full(a) */ count(*) from plan_stability a where owner='SYS';

COUNT(*)
----------
1020384

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------
SQL_ID 85hj3c8570fdu, child number 0
-------------------------------------
select /*+ full(a) */ count(*) from plan_stability a where owner='SYS'

Plan hash value: 363261562

-------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2476 (100)| |
| 1 | SORT AGGREGATE | | 1 | 6 | | |
|* 2 | TABLE ACCESS FULL| PLAN_STABILITY | 23092 | 135K| 2476 (1)| 00:00:30 |
-------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

2 - filter("OWNER"='SYS')

SQL>DECLARE
my_plans pls_integer;
BEGIN
my_plans := DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE(
sql_id => '85hj3c8570fdu',
sql_handle =>'SQL_425f937308b8fde3',
PLAN_HASH_VALUE =>363261562,
FIXED =>'YES'
);
dbms_output.put_line('Value is '||my_plans);
END;
/

SQL>select sql_handle, plan_name, enabled, accepted, fixed,sql_text from dba_sql_plan_baselines;

SQL_HANDLE PLAN_NAME ENA ACC FIX SQL_TEXT
------------------------------ ------------------------------ --- --- --- --------------------------------------------------------------------------------
SQL_425f937308b8fde3 SQL_PLAN_44rwmfc4bjzg3621540b0 YES YES YES select count(*) from plan_stability where owner='SYS'

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------
SQL_ID g3wubsadyrt37, child number 1
-------------------------------------
select count(*) from plan_stability where owner='SYS'

Plan hash value: 363261562

-------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2476 (100)| |
| 1 | SORT AGGREGATE | | 1 | 6 | | |
|* 2 | TABLE ACCESS FULL| PLAN_STABILITY | 23092 | 135K| 2476 (1)| 00:00:30 |
-------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

2 - filter("OWNER"='SYS')

We can check baseline information using dbms_xplan.display_sql_plan_baseline function.Let’s see why we didnt use FTS even though profile was successfully created

SQL> select * from table(
dbms_xplan.display_sql_plan_baseline(
sql_handle=>'SQL_425f937308b8fde3',
format=>'basic'));

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------

SQL handle: SQL_425f937308b8fde3
SQL text: select count(*) from plan_stability where owner='SYS'
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Plan name: SQL_PLAN_44rwmfc4bjzg3621540b0 Plan id: 1645559984
Enabled: YES Fixed: YES Accepted: YES Origin: MANUAL-LOAD
--------------------------------------------------------------------------------

Plan hash value: 3082746383

--------------------------------------------------
| Id | Operation | Name |
--------------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | SORT AGGREGATE | |
| 2 | INDEX RANGE SCAN| IDX_PLAN_STABI_OWNER |
--------------------------------------------------

--------------------------------------------------------------------------------
Plan name: SQL_PLAN_44rwmfc4bjzg3ec110d89 Plan id: 3960540553
Enabled: YES Fixed: NO Accepted: NO Origin: AUTO-CAPTURE
--------------------------------------------------------------------------------

Plan hash value: 3082746383

--------------------------------------------------
| Id | Operation | Name |
--------------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | SORT AGGREGATE | |
| 2 | INDEX RANGE SCAN| IDX_PLAN_STABI_OWNER |
--------------------------------------------------

From above we see that SQL_PLAN_44rwmfc4bjzg3621540b0 is showing plan which uses Index and not FTS.I am not sure if this is correct behavior or a bug. One reason could be that  same alias is not present in parent table.

I believe Sql Plan Management is superior to SQL Profile and outlines as it allows you to have multiple plans for a given sql statement. Also a nightly maintenance job runs automatic SQL tuning task and targets high-load SQL statements.If it thinks that there is better plan , then it adds that SQL Plan baseline to history. This is called Evolving Sql Plan Baselines.

Christian Antognini has written a good article on Automatic Evolution of SQL Plan Baselines which explains this feature.

Licensing

As per Optimizer blog article  ,DBMS_SPM is not licensed until we are loading plans from SQL Tuning set (which requires tuning pack).SQL profiles are  licensed and require diagnostic and tuning pack.

V$SQL_SHARED_CURSOR in 11.2.0.2

Oracle 11.2.0.2 has introduced new column “REASON” to V$SQL_SHARED_CURSORS. I came across this extremely useful column while debugging multiple child cursor issue (11.2.0.2 seems to have too many bugs which can cause multiple child cursors).

As per documentation , REASON column has been introduced in 11.2.0.2

REASON - CLOB	Child number, id, and reason the cursor is not shared. The content of this column is structured using XML.

Reason column is helpful as we can easily interpret the reason why child cursors are not being shared. e.g

<ChildNode>
<ChildNumber>151</ChildNumber>
<ID>9</ID>
<reason>PQ Slave mismatch(5)</reason>
<size>2x4</size>
<ctxpq_StmtId_pqctx>0</ctxpq_StmtId_pqctx>
<fxu_kxfxutqidb>2422463</fxu_kxfxutqidb>
</ChildNode>
<ChildNode><ChildNumber>151</ChildNumber>
<ID>3</ID>
<reason>Optimizer mismatch(2)</reason>
<size>4x4</size>
<parallel_query_default_dop>16</parallel_query_default_dop>
<kxfr_Default_DOP>80</kxfr_Default_DOP>
<isParallel>1</isParallel>
<Need_Default_Dop>0</Need_Default_Dop>
</ChildNode>
<ChildNode>
<ChildNumber>151</ChildNumber>
<ID>9</ID><reason>PQ Slave mismatch(5)</reason><size>2x4</size><ctxpq_StmtId_pqctx
>0</ctxpq_StmtId_pqctx><fxu_kxfxutqidb>1433944</fxu_kxfxutqidb></ChildNode>

We can verify the same thing by checking PQ_SLAVE_MISMATCH column

select sql_id,PQ_SLAVE_MISMATCH,OPTIMIZER_MODE_MISMATCH from  V$SQL_SHARED_CURSOR where sql_id='b9uz0akdcx58q' and child_number=151;

SQL_ID	      P O
------------- - -
b9uz0akdcx58q Y N

There is one more column PURGED_CURSOR which is new (I guess it came from 11.2.0.1) which tells if the cursor has been purged using dbms_shared_pool.purge package.

11gR1 CRS start failing with ORA-29702

This post was actually a comment from Sasi which came on previous article 10.2 CRS startup issue . I am converting it to a post so that we can get some feedback on this issue from other users. I suspect this to be caused by RHEL5 bug (fixed in RHEL5u6) related to NIC going down when multiple interface cards are being used. 

We had a similar error but the problem was different and thought of sharing it here.

We recently installed 11gR1 two node RAC and all was fine till last week and suddenly we saw the same error “ORA-29702: error occurred in Cluster Group Service operation”. Crs was not starting . some of the crs process were running and it was refusing to stop.

root@node1> /u01/app/crs/bin/crsctl stop crs
Stopping resources.
This could take several minutes.
Error while stopping resources. Possible cause: CRSD is down.
Stopping Cluster Synchronization Services.
Unable to communicate with the Cluster Synchronization Services daemon.

ASM alert log and database alert log had the below to say

ASM Alert Log:

Errors in file /u02/app/asm/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_2185.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON (ospid: 2185): terminating the instance due to error 29702
Mon Nov 22 20:02:16 2011
ORA-1092 : opitsk aborting process

Oracle database Alert Log:

ERROR: LMON (ospid: 3721) detects hung instances during IMR reconfiguration
Tue Nov 22 22:10:37 2011
Error: KGXGN polling error (16)
Errors in file /u03/app/oracle/diag/rdbms/ccbdrpd/ccbdrpd1/trace/ccbdrpd1_lmon_3721.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON (ospid: 3721): terminating the instance due to error 29702

Not much info in the trace files.

Looked at metalink note : Diagnosing ‘ORA-29702: error occurred in Cluster Group Service operation’ [ID 848622.1]
But the problems mentioned in it were not applicable to our site.

Looked at CRS alert log, CRSD logs and CSSD logs, there were heaps of information but not quite useful to nail down the issue. Could not see any error messages

Also, looked at

RAC ASM instances crash with ORA-29702 when multiple ASM instances start(ed) [ID 733262.1]

There it was mentioned, when using multiple NIC for cluster interconnect and if they are not bonded properly it could cause issues and that could be seen in the alert logs.

In our case NIC bonding was done properly. We have configured and bonded as below
• eth0 and eth1 bonded as bond0 – for public and
• eth2 and eth3 bonded as bond1 – for cluster interconnect

and alert log showed they were configured fine.

Interface type 1 bond1 192.xxx.x.x configured from OCR for use as a cluster interconnect
Interface type 1 bond0 xx.x.x.x configured from OCR for use as a public interface

If NIC bonding not done properly then you would see multiple entries for cluser interconnect in the alert log.

Well,though this was not the issue in our case but it gave me a lead to identify the root cause of the problem. As it was mentioned about bonding I wanted to check both channel bonding interface (ifcdfg-bond0 & ifcfg-bond1) and Ethernet interface configurations (ifcfg-eth0, ifcfg-eth1, ifcfg-eth2 & ifcfg-eth3)

Well, all configuration files were good except for ifcfg-bond1 file and the entries were as below,

root@node1>cat ifcfg-bond1

DEVICE=bond1
IPADDR=xxx.xxx.xx.x
NETMASK=255.xxx.x.x
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
TYPE=ethernet

On the 1st look they seem to be fine but when compared to ifcfg-bond0 the problem was obvious. Ifcfg-bond0 entries were as below,

root@node1> cat ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
NETMASK=255.xxx.x.x
IPADDR=xx.x.x.x
GATEWAY=xx.x.x.x
USERCTL=no
TYPE=BOND

If you look at line entry TYPE it’s mentioned as “TYPE=ethernet” in Ifcfg-bond1 and “TYPE=BOND” In Ifcfg-bond0.

Bingo…changed the configuration file and rebooted the server and all components came up fine. CRS, ASM and DB started and working fine.

But trying to find out why it worked fine during the installation and then stopped working suddenly.

AIOUG 2011 – I am Attending :)

All India Oracle User Group(AIOUG) is organizing the annual Oracle Conference in Bangalore on Friday, December 9th and Saturday December 10th, 2011.Arup Nanda is key speaker this year.

I will be attending this conference for first time and I am looking forward to meeting fellow bloggers/DBA’s. See you all there

You can find the registration link at http://www.aioug.org/sangam11.php

Changing CRS/Database timezone in 11.2.0.2 post install

I had installed a 11.2.0.2 RAC setup few days back with incorrect timezone. It had to be PDT but I installed with UTC.
Starting/stopping clusteware with correct timezone didn’t solve the issue.

In 11.2.0.2 Oracle stores timezone information in file $GRID_HOME/crs/install/s_config_(hostname).txt. In my case file looked like this

cd /oragrid/product/11.2/crs/install
cat s_crsconfig_prod1.txt
<strong>TZ=UTC</strong>
NLS_LANG=AMERICAN_AMERICA.AL32UTF8
TNS_ADMIN=
ORACLE_BASE=

To resolve the issue we need to change TZ to US/Pacific on all nodes and restart clusterware. So entry would be like

<strong>TZ=US/Pacific</strong>

On Restarting clusteware , database and clusteware starts with correct timezone.

In case you wish to have different database timezone only for Oracle database, then it is possible using srvctl command. E.g

srvctl setenv database -d orcl -t TZ=US/Pacific

You can confirm this with getenv command

[oracle@prod1]~% srvctl getenv database -d orcl
orcl:
TZ=US/Pacific

This would require database bounce. Also note that in case database is started manually it would not start with correct timezone. To unset the parameter use following command

[oracle@prod1]~% srvctl unsetenv database -d orcl -t TZ
[oracle@prod1]~% srvctl getenv database -d orcl
orcl:

Hope this helps

11.2.0.3 Patchset is available

11.2.0.3 patchset is now available for download.  10404530 is available on Linux x86,x86-64 bit only. Tanel Poder first tweeted this

[blackbirdpie id=”117213822904840193″]

Soon after this Surachart and Martin have started upgrading to 11.2.0.3. I expect a blog post soon from one of them 🙂

[blackbirdpie id=”117242670719574016″]

[blackbirdpie id=”117245101268410369″]