In Search of Plan Stability

1. In
Search
of
Plan
Stability
Part
1
Karen
Morton
Sr.
Technical
Consultant
Now
part
of
Accenture

2. karen.morton@enkitec.com
karenmorton.blogspot.com
@karen_morton

3. Topics
• Define
and
idenFfy
plan
stability/instability
issues
• Using
SQL
Profiles
and
Patches
to
stabilize
regressed
plans
• IntroducFon
to
SQL
Plan
Management
(SPM)

4. Flexibility
vs.
Stability
• Plan
Flexibility
– Core
feature
of
Cost-­‐based
OpFmizer
CBO
• Ever-­‐changing
opFmal
plans
as
data
changes
• Plan
Stability
– Desired
feature
for
business-­‐criFcal
transacFons
• Plans
remain
constant
regardless
of
data
changes
• Both
have
Pros
and
Cons!
– Oracle
provides
several
features
for
both

5. Plan
Flexibility:
Good
or
Evil?
• The
mission
of
the
CBO
is
to
compute
an
opFmal
plan
for
a
given
SQL
and
its
data
• Plans
have
many
(and
complex)
dependencies
– HeurisFcs
– Query
predicates
– Schema
object
staFsFcs…and
so
on…
• The
CBO
is
successful
most
of
the
Fme
– Produces
a
sub-­‐opFmal
plan
some
of
the
Fme

6. The
Recurrent
Nightmare
• A
business
criFcal
processes
experiences
an
occasional
slow
down
– Same
SQL
usually
runs
quickly
– Developers
claim
they
haven't
changed
anything
– Users
claim
they
do
the
same
thing
every
Fme
– DBAs
claim
nothing
has
changed
with
the
"system"
– But…the
business
keeps
reporFng
a
problem
to
YOU!

7. Why
Plans
Perform
Inconsistently
(1)
• Concurrency
issue
with
some
other
process(es)
• State
of
the
buffer
cache
• CPU
starvaFon
• Sudden
changes
in
data
volume
• RAC
node
affinity
• Inconsistent
performance
through
database
links

8. Why
Plans
Perform
Inconsistently
(2)
• Downgraded
parallel
execuFon
• Crossing
small-­‐table
threshold
• Resource
Manager
direcFve
• Plan
"flips"
• Many
others…

9. Why
do
plans
"flip"?
• Bind
variable
peeking
(histograms)
• Outdated
object
staFsFcs
• Missing
object
staFsFcs
• New
object
staFsFcs
• New
empty
parFFons
• Database
parameter
changes
• System
staFsFcs
changes
• Incomplete
set
of
hints
• Dropped,
invalid,
invisible,
or
new
indexes
• Index
or
table
rebuild
• Two
or
more
plans
with
similar
cost

10. How
to
MiFgate
Plan
Flipping
• Solid
object
staFsFcs
• Default
CBO
parameters
• Avoid
global
changes
• Good
SQL
formulaFon

11. Flexibility
and
Stability
• Plan
Flexibility
– Cardinality
Feedback
(CF)
– AdapFve
Cursor
Sharing
(ACS)
– SQL
Tuning
Advisor
(STA)
Profiles
• Plan
Stability
– CBO
Hints
– Stored
Outlines
(SO)
– SQL
Plan
Management
(SPM)
Manually
created
SQL
Profiles
and
SQL
Patches

12. Plan
Stability
implementaFon
• 8:
CBO
Hints
• 9i:
Stored
Outline
(SO)
• 10g:
SQL
Profile
• 11g:
SQL
Patch
and
SQL
Plan
Baseline
(SPB)
• 12c:
SQL
Plan
Baseline

13. CBO
Hints
• Since
Oracle
8
• Reliable
for
the
most
part
• Can
be
cumbersome
• Requires
SQL
modificaFon
– There
are
some
workarounds
• Strict
Syntax
– No
syntax
errors
• Does
not
require
Oracle
Tuning
Pack

14. Stored
Outline
• Since
Oracle
9i
• Deprecated
in
11g
– Use
SQL
Plan
Management
(SPM)
instead
• CollecFon
of
CBO
hints
associated
to
one
SQL
• If
one
hint
is
ignored
others
would
sFll
apply
• Does
not
require
Oracle
Tuning
Pack

15. SQL
Profile
• Since
Oracle
10g
• SQL
Tuning
Advisor
(auto)
or
DBMS_SQLTUNE
(manual)
• CollecFon
of
one
or
more
CBO
hints
associated
to
one
SQL
• If
one
hint
is
ignored
others
would
sFll
apply
• Requires
Oracle
Tuning
Pack
LimitaFon:
Full
set
of
hints
needed
to
"guarantee"
plan
stability

16. SQL
Profile
Types
(1)
• Based
on
Scaling
CBO
Hints
(Flexibility)
– OPT_ESTIMATE:
fudge
factor
– TABLE_STATS:
blocks,
table
rows
– INDEX_STATS:
blocks,
index
rows,
keys,
clustering
factor
– COLUMN_STATS:
column
length,
NDV,
nulls,
min,
max
OPT_ESTIMATE(@"SEL$5DA710D3",
INDEX_FILTER,
"F"@"SEL$1",
IDX$$_1AA260002,
SCALE_ROWS=8.883203639e-­‐06)
OPT_ESTIMATE(@"SEL$5DA710D3",
INDEX_SKIP_SCAN,
"F"@"SEL$1",
IDX$$_1AA260002,
SCALE_ROWS=8.883203639e-­‐06)
OPT_ESTIMATE(@"SEL$5DA710D3",
JOIN,
("B"@"SEL$1",
"A"@"SEL$1"),
SCALE_ROWS=4.446153275)
OPT_ESTIMATE(@"SEL$5DA710D3",
JOIN,
("C"@"SEL$1",
"A"@"SEL$1"),
SCALE_ROWS=7.884506683)
OPT_ESTIMATE(@"SEL$5DA710D3",
TABLE,
"C"@"SEL$1",
SCALE_ROWS=11.39782103)

17. SQL
Profile
Types
(2)
• Based
on
Plan
Outline
(Stability)
/*+
BEGIN_OUTLINE_DATA
USE_HASH(@"SEL$58A6D7F6" "S"@"SEL$1")
LEADING(@"SEL$58A6D7F6" "C"@"SEL$1" "S"@"SEL$1")
FULL(@"SEL$58A6D7F6" "S"@"SEL$1")
FULL(@"SEL$58A6D7F6" "C"@"SEL$1")
OUTLINE(@"SEL$1")
OUTLINE(@"SEL$2")
MERGE(@"SEL$1")
OUTLINE_LEAF(@"SEL$58A6D7F6")
ALL_ROWS
OPT_PARAM('_fix_control' '8560951:1')
OPT_PARAM('_b_tree_bitmap_plans' 'false')
DB_VERSION('11.2.0.4')
OPTIMIZER_FEATURES_ENABLE('11.2.0.4')
IGNORE_OPTIM_EMBEDDED_HINTS
END_OUTLINE_DATA
*/
Basically,
just
a
set
of
hints…

18. SQL
Profile
Example
(1)
SELECT *
FROM SPONSOR S
JOIN CARRIER C ON C.OID = S.CARRIER_OID
WHERE S.SPONSOR_TYP = 'EMPLOYER'
AND (nvl(S.TEMPORARY_PROSPECT_IND, 0) = 0
or C.DISPLAY_TEMP_PROSPECTS_IND = 1)
AND S.UPPER_SPONSOR_NM LIKE :1
ORDER BY S.UPPER_SPONSOR_NM ;
sql_id
=
afvwm281hms4n
Performance
regression
ater
upgrade
to
11.2.0.4
from
11.2.0.1.
Pre-­‐upgrade
response
Fme
0-­‐1
second.

19. SQL
Profile
Example
(2)
Plan hash value: 2184022130
---------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
---------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 34172 | 53M| | 40307 (1)| 00:08:04 |
| 1 | SORT ORDER BY | | 34172 | 53M| 89M| 40307 (1)| 00:08:04 |
|* 2 | HASH JOIN | | 34172 | 53M| | 28571 (1)| 00:05:43 |
| 3 | TABLE ACCESS FULL| CARRIER | 764 | 355K| | 14 (0)| 00:00:01 |
|* 4 | TABLE ACCESS FULL| SPONSOR | 34190 | 38M| | 28557 (1)| 00:05:43 |
---------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("C"."OID"="S"."CARRIER_OID")
filter(NVL("S"."TEMPORARY_PROSPECT_IND",0)=0 OR
"C"."DISPLAY_TEMP_PROSPECTS_IND"=1)
4 - filter("S"."UPPER_SPONSOR_NM" LIKE :1 AND "S"."SPONSOR_TYP"='EMPLOYER')
ExecuFon
plan
(tables
had
histograms
on
all
appropriate
columns)
Response
Fme
15-­‐30
seconds

20. SQL
Profile
Example
(3)
Plan hash value: 1255407507
--------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
--------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 34172 | 53M| | 18215 (1)| 00:03:39 |
| 1 | SORT ORDER BY | | 34172 | 53M| 89M| 18215 (1)| 00:03:39 |
|* 2 | HASH JOIN | | 34172 | 53M| | 6480 (1)| 00:01:18 |
| 3 | TABLE ACCESS FULL | CARRIER | 764 | 355K| | 14 (0)| 00:00:01 |
|* 4 | TABLE ACCESS BY INDEX ROWID| SPONSOR | 34190 | 38M| | 6466 (1)| 00:01:18 |
|* 5 | INDEX RANGE SCAN | SPONSOR_NM_NUK | 6410 | | | 51 (0)| 00:00:01 |
--------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("C"."OID"="S"."CARRIER_OID")
filter(NVL("S"."TEMPORARY_PROSPECT_IND",0)=0 OR "C"."DISPLAY_TEMP_PROSPECTS_IND"=1)
4 - filter("S"."SPONSOR_TYP"='EMPLOYER')
5 - access("S"."UPPER_SPONSOR_NM" LIKE :1)
filter("S"."UPPER_SPONSOR_NM" LIKE :1)
1st
opFmizaFon
auempt:
removed
histograms
Response
Fme
~10
seconds

21. SQL
Profile
Example
(4)
select a.snap_id, to_char(b.begin_interval_time,'mm/dd/yyyy hh24:mi') snap_tm,
a.instance_number, a.plan_hash_value,
a.executions_delta execs,
round(a.disk_reads_delta/a.executions_delta,0) avg_pio,
round(a.buffer_gets_delta/a.executions_delta,0) avg_lio,
round(a.rows_processed_delta/a.executions_delta,0) avg_rows,
round((a.elapsed_time_delta/1000000)/a.executions_delta,0) avg_time,
round((a.cpu_time_delta/1000000)/a.executions_delta,0) avg_cpu,
round((a.iowait_delta/1000000)/a.executions_delta,0) avg_iowait,
round((a.clwait_delta/1000000)/a.executions_delta,0) avg_clwait
from dba_hist_sqlstat a, dba_hist_snapshot b
where a.sql_id = '&sql_id'
and a.plan_hash_value = nvl(&phv,a.plan_hash_value)
and a.executions_delta > 0
and a.elapsed_time_delta > 0
and a.snap_id = b.snap_id
and a.instance_number = b.instance_number
and a.dbid = b.dbid
and b.snap_id in (select snap_id from dba_hist_snapshot
where begin_interval_time >= trunc(sysdate) - &nodays)
order by a.snap_id desc, a.instance_number, a.plan_hash_value ;
Checked
AWR
for
"good"
previous
execuFon
plans

22. SQL
Profile
Example
(5)
Snap# Snap Time Inst# Plan HV Execs Avg PIO Avg LIO Avg Rows Avg Time Avg CPU Avg IO
------ ---------------- ------ ----------- ------ -------- -------- --------- --------- -------- -------
67800 08/04/2014 20:45 3 1422290831 5 15 5288 7 0 0 0
67671 08/03/2014 12:30 6 1422290831 2 1 5286 6 0 0 0
67667 08/03/2014 11:30 2 1422290831 4 0 2636 1 0 0 0
67634 08/03/2014 03:15 3 1422290831 1 7 5307 5 0 0 0
67593 08/02/2014 17:00 6 1422290831 6 0 2640 2 0 0 0
67571 08/02/2014 11:30 2 1422290831 5 1 4204 1 0 0 0
67568 08/02/2014 10:45 6 1422290831 19 55 4602 28 0 0 0
67567 08/02/2014 10:30 6 1422290831 14 22 4942 10 0 0 0
-----------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-----------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 7009 (100)| |
| 1 | NESTED LOOPS | | 34107 | 53M| 7009 (1)| 00:01:25 |
| 2 | TABLE ACCESS BY INDEX ROWID| SPONSOR | 34168 | 37M| 3590 (1)| 00:00:44 |
| 3 | INDEX RANGE SCAN | SPONSOR_NM_NUK | 35614 | | 27 (0)| 00:00:01 |
| 4 | TABLE ACCESS BY INDEX ROWID| CARRIER | 1 | 476 | 1 (0)| 00:00:01 |
| 5 | INDEX UNIQUE SCAN | CARRIER_PK | 1 | | 1 (0)| 00:00:01 |
-----------------------------------------------------------------------------------------------
Found
plan_hash_value
1422290831
from
prior
to
the
upgrade

23. SQL
Profile
Example
(6)
declare
ar_profile_hints sys.sqlprof_attr;
cl_sql_text clob;
l_profile_name varchar2(30);
begin
Step
1:
Get
the
set
of
hints
that
make
up
the
good
plan
from
AWR
in
dba_hist_sql_plan
select extractvalue(value(d), '/hint') as outline_hints
bulk collect
into ar_profile_hints
from xmltable('/*/outline_data/hint'
passing (
select xmltype(other_xml) as xmlval
from dba_hist_sql_plan
where sql_id = '&&sql_id'
and plan_hash_value = &&plan_hash_value
and other_xml is not null
)
) d;
Created
a
SQL
Profile
using
the
historically
good
plan
from
AWR

24. SQL
Profile
Example
(7)
select sql_text, 'PROF_fix_for_&&sql_id'
into cl_sql_text, l_profile_name
from dba_hist_sqltext
where sql_id = '&&sql_id';
dbms_sqltune.import_sql_profile(
sql_text => cl_sql_text,
profile => ar_profile_hints,
category => 'DEFAULT',
name => l_profile_name,
force_match => false
replace => true
);
end;
/
Step
2:
Get
the
SQL
text
for
this
SQL_ID
from
AWR
in
dba_hist_sqltext
Step
3:
Auach
the
good
plan
from
AWR
to
the
SQL
using
dbms_sqltune.import_sql_profile
Profile
now
in
place
named
PROF_fix_for_afvwm281hms4n

25. SQL
Profile
Example
(8)
-----------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-----------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 7009 (100)| |
| 1 | NESTED LOOPS | | 34107 | 53M| 7009 (1)| 00:01:25 |
| 2 | TABLE ACCESS BY INDEX ROWID| SPONSOR | 34168 | 37M| 3590 (1)| 00:00:44 |
| 3 | INDEX RANGE SCAN | SPONSOR_UPPER_NM_NUK | 35614 | | 27 (0)| 00:00:01 |
| 4 | TABLE ACCESS BY INDEX ROWID| CARRIER | 1 | 476 | 1 (0)| 00:00:01 |
| 5 | INDEX UNIQUE SCAN | CARRIER_PK | 1 | | 1 (0)| 00:00:01 |
-----------------------------------------------------------------------------------------------------
Note
-----
- SQL profile "PROF_fix_for_afvwm281hms4n" used for this statement
Execute
the
SQL
again
and
verify
the
profile
is
being
used

26. SQL
Patch
• Since
Oracle
11g
• DBMS_SQLDIAG
or
SQL
Repair
Advisor
– Available
as
part
of
Enterprise
EdiFon
• Also
DBMS_SQLDIAG_INTERNAL
• CollecFon
of
one
or
more
CBO
hints
associated
to
one
SQL
• If
one
hint
is
ignored
others
would
sFll
apply
LimitaFon:
Can
only
be
applied
to
outermost
query
block

27. begin
sys.dbms_sqldiag_internal.i_create_patch(
sql_text => 'select count(*), max(col1)
from (select * from my_view where col2 = 99)',
hint_text => 'gather_plan_statistics',
name => 'gps_test');
end;
/
SQL
Patch
Example
(1)
Easy
way
to
add
a
hint
to
a
SQL
statement
that
cannot
be
touched
directly.
Hint
is
applied
only
at
the
outermost
query
block.

28. SQL
Patch
Example
(2)
explain plan for
select count(*), max(col1)
from (select * from my_view where col2 = 99);
select * from table(dbms_xplan.display(format=>'basic +note'));
-------------------------------------------------------------
| Id | Operation | Name |
-------------------------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | SORT AGGREGATE | |
| 2 | TABLE ACCESS BY INDEX ROWID| MY_TABLE1 |
| 3 | INDEX RANGE SCAN | MY_TABLE1_IDX1 |
| 4 | TABLE ACCESS BY INDEX ROWID| MY_TABLE2 |
| 5 | INDEX UNIQUE SCAN | MY_TABLE2_PK |
-------------------------------------------------------------
Note
-----
- SQL patch "gps_test" used for this statement
Verify
plan
to
see
if
SQL
Patch
was
used.

29. Resources
• SQL
Profiles
– hup://docs.oracle.com/cd/E11882_01/server.112/e41573/
sql_tune.htm#PFGRF02605
– hup://kerryosborne.oracle-­‐guy.com/2009/04/oracle-­‐sql-­‐
profiles/
• SQL
Patches
– hups://blogs.oracle.com/opFmizer/entry/how_can_i_hint_a
– hups://blogs.oracle.com/opFmizer/entry/
addiFonal_informaFon_on_sql_patches

30. SQL
Plan
Management
(SPM)
• One
or
more
persistent,
opFmal
plans
per
SQL
• Plan
Stability
– Only
known
and
accepted
plans
can
be
executed
• Plan
Flexibility
– Capture
new
plans
and
evaluate
their
performance
"off-­‐line"
– New
plans
are
acknowledged
but
not
executed
unFl
"evolved"
Goal
Plan
stability
with
controlled
flexibility

31. SQL
Plan
Baselines
(1)
• A
set
of
plans
available
to
the
CBO
for
a
given
SQL
– IdenFfied
by
SQL
Handle
and
Signature
• Hash
funcFon
on
SQL
Text
• dbms_sqltune.sqltext_to_signature
– View
dba_sql_plan_baselines
• enabled
=
YES
• accepted
=
YES
• reproduced
=
YES

32. SQL
Plan
Baselines
(2)
• Stores
plan_hash_value
and
verifies
plan
can
be
reproduced
before
using
• Provides
stability
at
the
expense
of
Fme
to
review
and
evolve
new
plans
(automaFc
plan
evoluFon
available)
• Must
match
SQL
text
exactly
– No
opFon
for
force_matching
(as
available
with
SQL
Profiles)

33. Summary
(1)
• Plan
flexibility
and
plan
stability
need
to
be
managed
for
opFmal
performance
• SQL
Profiles
and
SQL
Patches
can
be
used
to
apply
one
or
more
hints
in
an
auempt
to
influence
or
"lock
in"
a
plan
• SQL
Profiles
can
use
force_matching
to
allow
mulFple
SQL
statements
to
use
the
same
profile
(if
SQL
differs
only
by
its
literals)
• SQL
Plan
Management
is
the
next
step
in
the
evoluFon
of
balancing
plan
flexibility
and
plan
stability

34. Summary
(2)
• SQL
Profiles
and
SQL
Patches
can
be
used
as
workarounds
– Need
to
idenFfy
root
cause
of
the
issue
and
correct
– Then
remove
profile
or
patch
• SQL
Plan
Management
is
a
soluFon
– When
no
other
alternaFve
is
viable

35. What's
ahead
in
Part
2
• How
SQL
Plan
Management
(SPM)
works
• How
SQL
Plan
Baselines
are
created
• How
SQL
Plan
Baselines
are
evolved/enabled
for
use
Tenta1vely
scheduled
for
Nov.
12

36. Thank
You!