2. In this Session
1. AAS
Single Metric
Shows DB Performance
Two methods to calculate
Sampling
Time statistics
1. Yardstick
Max CPU
CPU Count
To measure AAS against
1. Subcomponents
CPU
Waits
Time series
02/20/13 Copyright 2006 Kyle Hailey #.2
2
3. Database Performance
How quick can you find
Bottleneck in DB
If DB is idle
Current DB Load
what is DB Load ? what the *!####!
What do you use? *!*? is the
Statspack/AWR
database
V$active_session_history
Alerts
doing ?!
what do you alert on ?
02/20/13 #.3
3
4. Active Session
Database Machine
SGA Shadow
Shadow
Process Process
Shadow
Shadow Shadow
Process
Process Process
SQL* SQL*
Plus Plus
Application
Copyright 2006 Kyle Hailey #.4
5. Active Session
Shadow
Process
Database (shadow process)
idle Query 1 idle Query 2 idle Query 3 idle
Active Active Active
Time
Typing waiting typing waiting typing waiting Have a coffee
SQL*Plus (ie application)
Copyright 2006 Kyle Hailey #.5
6. Active Session
Database
User 1
User 2
User 3
User 4
=
4 Active Sessions
3
2
1
Graph represents # of sessions active, but also represents
amount of time active in the database
#.6
7. Active Session
4 Active Sessions
3
2
1
For Every Active Session there is a user (or application) waiting
1
2
3
4 Users Waiting
Copyright 2006 Kyle Hailey #.7
8. Measuring Active Sessions
Sampling Every Second
...
User 1
User 2
User 3
If happens a lot or for long … we’ll catch it, guaranteed
Fast query run often
Fast query run rarely
Slow query
Copyright 2006 Kyle Hailey #.8
10. Active Session
Database
idle Query 1 idle Query 2 idle Query 3 idle
To execute a query , some CPU will be used but we also might spend time
waiting for IO or on waiting for concurrency resources like latches
idle Query 1 idle Query 2 idle Query 3 idle
Database
CPU IO Wait
Activity can thus further be broken down into the type of activity: CPU, IO or WAIT
Latency
Work Contention
Copyright 2006 Kyle Hailey #.10
19. The Power ASH gives AAS
DB Home
Performance
Based on TIME AAS=db
(events Statistics) time/elapsed time
Top Activity
AAS=count active
Based on
users /samples
ASH
Copyright 2006 Kyle Hailey #.19
20. DB TIME = area under the curve
Height = # of Sessions
Width = seconds
Area under curve = DB Time
n
DB Time = active sessions(ti) * Δt
DB Time = sum of active time in database
Copyright 2006 Kyle Hailey #.20
21. AAS Sources
AAS = DB Time/Elapsed Time
1. Manually from
v$sysstat (9i : v$system_event )
1. Statspack
Need several calculations
1. AWR
One calculation
1. OEM 10g
Directly displayed
02/20/13 Copyright 2006 Kyle Hailey #.21
21
22. 1. Manually
AAS = DB TIME / Elapsed Time
DB Time (DBT) = Time Spent in Database
DB TIME (10g) = select value from v$sysstat
select value from v$sysstat
where name = ‘DB time’
where name = ‘DB time’;
‘DB time’;
DB TIME (9i) = Select sum(time_waited) from v$system_event
Select sum(time_waited) from v$system_event
where event not in (( ... idle events …);
where event not in ... idle events …);
+
+
Select value from v$sysstat
Select value from v$sysstat
where name = ‘CPU used by this session’;
where name = ‘CPU used by this session’;
still need to take delta values
Note: stats$idle_event : 70
v$event_name.wait_class=‘Idle’ :62
Copyright 2006 Kyle Hailey #.22
23. 2. Statspack AAS
Look for
Elapsed Time
Top 5 Timed Events
Start at line 52 of about 1300
02/20/13 Copyright 2006 Kyle Hailey #.23
23
24. 2. Statspack AAS
Elapsed Time
STATSPACK report for
STATSPACK report for
LABSF03 1420044432
LABSF03 1420044432 labsf03
labsf03 1 10.1.0.2.0 NO labsfr
1 10.1.0.2.0 NO labsfr
Snap Id
Snap Id Snap Time
Snap Time Sessions Curs/Sess
Sessions Curs/Sess
Begin Snap:
Begin Snap: 1 03-Apr-06 12:34:06
1 03-Apr-06 12:34:06 18
18 5.6
5.6
End Snap:
End Snap: 2 03-Apr-06 12:34:36
2 03-Apr-06 12:34:36 18
18 4.8
4.8
Elapsed:
Elapsed: 1.00 (mins)
1.00 (mins)
Look at Top 5 Timed Events
Top 5 Timed Events
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~ % Total
% Total
Event
Event Waits
Waits Time (s)
Time (s) Call Time
Call Time
buffer busy waits
buffer busy waits 2,748
2,748 250
250 78.72
78.72
CPU time
CPU time 32
32 10.16
10.16
free buffer waits
free buffer waits 1,588
1,588 15
15 4.63
4.63
write complete waits
write complete waits 10
10 88 2.51
2.51
log buffer space
log buffer space 306
306 55 1.51
1.51
02/20/13 Copyright 2006 Kyle Hailey #.24
24
25. 2. Statspack AAS
Top 5 Timed Events
Top 5 Timed Events
Event Time (s)
DBTIME= CPU + WAITS Event
-----------------
Time (s)
----------------- -----
-----
CPU = 32 buffer busy waits
buffer busy waits 250
250
CPU time
CPU time 32
32
WAITS = 250+15+8+5 = 278 secs free buffer waits 15
free buffer waits 15
DBTIME=320 write complete waits
write complete waits 88
log buffer space 5
Elapsed Time = 60 secs log buffer space 5
320 secs / 60 secs
AAS = 5.1
02/20/13 Copyright 2006 Kyle Hailey #.25
25
26. 3. AWR Report
AAS = DB Time/Elapsed Time
23.56/59.66 = 0.39
AAS= 0.39
02/20/13 #.26
26
28. Got AAS, Now What ?
We Need one more item:
CPU Count
# of CPUs available on System
Shared with other applications
Need to track CPU used on the system as well
On dual & quad cores, lower the CPU count
Represents max active sessions that can do work
02/20/13 Copyright 2006 Kyle Hailey #.28
28
29. CPU Count
# of CPUs available in
Statspack 10g
AWR report
OEM 10g
Statspack 9i # of CPUs missing # of CPUs
SQLPLUS> show parameters cpu_count
SQLPLUS> show parameters cpu_count
NAME
NAME VALUE
VALUE
------------------ ----------
------------------ ----------
cpu_count
cpu_count 22
02/20/13 Copyright 2006 Kyle Hailey #.29
29
30. What’s the DB Doing?!
It’s 2am … your manager calls
Whip out the stethoscope:
AAS
what the *!####!*!
*? is the database
doing ?!
02/20/13 Copyright 2006 Kyle Hailey #.30
30
31. AAS Formulas
Use CPU count as yardstick:
AAS < 1 Ideal world –
Database is not blocked one database
AAS ~= 0 solution
Database basically idle
Problems are in the APP not DB
track CPU at OS
AAS < # of CPUs
CPU available
Are any single sessions 100% active?
AAS > 1
AAS > # of CPUs still want to know
Could have performance problems if a single user is
AAS >> # of CPUS 100% active
There is a bottleneck
02/20/13 Copyright 2006 Kyle Hailey #.31
31
32. Available CPU vs AAS
Statspack
DBTIME= CPU + WAITS AAS = 5.1
AAS = 5.1 AAS far above
CPU = 32
WAITS = 250+15+8+5 = 278 secs available CPU
DBTIME=320
Elapsed Time = 60 secs
# of CPU = 2
# of CPU = 2 => problem
320 secs / 60 secs
AAS = 5.1
AWR Report AAS = 0.39
AAS = 0.39
# of CPU = 2
# of CPU = 2
AAS < 1 , database is fine
AAS = 0.75
AAS = 0.75
# of CPU = 2
# of CPU = 2
AAS < 1 , database is fine
02/20/13 Copyright 2006 Kyle Hailey #.32
32
33. Going Farther with AAS
AAS can tell you a lot
But it’s components tell you much more
To go farther need the components of AAS
1. CPU
2. Wait
3. Value over time
Only OEM 10g shows the value over time, Statspack and AWR
are aggregated over the snapshot period
02/20/13 Copyright 2006 Kyle Hailey #.33
33
34. EM DB Home Page
Copyright 2006 Kyle Hailey #.34
35. OEM 10g Perf Pages
DB Home
AAS Point in Time
Performance
AAS over Time
Copyright 2006 Kyle Hailey #.35
36. AAS Components : OEM 10g
OEM 10g
Performance Page
Real CPU available:
Max CPU - non instance CPU
Available CPU AAS:
CPU + WAIT
02/20/13 #.36
36
37. OEM 10g
Looks
Relax Get to Work! OK
But …
02/20/13 #.37
37
38. Idle Database – Perf Page
Value of proving the database is
Idle
It’s the Databases Fault
How many times do you hear that?
Database Idle
No load on database
Database “performance” is fine
Under utilized
Problem lies elsewhere
Saved me time and stress many
times
And weeks of debate about where the
problem is coming from
Copyright 2006 Kyle Hailey #.38
39. More than AAS and #CPU
Knowing your DB Profile
Copyright 2006 Kyle Hailey #.39
40. When to Tune
1. Machine
a) CPU
Response times skewed
100% CPU might be fine
Users wait in queue (run queue) => machine
underpowered
a) Memory
Paging
Wait times skewed (ex : latch free)
Erratic response times ( ex : ls )
1. Oracle Host CPU
Memory
1) Waits > CPU ? Oracle Load
(AAS)
tune waits AAS >
#CPU
1) CPU > 100% ? Waits > CPU >
CPU
tune top CPU SQL AAS > 1
Top Session
Waits
Top Wait Top SQL
1) Else
It’s the application
Object Detail SQL Detail Wait Detail Session Detail File Detail
#.40
Copyright 2006 Kyle Hailey
41. Limited Analysis
What if you find a problem ?
Of the 800 waits which in order to solve most need to know
What SQL
Which sessions
Values of P1, P2 and P3
Statspack and AWR fail
AAS = DB TIME / Elapsed Time
But there is another way …
02/20/13 #.41
41
42. AAS based on ASH
ASH - Active Session History
v$active_session_history
Samples sessions once a second
AAS = count(*) / elapsed_seconds
A statistical approximation, but surprisingly close
ASH data source empowers drilldowns
Top Sql
Top Waits
Details p1,p2,p3 and more
02/20/13 Copyright 2006 Kyle Hailey #.42
42
43. ASH
vs
Statistics
Statistics
are more expensive
have lag time
lack clear identification of culprits
Copyright 2006 Kyle Hailey #.43
44. Statistic Lag Time
Samples (ASH)
Slight Lags
Statistics
Copyright 2006 Kyle Hailey #.44
45. CPU Lag Problem
ASH is the only way to see CPU usage realtime
V$sysstat reports CPU but
isonly updated at the end of the call.
Long calls look deceiving like no CPU is being used
Time Model also reports CPU
Updated quicker
Copyright 2006 Kyle Hailey #.45
46. CPU in ASH vs Stats
Copyright 2006 Kyle Hailey #.46
47. 2. OEM 10g : Top Activity
•Top Activity
•Based on ASH
•Enables Drilldowns
•Top SQL
•Top Session
•Drill into a session
• Stats
• Raw waits
• Open cursors
• General info
•Drill into a SQL
• Stats and text
• Users executing
• Explain plan
• Tuning options
02/20/13 #.47
47
48. Top Activity : Based on ASH
missing
Thanks
To
ASH
Copyright 2006 Kyle Hailey #.48
49. AAS – %Session Time Issue
Whenever AAS > 1
I want to know if any
one session is 100% active
Unfortunately OEM isn’t set up for this
(ASHMON, my free tool, is )
Copyright 2006 Kyle Hailey #.49
50. Top Activity: ASH Sessions
Many Users Active
On Performance Page, no way to tell how many users
Copyright 2006 Kyle Hailey #.50
51. Top Activity: ASH Sessions
Two Users Active
Shown in % DB Time
Missing % Session Time
Copyright 2006 Kyle Hailey #.51
52. OEM 10g Perf Pages
DB Home
Top Activity
Performance
Top Activity
SQL
Session
SQL
Session
Copyright 2006 Kyle Hailey #.52
55. AAS from ASH
1. ASHRPT
Based entirely on
v$active_session_history
@?/rdbms/admin/ashrpt.sql
Exec ASH_REPORT_TEXT/HTML
select * from table
(dbms_workload_repository.ash_report_text(
(select dbid from v$database),
1,
sysdate – 1/24,
sysdate )) ;
02/20/13 Copyright 2006 Kyle Hailey #.55
55
56. 1. ASHRPT
ASH Report For TESTDB/testdb
ASH Report For TESTDB/testdb
DB Name DB Id Instance Inst Num Release RAC Host
DB Name DB Id Instance Inst Num Release RAC Host
TESTDB 2371570538 testdb 11 10.2.0.1.0 NO sdbe604a
TESTDB 2371570538 testdb 10.2.0.1.0 NO sdbe604a
CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size
CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size
22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)
1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)
Analysis Begin Time: 21-Apr-06 12:00:01
Analysis Begin Time: 21-Apr-06 12:00:01
Analysis End Time: 21-Apr-06 12:05:01
Analysis End Time: 21-Apr-06 12:05:01
Elapsed Time: 5.0 (mins)
Elapsed Time: 5.0 (mins)
Sample Count: 3,716
Sample Count: 3,716
Average Active Sessions: 12.39
Average Active Sessions: 12.39
Avg. Active Session per CPU: 6.19
Avg. Active Session per CPU: 6.19
Report Target: None specified
Report Target: None specified
Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)
Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)
Avg Active
Avg Active
Event Event Class %% Activity
Event Event Class Activity Sessions
Sessions
CPU ++ Wait for CPU CPU 67.98 8.42
CPU Wait for CPU CPU 67.98 8.42
enq: TX -- row lock contention Application 23.98 2.97
enq: TX row lock contention Application 23.98 2.97
buffer busy waits Concurrency 4.66 0.58
buffer busy waits Concurrency 4.66 0.58
latch: cache buffers chains Concurrency 2.26 0.28
latch: cache buffers chains Concurrency 2.26 0.28
02/20/13 Copyright 2006 Kyle Hailey #.56
56
57. 1. ASH RPT
1) General info
1) General info 9) Top SQL using literals
9) Top SQL using literals
2) Top User Events ***
2) Top User Events *** 10) Top Sessions ***
10) Top Sessions ***
3) Top Background Events
3) Top Background Events 11) Top Blocking Sessions
11) Top Blocking Sessions
4) Top Event P1/P2/P3 Values
4) Top Event P1/P2/P3 Values 12) Top Sessions running PQs
12) Top Sessions running PQs
5) Top Service/Module
5) Top Service/Module 13) Top DB Objects
13) Top DB Objects
6) Top Client IDs
6) Top Client IDs 14) Top DB Files
14) Top DB Files
7) Top SQL Command Types
7) Top SQL Command Types 15) Top Latches
15) Top Latches
8) Top SQL Statements ***
8) Top SQL Statements *** 16) Activity Over Time ***
16) Activity Over Time ***
02/20/13 Copyright 2006 Kyle Hailey #.57
57
58. 1. ASHRPT over Time
Waits over Time
Not in AAS
Difficult but better
than nothing
Compare to …
02/20/13 #.58
58
59. 3. Custom Scripts
Hate Graphics ?
Query v$active_session_history directly
Join to dba_hist_active_sess_history for week of data
act.sql
Like top 5 timed events
Aveact.sql
Charts with text AAS by hour (15 minute, minute , etc)
Aveactn.sql
Ditto,
with top 2 wait events per bucket
Following Scripts Available on
http://perfvision.com/ashscripts.php
02/20/13 #.59
59
60. 3. Custom Scripts
@act
@act
Analysis Begin Time :
Analysis Begin Time : 2007-07-24 11:04:48
2007-07-24 11:04:48
Analysis End
Analysis End Time :
Time : 2007-07-24 11:19:45
2007-07-24 11:19:45
Start time, mins ago:
Start time, mins ago: 15
15
Request Duration
Request Duration :: 15
15
Collections
Collections :: 528
528
Data Values
Data Values :: 3327
3327
Elapsed Time: 15 mins
Elapsed Time: 15 mins
WAIT_EVENT
WAIT_EVENT CNT
CNT % Active Ave_Act_Sess
% Active Ave_Act_Sess
latch free
latch free 10
10 .3
.3 .02
.02
log buffer space
log buffer space 13
13 .39
.39 .02
.02
buffer busy waits
buffer busy waits 14
14 .42
.42 .03
.03
db file scattered read
db file scattered read 15
15 .45
.45 .03
.03
library cache pin
library cache pin 78
78 2.34
2.34 .15
.15
log file sync
log file sync 213
213 6.40
6.40 .40
.40
ON CPU
ON CPU 726
726 21.82
21.82 1.38
1.38
enqueue
enqueue 855
855 25.70
25.70 1.62
1.62
db file sequential read
db file sequential read 1399
1399 42.05
42.05 2.65
2.65
sum
sum 6.30
6.30
02/20/13 Copyright 2006 Kyle Hailey #.60
60
65. Summary
AAS: Two Sources Host CPU
Memory
1. v$system_event & v$sysstat Oracle Load
(AAS)
Performance page in OEM
Statspack, awr report AAS >
Indirect – based on time converted to AAS #CPU
Accurate
Lags (especially CPU) AAS > Waits > CPU >
1 CPU Waits
Limits analysis
Top Session Top Wait Top SQL
1. v$active_session_history
Top activity in OEM
ashrpt
Direct measure of AAS Object Detail SQL Detail Wait Detail Session Detail File Detail
Real time
Approximation
***Allows drilldowns***
SQL Tuning
Tuning: ADDM Advisor
Machine first – CPU, Memory
Oracle
AAS ~= 0 Idle
AAS > 1 : possible sessions blocked
AAS > # CPU : bottleneck
02/20/13 Copyright 2006 Kyle Hailey #.65
65
66. Q1
What is the easiest way to find the load on the database
b. Average Active Sessions takes into
account all the major criteria in Oracle
a. top 5 timed eventsperformance - cpu usage, wait time, elapsed
time
b. Average Active Sessions and CPU count
others:
c. The transaction rate
d. The commit rate a. lacks the elapsed time
c & d - the amount of load created by
commits and transactions varies drastically
between database to database and even
between different loads on a database during
the day
Copyright 2006 Kyle Hailey #.66
67. Q2
Average active sessions, the measurement in the
performance chart in OEM 10g can be calculated by which
formula(s)
a. db time / elapsed time
b. wait time / elapsed time
answer
c. count of rows in v$active_session_history over an
a,c,d
interval/ number of samples in interval
c, d are equivalent because ASH samples
d. count of rows in v$active_session_history / seconds of
once a second
elapsed time
others
b doesn't work because it is missing CPU
time
Copyright 2006 Kyle Hailey #.67
68. Q3
If Average Active Session is less than one, we can say
a. the database is using less than 100% CPU on machine
b. the database is using 100% CPU
c. No session is completely blocked
d. Database performance should be acceptable
answer
a, c and d
Copyright 2006 Kyle Hailey #.68
69. Q4
An Average Active Session value of zero
means
a. the database is down
b. the database is hung
c. the database is idle
d. the database is overloaded
answer
c
Copyright 2006 Kyle Hailey #.69
70. Q5
CPU is updated in v$sysstat
a. when call finishes
b. every second
c. every five seconds
d. in real time
a - which can be a problem in
monitoring tools if the call last a long
time showing no CPU usage until the
call finishes, then reporting an
unrealistic spike of CPU when the
call finishes
Copyright 2006 Kyle Hailey #.70
Hinweis der Redaktion
adf
Statspack is probably the most reliable source of performance information. The statspack report, ?/rdbms/admin/spreport.sql , generates over 1000 lines of information, but the first and possible only place to go in the report is “Top 5 Timed Events”. In “Top 5 Timed Events” we can determine if the database has any performance issues. If it does have performance issues, then we can find out if it’s CPU or a wait. If it’s a wait we can tune that particular wait.
If the users call up saying the database is hanging and AAS < 1 you know it’s not true. If AAS is near 0 you even know that the database is idle and that the users or application is not requesting any work from Oracle
OEM DB Home page only shows AAS at a point in time which is limited. Click on the performance page tab to get a time line view.
In order to tune an Oracle database the first step in a complete analysis is to verify the machine because there are a couple of factors that can only be clearly determined by looking at machine statistics. Those two factors are Memory Usage CPU Usage Memory and CPU problems will have tell tale repercussions on Oracle performance statistics and thus can be deduced from just looking at the Oracle statistics, but it is clear just to start with the machine statistics. CPU For CPU, we check the “run queue” which is the number of processes that are ready to run but have to wait for the CPU. A machine free of CPU contention would have a run queue of 0 and could have CPU usage near 100% at the same time. A high CPU usage can be a good sign that the system is being utilized fully. On the other hand a high run queue will indicated that there is more demand for CPU than CPU power available. High run queue can be determined via Oracle statistics by looking at ASH data and seeing if more sessions are marked as being on the CPU than the number of CPUs available. For example if there are 4 sessions average active on CPU in the ASH data and only 2 CPUs then the machine is CPU bound. Solutions for high run queue are either to add more processors or reduce the load on the CPU. If the CPU is mainly being used by Oracle, then that is going to mean tuning the application and ther SQL queries. Memory If the machine is paging out to disk it means there is a memory crunch and can dramatically slow down Oracle. Oracle will sometimes indicate a paging problem through a spike in “latch free” waits but the only guarenteed method of diagnosing this problem is looking at the machine statistics. Machines have statistics for paging and free memory. Often there can be some free memory even when there is paging out because machines start paging out before memory is completely filled. Solutions if the machine is paging out are either to add more memory or to reduce memory usage. Memory usage can be reduced by reducing Oracle cache sizes or reducing Oracle session memory usage.
12 ==================================================================== Chapt 1 ASH CPU is updated in v$sysstat a. time call finishes b. every second c. every five seconds d. in real time answer a - which can be a problem in monitoring tools if the call last a long time showing no CPU usage until the call finishes, then reporting an unrealistic spike of CPU when the call finishes 7 ==================================================================== Chapt 1 ASH Which view can you query directly to get the specific waits for that occurred 30 minutes ago: a. v$active_session_history b. v$waitclassmetric c. v$system_event d. v$session_wait_history e. v$waitclassmetric_history answer a only others b - last 60 seconds only c - cumulative info since db startup d - last 10 waits only e - only wait groups, not wait events, but has the history for last hour 8 ==================================================================== Chapt 1 ASH ASH (v$active_session_history) is a revolutionary data source for monitoring and analyzing database performance. The view v$active_session_history is new in 10g, but most of the data needed in order to simulate v$active_session_history by hand has been available since which version a. 6 b. 7 c. 8 d. 9 e. 10 answer b - since version 7 when wait events were introduced along with the view v$session_wait which is the foundation for ASH 4 ==================================================================== Chapt 1 ASH How can immediate find the top IO consuming SQL statement in the last 60 seconds a. v$active_session_history b. v$sqlstats c. v$sql d. v$sqlarea answer a only others b,c,d - only have cumulative values since database startup