Instance and SQL tuning with EM12c Cloud Control is so easy, it is not even much fun anymore. Also, not every customer may have the appropriate license or database edition, or all you have available remotely is a command-line login to a database. This presentation showcases a few open-source database tuning tools such as Snapper and ASH replacements that DBAs can use to gather and review metrics and wait events from the command line and even in standard edition.

1. hitchhikers guide to free
Oracle tuning tools
a quick guide
Björn Rost

2. © 2015 Pythian Confidential2

3. • Consultant
– Oracle Database HA, Tuning, etc
– Linux&Solaris automation
• Oracle ACE Director
• (ex) president RAC SIG
• organizer OTN EMEA tour
© 2016 Pythian3
ABOUT ME

4. ABOUT PYTHIAN
Pythian’s 400+ IT professionals
help companies adopt
and manage disruptive
technologies to better compete
© 2016 Pythian. Confidential 4

5. © 2016 Pythian. Confidential 5
Systems currently
managed by Pythian
EXPERIENCED
Pythian experts
in 35 countries
GLOBAL
Millennia of experience
gathered and shared over
19 years
EXPERTS
11,800 2400

6. © 2016 Pythian. Confidential 6
TECHNICAL EXPERTISE
CLOUD
ADVANCED
ANALYTICS
DATABASES
BIG DATA
DEVOPS
Using the disruptive nature
of cloud for accelerated,
cost-effective growth
INFRASTRUCTURE
MANAGEMENT
Mining data for insights &
business transformation
using data science
Ensuring databases are
reliable, secure, available
and continuously optimized
Harnessing the
transformative power of
data on a massive scale
Providing critical velocity in
software deployment by
adopting DevOps practices
Transforming and managing
the IT infrastructure that
supports the business

7. WHY ORACLE?
7
• basic operations same
between DBs
• instrumentation makes the
difference
• allows to deeply investigate
and explain issues
© 2016 Pythian

8. (WRONG) TUNING CYCLE
© 2016 Pythian8
guess
change
hope

9. DON’T BE A STORMTROOPER DBA
© 2016 Pythian9

10. TUNING CYCLE
© 2016 Pythian10
understand
change
measure

11. TUNING IS SIMPLE, REALLY
11 © 2016 Pythian

12. LAST WEEK’S EXAMPLE
• increasing gap in standby db
– change max_archive_processes parameter
– change some other parameters
– try rman recovery vs MRP
– increase parallelism
– reduce parallelism
– try more silver bullets…
– have SRE look into IO issues
– have SRE look at CPU issues
12 © 2016 Pythian
guess
change
hope

13. LAST WEEK’S EXAMPLE
• select event from v$session
where program like '%MRP%’
– PX deq credit: send blkd
– PX deq credit: need buffer
• search for that on MOS
– 10.2 Parallel Standby Recovery Process Is Very Slow (Doc ID 420337.1)
• PARALLEL_EXECUTION_MESSAGE_SIZE=16k
13 © 2016 Pythian
understand
change
measure

14. TYPES
14
report real-time historic drill-down
© 2016 Pythian

15. ENTERPRISE MANAGER
• incredibly useful and thought through
• from vendor, perfectly integrated
• many common views and methods
– coloured AAS view
– Top-SQL view
– Real-Time SQL monitor
– blocking session tree
– AWR Reports
– ADDM and other advisors
16 © 2016 Pythian

16. ENTERPRISE MANAGER
© 2015 Pythian Confidential17

17. SO WHY USE CLI TOOLS?
• not licensed for Diag&Tuning
– Standard Edition
– EE without Diag&Tuning (a cardinal sin)
• very low throughput or no GUI access
• EM not setup/installed
– 12c EM express can get the job done
• you simply want to look like a nerd
18 © 2016 Pythian

18. WHY OPEN SOURCE
• see and learn
– you can actually look at the code you are running
• collaborate and contribute
– want a new feature? implement yourself!
• free(*) means
– no costs (duh)
– get up and running really fast
– don’t worry about audits
19 © 2016 Pythian

19. © 2015 Pythian Confidential20

20. RLSQLPLUS
• sqlplus on steroids
• edit sqlplus CLI
• history
– up/down arrow
– search with CTRL-r
• auto-completion
– against custom wordlist
21 © 2016 Pythian

21. RLSQLPLUS INSTALLATION STEP1
22
[root@kickpuppet ~]# yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/rlwrap-0.42-1.el6.x86_64.rpm
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
rlwrap x86_64 0.41-1.el6 epel 92 k
Transaction Summary
================================================================================
Install 1 Package(s)
Installing : rlwrap-0.41-1.el6.x86_64 1/1
Verifying : rlwrap-0.41-1.el6.x86_64 1/1
Installed:
rlwrap.x86_64 0:0.41-1.el6
Complete!
© 2016 Pythian

22. RLSQLPLUS INSTALLATION STEP2
• optional: download or generate wordfile
– for autocompletion
23
$ curl -o wordfile.txt
http://blog.tanelpoder.com/files/scripts/setup/wordfile_11gR2.txt
http://blog.tanelpoder.com/files/scripts/setup/
© 2016 Pythian

23. RLSQLPLUS INSTALLATION STEP 3
• create aliases around sqlplus
– and rman
– and dgmgrl
• avoid overriding the original name
24
alias rlsqlplus='rlwrap -D2 -irc -b'''"@(){}[],+=&^%#;|''' -f ~/wordfile.txt
sqlplus'
alias rlrman='rlwrap -D2 -irc -b'''"@(){}[],+=&^%#;|''' rman'
© 2016 Pythian

24. SQLCL
• sqlplus “replacement” from SQLdeveloper team
– history and editor
– auto-completion
– auto-formatting
• ansiconsole for col widths
• json, xml, csv etc
– tons of good stuff
25 © 2016 Pythian

25. © 2015 Pythian Confidential26

26. AAS VIEW
• my favourite real-time DB dashboard
• one-glance DB activity overview
• colour-coded
– red -> blocking, app, concurrency
– blue -> IO, missing indexes, etc
– green -> CPU, executing too much?
– other -> weird, take a look
27 © 2016 Pythian

27. AAS VIEW
28 © 2016 Pythian

28. WEB-ASH
29
• simple and free
• reads v$session by default
• stores data in plain files
• lean javascript to display
http://datavirtualizer.com/web-ash-w-ash/
© 2016 Pythian

29. MOATS
• top-like output that refreshes
• excellent overview
• needs small installation on database
• samples v$-views while running
• improved forks
– MOATS 2.0
– SQL Dashboard
30 © 2016 Pythian

30. MOATS OUTPUT
31
MOATS: The Mother Of All Tuning Scripts v1.0 by Adrian Billington & Tanel Poder
http://www.oracle-developer.net & http://www.e2sn.com
+ INSTANCE SUMMARY ------------------------------------------------------------------------------------------+
| Instance: KICKPUPPET | Execs/s: 581.6 | sParse/s: 60.0 | LIOs/s: 4989.5 | Read MB/s: 0.7 |
| Cur Time: 19-Nov 21:11:09 | Calls/s: 164.2 | hParse/s: 0.0 | PhyRD/s: 86.7 | Write MB/s: 1.6 |
| History: 0h 9m 45s | Commits/s: 50.3 | ccHits/s: 578.8 | PhyWR/s: 112.7 | Redo MB/s: 0.2 |
+------------------------------------------------------------------------------------------------------------+
+ TOP SQL_ID (child#) -----+ TOP SESSIONS ---------+ + TOP WAITS -------------------------+ WAIT CLASS -+
| 29% | 7t0959msvyt5g (0) | 78,65 | | 29% | db file sequential read | User I/O |
| 14% | () | | | 29% | ON CPU | ON CPU |
| 14% | 147a57cxq3w5y (0) | 70 | | 14% | oracle thread bootstrap | Other |
| 14% | gh2g2tynpcpv1 (0) | 65 | | 14% | resmgr:cpu quantum | Scheduler |
| 14% | 4phvdvx32a3mf (0) | 73 | | 14% | db file parallel read | User I/O |
+--------------------------------------------------+ +--------------------------------------------------+
+ TOP SQL_ID ----+ PLAN_HASH_VALUE + SQL TEXT ---------------------------------------------------------------+
| 7t0959msvyt5g | 856749079 | SELECT ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORD |
| | | ER_TOTAL, SALES_REP_ID, PROMOTION_ID, WAREHOUSE_ID, DELIVERY_TYPE, COST |
+ ---------------------------------------------------------------------------------------------------------- +
| 147a57cxq3w5y | 0 | BEGIN :1 := orderentry.browseproducts(:2 ,:3 ,:4 ); END; |
+ ---------------------------------------------------------------------------------------------------------- +
| gh2g2tynpcpv1 | 0 | INSERT INTO CUSTOMERS ( CUSTOMER_ID , CUST_FIRST_NAME , CUST_LAST_NAME |
| | | , NLS_LANGUAGE , NLS_TERRITORY , CREDIT_LIMIT , CUST_EMAIL , ACCOUNT_MG |
+ ---------------------------------------------------------------------------------------------------------- +
| 4phvdvx32a3mf | 0 | begin prvt_ilm.stopjobs(-1,true,true,:1); end; |
+ ---------------------------------------------------------------------------------------------------------- +
© 2016 Pythian

31. MOATS 2.0 OUTPUT
32 © 2016 Pythian

32. SNAPPER
• realtime session monitor and drill-downs
• just one sqlscript to run
– no installation needed
• runs for sample period
– “snaps” v$session etc
– reports on diffs
• two modes
– “ash” for v$session
– stats for v$sesstat
33
http://tech.e2sn.com/oracle-scripts-and-tools/session-snapper
© 2016 Pythian

33. SNAPPER ARGUMENTS
• snapper ash 5 1 all
– mode can be ash, stat or custom column list
– seconds to run/sample
– number of runs
– “all” or filter on v$session columns
• sid=42
• username=brost
34 © 2016 Pythian

34. SNAPPER DEFAULT MODE
35
SQL> @snapper ash 5 1 all
Sampling SID all with interval 5 seconds, taking 1 snapshots...
-- Session Snapper v4.15 - by Tanel Poder ( http://blog.tanelpoder.com )
-- Enjoy the Most Advanced Oracle Troubleshooting Script on the Planet! :)
----------------------------------------------------------------------------------------------------
Active% | INST | SQL_ID | SQL_CHILD | EVENT | WAIT_CLASS
----------------------------------------------------------------------------------------------------
10% | 1 | | 0 | db file async I/O submit | System I/O
10% | 1 | | | log file sync | Commit
7% | 1 | 5ckxyqfvu60pj | 0 | db file sequential read | User I/O
7% | 1 | | 0 | log file parallel write | System I/O
5% | 1 | | | ON CPU | ON CPU
5% | 1 | 3fw75k1snsddx | 0 | db file sequential read | User I/O
5% | 1 | | 0 | ON CPU | ON CPU
5% | 1 | c13sma6rkr27c | 0 | db file parallel read | User I/O
5% | 1 | gzhkw1qu6fwxm | 0 | ON CPU | ON CPU
5% | 1 | 0w2qpuc6u2zsp | 0 | ON CPU | ON CPU
-- End of ASH snap 1, end=2016-04-24 15:24:37, seconds=5, samples_taken=42
© 2016 Pythian

35. AAS IN SNAPPER
36
SQL> @snapper ash=wait_class 5 1 all
Sampling SID all with interval 5 seconds, taking 1 snapshots...
-- Session Snapper v4.11 - by Tanel Poder ( http://blog.tanelpoder.com )
-- Enjoy the Most Advanced Oracle Troubleshooting Script on the Planet! 
-------------------------
Active% | WAIT_CLASS
-------------------------
44% | System I/O
19% | Commit
7% | Scheduler
7% | ON CPU
5% | User I/O
-- End of ASH snap 1, end=2014-11-19 16:23:06, seconds=5, samples_taken=43
© 2016 Pythian

36. SNAPPER STATS MODE
37
SQL> @snapper stats,gather=tw 15 1 sid=70
Sampling SID sid=70 with interval 15 seconds, taking 1 snapshots...
-- Session Snapper v4.11 BETA - by Tanel Poder ( http://blog.tanelpoder.com ) - Enjoy the Most Advanced Oracle Troubleshooting Script on the Planet!
:)
---------------------------------------------------------------------------------------------------------------------------------------------------
SID, USERNAME , TYPE, STATISTIC , DELTA, HDELTA/SEC, %TIME, GRAPH , NUM_WAITS, WAITS/SEC, AVERAGES
---------------------------------------------------------------------------------------------------------------------------------------------------
70, SOE , TIME, repeated bind elapsed time , 300, 19.8us, .0%, [ ], , ,
70, SOE , TIME, parse time elapsed , 9272, 611.99us, .1%, [ ], , ,
70, SOE , TIME, PL/SQL execution elapsed time, 230999, 15.25ms, 1.5%, [# ], , ,
70, SOE , TIME, DB CPU , 569000, 37.56ms, 3.8%, [@ ], , ,
70, SOE , TIME, sql execute elapsed time , 1730975, 114.25ms, 11.4%, [## ], , ,
70, SOE , TIME, DB time , 2038543, 134.55ms, 13.5%, [## ], , , .88 % unaccounted
70, SOE , WAIT, log file sync , 235408, 15.54ms, 1.6%, [W ], 183, 12.08, 1.29ms average
70, SOE , WAIT, db file sequential read , 1123224, 74.14ms, 7.4%, [W ], 397, 26.2, 2.83ms average
70, SOE , WAIT, db file parallel read , 52763, 3.48ms, .3%, [ ], 3, .2, 17.59ms average
70, SOE , WAIT, latch: In memory undo latch , 499, 32.94us, .0%, [ ], 1, .07, 499us average
70, SOE , WAIT, resmgr:cpu quantum , 51794, 3.42ms, .3%, [ ], 3, .2, 17.26ms average
70, SOE , WAIT, SQL*Net message to client , 1458, 96.23us, .0%, [ ], 393, 25.94, 3.71us average
70, SOE , WAIT, SQL*Net message from client , 60177, 3.97ms, .4%, [ ], 393, 25.94, 153.12us average
70, SOE , WAIT, PL/SQL lock timer , 12918348, 852.66ms, 85.3%, [WWWWWWWWW ], 1328, 87.65, 9.73ms average
70, SOE , WAIT, events in waitclass Other , 960, 63.36us, .0%, [ ], 2, .13, 480us average
-- End of Stats snap 1, end=2014-11-20 08:04:39, seconds=15.2
© 2016 Pythian

37. TOP SQL
38
SQL> @snapper ash=sql_id+wait_class 15 1 user=soe
Sampling SID user=soe with interval 15 seconds, taking 1 snapshots...
-- Session Snapper v4.11 BETA - by Tanel Poder ( http://blog.tanelpoder.com )
-- Enjoy the Most Advanced Oracle Troubleshooting Script on the Planet! :)
-------------------------------------------
Active% | SQL_ID | WAIT_CLASS
-------------------------------------------
14% | | Commit
5% | 7hk2m2702ua0g | User I/O
4% | 5ckxyqfvu60pj | User I/O
3% | 7t0959msvyt5g | User I/O
3% | gh2g2tynpcpv1 | User I/O
2% | 0w2qpuc6u2zsp | Scheduler
2% | g81cbrq5yamf5 | User I/O
2% | 7ws837zynp1zv | User I/O
2% | f7rxuxzt64k87 | User I/O
2% | 1qf3b7a46jm3u | User I/O
-- End of ASH snap 1, end=2014-11-20 06:14:42, seconds=15, samples_taken=100
© 2016 Pythian

38. SQLDEVELOPER
39 © 2016 Pythian

39. SQLDEVELOPER
40 © 2016 Pythian

40. HISTORIC PERFORMANCE
• moats, snapper etc only sample when running
• EE (+diag/tuning) has ASH and AWR
41 © 2016 Pythian

41. AWR REPORTS -> STATSPACK
• good for a high-level overview
• AWR has a bit more “stuff” than statspack
• AWR is setup automatically (60min snaps)
– be mindful of averages
• but AWR needs EE and diag&tuning pack
42 © 2016 Pythian

42. STATSPACK IN 12C
• no problem in non-CDB
• with multitenant would make most sense in CDB
– but not really supported due to common user c#...
• there is a nasty and unsupported workaround
43
https://jonathanlewis.wordpress.com/2013/07/04/12c-statspack-hack/
© 2016 Pythian

43. ORASASH
• does pretty much what ASH does (but for free)
• sample and store information from v$-views
• mostly compatible with EE ASH
– ashmon
– ashmasters etc
• stores data from multiple sources in repo db
– minimal footprint in source, only view and dblink
44
http://pioro.github.io/orasash/
© 2016 Pythian

44. SQLT
• oracle supported scripts
• needs a schema an installation in db
– (this is annoying)
• will gather *everything* about a single SQL
– option to use diag&tuning
• output is one zipped html archive
45
Metalink Doc id 215187.1
© 2016 Pythian

45. SQLD360
• sql scope (just like SQLT)
• needs no installation
• one zipfile including html output
46
http://mauro-pagano.com/2015/02/16/sqld360-sql-diagnostics-collection-made-faster/
© 2016 Pythian

46. SQLD SCREENSHOTS
47 © 2016 Pythian

47. SQLD SCREENSHOTS
48 © 2016 Pythian

48. SQLD INCLUDES SQL
49
SELECT MIN(a.snap_id) snap_id,
TO_CHAR(a.begin_interval_time, 'YYYY-MM-DD HH24:MI') begin_time,
TO_CHAR(a.end_interval_time, 'YYYY-MM-DD HH24:MI') end_time,
NVL(TRUNC(SUM(b.elapsed_time_total/NVL(NULLIF(executions_total,0),1)/1e6),3),0) elapsed_time,
NVL(TRUNC(SUM(b.cpu_time_total/NVL(NULLIF(executions_total,0),1)/1e6),3),0) cpu_time,
NVL(TRUNC(SUM(b.iowait_total/NVL(NULLIF(executions_total,0),1)/1e6),3),0) iowait,
NVL(TRUNC(SUM(b.clwait_total/NVL(NULLIF(executions_total,0),1)/1e6),3),0) clwait,
NVL(TRUNC(SUM(b.apwait_total/NVL(NULLIF(executions_total,0),1)/1e6),3),0) apwait,
NVL(TRUNC(SUM(b.ccwait_total/NVL(NULLIF(executions_total,0),1)/1e6),3),0) ccwait
FROM (SELECT snap_id, instance_number, elapsed_time_total, cpu_time_total,
iowait_total, clwait_total, apwait_total, ccwait_total, executions_total
FROM dba_hist_sqlstat
WHERE sql_id = 'gm9dkz4u1hwfv') b,
(SELECT snap_id, instance_number, begin_interval_time, end_interval_time
FROM dba_hist_snapshot
WHERE snap_id BETWEEN 0 AND 166142) a
WHERE a.snap_id = b.snap_id(+)
AND a.instance_number = b.instance_number(+)
AND 'Y' = 'Y'
AND a.instance_number = a.instance_number
GROUP BY
TO_CHAR(a.begin_interval_time, 'YYYY-MM-DD HH24:MI'),
TO_CHAR(a.end_interval_time, 'YYYY-MM-DD HH24:MI')
ORDER BY
TO_CHAR(a.end_interval_time, 'YYYY-MM-DD HH24:MI');
1850 rows selected.
© 2016 Pythian

49. EDB360
• database scope
• no installation needed!
– just run a sqlfile
• creates one zipfile with html for offline viewing
– also AWR etc
• T | D | N control for access packs
• convenient: graphs
50
http://www.enkitec.com/products/edb360
© 2016 Pythian

50. EDB SCREENSHOTS
51 © 2016 Pythian

51. TUNAS360
• best of sqld and snapper
• samples v$-views for period
– works without AWR/ASH license!
• produces output like sqld
52 © 2016 Pythian

52. TUNAS SCREENSHOTS
53 © 2016 Pythian

53. TUNAS SCREENSHOTS
54 © 2016 Pythian

54. GOOD OLD 10046 TRACE
• guaranteed to catch everything (unlike sampling)
– why not turn on instance-wide?
• overhead vs usefulness?
• no license needed at all
55 © 2016 Pythian

55. READING TRACEFILES
• sqldeveloper
• trcanlzr
– needs installing but now also part of SQLT
• Kyle Hailey’s script
– https://github.com/khailey/oracle_trace_parsing/
• Chris Antognini’s script
– https://antognini.ch/downloads/top2/chapter03/tvdxtat
_40beta10_20140630.zip
56 © 2016 Pythian

56. REVIEW
• make sqlplus great
again
– rlsqlplus
– sqlcl
• real-time analysis
– snapper
– moats
– web-ash
– sqldeveloper instance
viewer
– TUNAs360
57
• historical analysis
– Orasash / S-ASH
– statspack
– SQLT
– eDB360
– sqld360
• 10046 tracing
– raw
– various analyzers
© 2016 Pythian

58. 59
@brost
http://www.pythian.com/blog
rost@pythian.com
© 2016 Pythian