Presentation given at IOUG Collaborate 2015 (and in other versions in the past)

1. REMINDER
Check in on the
COLLABORATE mobile app
Top 10 Oracle database tuning tips
Guy Harrison,
Executive Director, Information Mgt R&D,
Dell Software Group
Session ID#: 442
@guyharrison

2. Top 10 Oracle
database tuning
tips
Guy Harrison,
Executive Director, Information Mgt R&D,
Dell Software Group

3. Dell Software Group3
Web: guyharrison.net
Email: guy.harrison@software.dell.com
Twitter: @guyharrison
Introductions

4. 4
Dell Software Group

5. 5
Dell Software Group

6. 6
Dell Software Group

7. 7
Dell Software Group
Dell and Quest – a brief history

8. Dell Software Group8
But Seriously

9. Dell Software Group9
The top 10
Be methodical and empirical
Optimize your database design
Index Wisely
Write efficient code
Optimize the optimizer
Tune SQL and PL/SQL
Monitor and Manage contention
Optimize memory to reduce IO
Tune IO last, but TUNE it well

10. Dell - Restricted - Confidential
Be methodical and
empirical

11. Dell Software Group11
Hint 1: Methodical and Empirical tuning
Methodical:
• Have a plan
• Focus on root causes, not symptoms
• Hypothesis and experiment, not trial and
error
Empirical
• Measure performance
• Compare before and after
• What is your control group?
• Beware of “old DBA tales” and “silver
bullets”

12. Dell Software Group12
Hint 1: Methodical and Empirical tuning
Performance
Troubleshooting
• Identify the element of your workload which is
contributing most to service time
• Workload could be SQL or logical transaction
• Element is typically CPU time + Oracle wait
time
• Find a way to reduce that element
Tuning by layers
• As above, but prioritize higher layers of the
stack (which have flow on effects into lower
layers)
• Distinguish between symptoms and causes

13. Dell Software Group13
Basics of a database request

14. Dell Software Group14
The 4 layers in database performance

15. Dell Software Group15
What’s wrong with the
donkey?

16. Dell Software Group16
Measurement
• Standard SQL views
• AWR reports
• DB control/Cloud control
• Toad/Spotlight other 3rd party
tools
• A single query can tell you a lot
http://guyharrison.net/OPSGSa
mples/Ch03/timeModelSimple.
sql

17. Dell - Restricted - Confidential
Database design

18. Dell Software Group18
Database design
Third Normal
form (3NF)
• The key, the whole key and nothing but the key
• OR – Don’t repeat yourself (DRY)
3NF discretion
• Datatypes (VARCHARs vs LOBS, etc)
• Subtypes
• NULLS
Denormalization
• Replicate column values to avoid joins
• Summary tables and materialized views
• Vertical partitioning

19. Dell Software Group19
3NF is mostly Don’t Repeat Yourself (DRY)
• We don’t want to repeat the student name
every time they take a new test
• We don’t want to repeat the test_name for
every student that takes it
• We don’t want the “repeating group” of
answers
– Also, we might not know how many questions
there will be in future tests

20. Dell Software Group20
3NF version
• Students take tests
• Tests have questions
• Students taking tests provide answers

21. Dell Software Group21
Don’t go to far!
• In this example the designer pulled address
into a separate table and city, country into
another table.
• It’s correct in theory, but it means that
every time we want an employees address
we have to join three tables.
• It would have been better to have just one
table

22. Dell Software Group22
Logical to Physical: Subtypes
“Customers are people too”

23. Dell Software Group23
Subtypes
• It’s usually better to put all the data in one table, or have two tables for each subtype
• Having a “supertype” table and two “subtype” tables means you always have to join tables
Probably don’t
want to do this

24. Dell Software Group24
Denormalization

25. Dell - Restricted - Confidential
Index Wisely

26. Dell Software Group26
Indexes
• Choosing the best set of indexes is crucial
• Create the best set of Concatenated (multi-column) indexes that will
support your queries.
• Indexes slow down transactions, so don’t create too many
Indexes exist mainly to improve performance
• You use the index when you want to look up one thing or a few
things
• You don’t use the index to read the whole book or read a whole
chapter
• You do use an index when you want to go to a specific page
Think of indexes on a table like indexes in a book

27. Dell Software Group27
Concatenated index effectiveness
• The more columns in the
index the less IO it takes to
resolve a query using all
the columns
• The order of the columns
affects how the index can
be used
• If the index has all the
columns, don’t need to
touch the table at all
SELECT cust_id
FROM sh.customers c
WHERE cust_first_name = 'Connor'
AND cust_last_name = 'Bishop'
AND cust_year_of_birth = 1976;
0 500 1000 1500
None
last name
last+first name
last,first,BirthYear
last,first,birthyear,id
1459
63
6
4
3
Logical IO

28. Dell Software Group28
Index overhead
• Indexes speed up
queries, but make DML
(insert, update, Delete)
slower
• This chart shows how
inserts slow down as
more and more indexes
are added
• Deleting a row with lot’s
of indexes is particularly
expensive
0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000
1 (PK only)
2
3
4
5
6
7
1,191
6,671
8,691
10,719
12,727
14,285
16,316
Logical reads required
Numberofindexes

29. Dell Software Group29
1
10
100
1000
0 10 20 30 40 50 60 70 80 90 100
ElaspedTime(s)
Pct of table accessed
Full Scan no caching
Index sorted data, no caching
Index unsorted, cached data
Full Table scan, cached data
Index or FTS?
Break even points

30. Dell - Restricted - Confidential
Application code

31. Dell Software Group31
Database coding guidelines
SQL
Execution
• Don’t call the database unless you have
to – cache data instead.
• Reduce “hard” parsing by using bind
variables
Transaction
design
• Minimize lock duration using optimistic
and Pessimistic locking strategies
Network
overhead
• Array fetch and Insert
• Stored procedures

32. Dell Software Group32
Parse overhead
• It’s easy enough in most programming languages to create a new SQL every time you execute
the query:

33. Dell Software Group33
Better to use the same SQL with different
arguments:
• Prepare the statement once, then execute many times with different
arguments
• Using bind variables

34. Dell Software Group34
Reduction in parse (SQL Compile) time
1,000 executions of the code on preceding
two slides in Oracle
0 200 400 600 800 1000 1200 1400
No Bind variables
Bind Variables
Parse Time Other

35. Dell Software Group35
Designing transactions
• There are two ways to design transaction locking
• Pessimistic works best if you think someone else is going to “grab” your row before you’re
finished
• Optimistic works best if you thing no one else will “grab” the row
Durationoflock
Durationoflock

36. Dell Software Group36
Reduce Network traffic
• “Round trips” to the database can add a lot of overhead
• Two ways to reduce round trips:
– Use the “Array” interface in your program code
– Use stored procedures for complex interactions with the database

37. Dell Software Group37
Array fetch performance
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
40,000
0 20 40 60 80 100 120 140
Array fetch size
Logical Reads Network round trips

38. Dell Software Group38
Network – stored procedures
• A stored procedure is code stored in the database
• If you have a transaction that goes “back and forth” to the database, consider a stored
procedure
• ESPECIALLY if you are working across a slow network

39. Dell - Restricted - Confidential
Optimize the
optimizer

40. Dell Software Group40
Object Statistics
Cardinality
Estimates
IO and CPU
Estimates
DB parameters
And config
Cost estimateSystem Statistics
Table and index
Structure
Optimizer inputs
• Remember: Garbage In : Garbage
Out
• The optimizer can only do a good
job if statistics are up to date

41. Dell Software Group41
Histograms
• Indexes are only good for getting
small amounts of the table
• So it might be a good idea to use an
index to get “New Zealand”
customers, but not “United States”
• A histogram allows the optimizer to
understand how the data is
distributed and make the best
decision
• Create histograms to correct bad
plans on skewed tables
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
20,000
SaudiArabia
SouthAfrica
Turkey
NewZealand
Denmark
Argentina
Singapore
Japan
Poland
China
Australia
Brazil
Canada
Spain
France
UnitedKingdom
Italy
Germany
UnitedStatesofAmerica
Numberofrows

42. Dell Software Group42
Without a histogram
Number of rows
the estimated
Real number of
rows
Optimizer chooses
not to use an index

43. Dell Software Group43
With a histogram
Optimizer estimate
is correct
Optimizer chooses
to use an index

44. Dell Software Group44
DBMS_STATS
GATHER_INDEX_STATS , GATHER_SCHEMA_STATS,
GATHER_TABLE_STATS
Basic statistics
method_opt => 'FOR ALL INDEXED COLUMNS SIZE AUTO'
Histograms
method_opt =>
'FOR ALL COLUMNS FOR COLUMNS (col1, col2)'
Multi-column extended statistics
method_opt => 'FOR ALL COLUMNS FOR COLUMNS
(function(column)))'
Expression extended statistics
DBMS_STATS.gather_system_stats (
gathering_mode => 'NOWORKLOAD');
Non-workload system statistics
DBMS_STATS.gather_system_stats
(gathering_mode => 'INTERVAL',
interval => 60);
Workload system statistics

45. Dell - Restricted - Confidential
Tune SQL and
PL/SQL

46. Dell Software Group46
Find them and tune them
Find SQL
• Mine V$SQL
• ASH, AWR tables
• Database control, grid/cloud control
• Toad, Spotlight
Tune SQL
• SQL Profiles
• SQL Baselines
• Rewrites
• Hints (be careful)
• SQL Optimizer

47. Dell Software Group47
V$SQL is your friend
• Also, V$SQL_PLAN,
V$SQL_PLAN_STATISTICS
• Tkprof, session trace

48. Dell Software Group48

49. Dell Software Group49
Now let’s look at the
donkey

50. Dell - Restricted - Confidential
Monitor and manage
contention

51. Dell Software Group51
Contention – the proverbial bottleneck
Application
Demand for DB
services
Contention for limited or
serialized resources causes
waits and or queuing
Apparent
demand at lower
layers is reduced

52. Dell Software Group52
Types of contention
Locks
Usually locking problems
are due to application
locks (remember
optimistic locking)?
Sometimes internal locks
can cause problems.
Latches and
Mutex
Latches are very light
weight locks that
protect memory instead
of tables
Buffer
contention
When sessions have to
wait for a memory
“buffer” to become
available

53. Dell Software Group53
Latches and mutex contention
• Latches are like locks, but
instead of protecting table
rows, they protect memory
(buffers)
• If two sessions try to access the
same area of memory, then one
will wait
• Instead of “sleeping” (like a lock)
they waiting session will “spin”
on the CPU for a very short
time
• Latch problems may indicate
“hot” blocks
• They might cause CPU drain
(because of spinning)
Buffers
Database
files
user user user

54. Dell Software Group54
Database
files
Buffers
Database
Writer
User
Buffer
Waits
Write
dirty
blocks
to disk
Write to
buffers
Read
from
disk
Read from
buffers
Free buffer waits
• Database buffers improve
performance by caching data in
memory
• When buffers are modified they are
called “dirty” – DBWR writes to disk
• When all the blocks are “dirty” then
sessions have to wait for the buffers to
be written before new data can be
read
• This might mean that your DBWR can’t
write to disk fast enough

55. Dell Software Group55
Specific contention scenarios
Lock/Latch Possible cause
Library cache mutex Hard parsing no bind variables. Try
CURSOR_SHARING=SIMILAR
Library cache pin PL/SQL package parsing and invalidations
Shared pool latch Hard parsing and possibly excessive SGA resizing
Cache buffers chains Hot blocks, very high logical read rates
Free Buffer waits DBWR failing to “keep up” with block changes
Flashback buf free Insufficient IO bandwidth for flashback area
Buffer busy Hot blocks – partitioning might help

56. Dell - Restricted - Confidential
Optimize memory to
reduce IO

57. Dell Software Group57
Memory is primarily used to avoid
IO
• Buffer cache avoids IO to table/index tablespaces
• PGA avoids IO to temporary tablespace
Buffer pools
Program Global
Area (PGA)
Sort area
Hash Area
Data (USER)
tablespace
Temporary
tablespace
Oracle Session

58. Dell Software Group58
Temp tablespace IO can easily overwhelm
table/index IO
Time
Available Memory
Table/Index IO CPU Time Temp Segment IO
Less MemoryMore Memory
Memory Sort
Single Pass
Disk Sort
Multi-pass
Disk Sort

59. Dell Software Group59
Automatic Memory Management
• Introduced in 11g, AMM manages allocations between and within
PGA and SGA

60. Dell Software Group60
AMM can (rarely?) lead to thrashing or starvation
• ALWAYS set minimum values for key components of the SGA

61. Dell - Restricted - Confidential
Tune IO last, but tune
it well

62. Dell Software Group62
IO Tuning
• Disk IO is the slowest part of the database system, so it’s critical
to performance
• FIRST:
– Tune SQL and application
– Remove contention
– Allocate memory
• Only when you’ve done that will your IO demand be realistic.
Then you can tune your IO

63. Dell Software Group63
Basics of IO tuning (magnetic disks)
• The amount of IO you can support depends on the number of disks you have
• Provide enough disks to support the amount of IO you need
• even if that means the disks are not filled with data
• Magnetics disks can do between 75-150 IO per second (IOPS) – do the math!
0 20 40 60 80 100 120 140 160 180 200
SATA 7,200 rpm
SATA 10,000 rpm
SAS 10,000 rpm
SAS 15,000 rpm
IO/ps

64. Dell Software Group64
Disk Drive latency
• Latency is the time taken to perform a single IO
• Most disk drives can return an IO in 5-10 ms
– Faster if the disk has a memory cache
• Disk latency increases with:
– Throughput (best at 50-75% of maximum)
– Disk fill (best at 50% capacity)
• To get best performance disks should be
– Sparsely populated
– Under only moderate load
• RAID 0+1 is the preferred configuration
0
10
20
30
40
50
60
70
80
90
100
0 100 200 300 400 500
ResponseTime(ms)
IO/second

65. Dell Software Group65
The more that things change....

66. Dell Software Group66
Disks are not getting faster

67. Dell Software Group67
SSD - Cheaper by the IO
4,000
80
25
15
0 1,000 2,000 3,000 4,000 5,000
Magnetic Disk
SSD SATA Flash
SSD PCI flash
SSD DDR-RAM
Seek time (us)

68. Dell Software Group68
But not by
the GB
0.35 0.28 0.21 0.17 0.13
2.9
2.2
1.7
1.3
1
10
7.4
5.3
3.2
2.3
0
2
4
6
8
10
12
2011 2012 2013 2014 2015
$$/GB
HDD MLC SDD SLC SSD
0.35
0.28
0.21
0.17
0.13
2.9
2.2
1.7
1.3
1
2.3
0.1
1
10
2011 2012 2013 2014 2015
$$/GB
HDD MLC SDD SLC SSD

69. Dell Software Group69
Tiered storage management
Main Memory
DDR SSD
Flash SSD
Fast Disk (SAS, RAID 0+1)
Slow Disk (SATA, RAID 5)
Tape, Flat Files, Hadoop
$/IOP
$/GB

70. Dell Software Group70
Random reads – FusionIO
2,211
583
121
0 500 1000 1500 2000 2500
SAS disk, no
flash cache
SAS disk, flash
cache
Table on SSD
Elapsed time (s)
CPU
Other
DB File IO
Flash cache IO

71. Dell Software Group71
Full table scans
Flash cache doesn’t
accelerate Full table scans b/c
scans use direct path reads
and flash cache only
accelerates buffered reads
418
398
72
0 100 200 300 400 500
SAS disk, no flash cache
SAS disk, flash cache
Table on SSD
Elasped time (s)
CPU
Other
DB File IO
Flash Cache IO

72. Dell Software Group72
Disk Sorts – temp tablespace SSD vs HDD
0
500
1000
1500
2000
2500
3000
3500
4000
050100150200250300
Elapsedtime(s)
Sort Area Size
SAS based TTS SSD based TTS
Single Pass
Disk Sort
Multi-pass
Disk Sort

73. Dell Software Group73
Concurrent redo workload (x10)
1,605
1,637
397
331
1,944
1,681
0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500
SAS based redo log
Flash based redo log
Elapsed time (s)
CPU
Other
Log File IO

74. Dell - Restricted - Confidential
Tune RAC

75. Dell Software Group75
RAC will scale well,
providing that….
a) The time taken to
transfer a block across
the interconnect is
much less than the
time taken to read
from disk
Load is reasonably well balanced
across the instances in the cluster
The overhead of maintaining cluster
consistency does not dominate
overall response time

76. Dell Software Group76
Toad: your
companion
in the Big
Data
revolution

77. Dell Software Group77
Dell In-Memory Appliances for Cloudera Enterprise
Mid-Size Configuration
16 Node Cluster
R720- 4 Infrastructure Nodes
R720XD- 12 Data Nodes
Force10- S4810P
Force10- S55
~528TB (disk raw space)
~4.5 TB (raw memory)
Starter Configuration
8 Node Cluster
R720- 4 Infrastructure Nodes
R720XD- 4 Data Nodes
Force10- S55
~176TB (disk raw space)
~1.5TB (raw memory)
Small Enterprise
Configuration
24 Node Cluster
R720- 4 Infrastructure Nodes
R720XD- 20 Data Nodes
~880TB (disk raw space)
~7.5 TB (raw memory)
Expansion Unit- R720XD-4 Data, Cloudera Enterprise Data Hub, Scale in Blocks

78. Dell Software Group78
Dell appliances for any database
• Dell provides appliances and reference
architectures specifically designed for:
– Oracle
– SQL Server
– HANA
– SSD database acceleration
– Large memory footprints

79. 79
Dell Software Group
Thank you.

80. Please complete the session
evaluation
We appreciate your feedback and insight
You may complete the session evaluation either
on paper or online via the mobile app