Tutorial given solo. minor alterations made.

1. Optimizing MySQL
Essentials
http://joind.in/3420

2. About Me
• Ligaya Turmelle
• Senior Technical Support Engineer
• MySQL Now (AKA Oracle)
• ~3 years

3. Agenda
• Basics
• Hardware and OS
• Server
• variables
• Queries
• slow query log and EXPLAIN
• example query

4. Basics
1+1 = 3

5. • Definition of Performance:
Characterized by the amount of useful
work accomplished by a computer
system compared to the time and
resources used.

6. This can cover a lot of different areas
(potentially at the same time) in a LAMP
stack
• Application
• Network
• Hardware
• Database

7. Learn how to benchmark
• There are lots of benchmarking tools
out there.
• Use what is appropriate to measure
what you are trying to improve

8. OS and Hardware

9. OS
• Linux vs. Windows
• Linux vs. Linux

10. Linux
• swappiness = 0
• IO scheduler
• HDD = deadline
• SSD or you have a good RAID
controller = noop

11. Hardware
• RAM
• Disks
• CPU

12. Basic Linux commands
• top
• iostat
• vmstat

13. Server Settings
So what should i tweak?

14. Knobs, Levers and
Buttons
• mysqld
• MyISAM
• InnoDB
• Others

15. MySQLD
what knobs to turn?

16. MySQLD
Many think that yum install / apt-get install
and they’re done.
• No, we’re just beginning!
• Logging
• Memory Usage:
• Global
• Session
• etc

17. MySQLD
( logging )

18. MySQLD
( logging )
• log_bin = <filename>

19. MySQLD
( logging )
• log_bin = <filename>
• general_log = 0
general_log_file = <filename>

20. MySQLD
( logging )
• log_bin = <filename>
• general_log = 0
general_log_file = <filename>
• slow_query_log = 1
slow_log_file = <filename>
long_query_time = 1
• log_queries_not_using_indexes
min_examined_row_limit = 100

21. MySQLD
( memory usage )
• Memory is allocated...
• Globally - server wide, usually allocated
only at server startup and exist only “once”
• Session - each connection or event

22. MySQLD
( Timers )

23. MySQLD
( Timers )
• connect_timeout ( 10 )

24. MySQLD
( Timers )
• connect_timeout ( 10 )
• wait_timeout = 3

25. MySQLD
( Timers )
• connect_timeout ( 10 )
• wait_timeout = 3
• expire_log_days ( 7 )

26. MySQLD
( Memory / global )

27. MySQLD
( Memory / global )
• query_cache_type = 0 ( yes, 0! )
• query_cache_size

28. MySQLD
( Memory / global )
• query_cache_type = 0 ( yes, 0! )
• query_cache_size
• max_heap_table_size /
tmp_table_size

29. MySQLD
( Memory / global )

30. MySQLD
( Memory / global )
• table_open_cache

31. MySQLD
( Memory / global )
• table_open_cache
• thead_cache_size

32. MySQLD
( Memory / global )
• table_open_cache
• thead_cache_size
• skip_name_resolve = 1

33. MySQLD
( Memory / global )
• table_open_cache
• thead_cache_size
• skip_name_resolve = 1
• performance_schema = 1

34. MySQLD
( memory / global )
• read_buffer_size /
read_rnd_buffer_size
• sort_buffer_size
• join_buffer_size
} Also
configurable
per session

35. MyISAM
what levers to pull

36. MyISAM

37. MyISAM
• key_buffer_size
• Key_reads / Key_read_requests
• Key_writes / Key_write_requests

38. MyISAM
• concurrent_insert
• 0 - Never, 1 - Hole free, 2 - Holes

39. InnoDB
what buttons to press

40. InnoDB
Many options only a few covered
• innodb_buffer_pool_size
- caches both indexes and data
- upwards of 85% available RAM
• innodb_flush_log_at_trx_commit
- 0 - written and flushed to disk once per second
- 1 - written and flushed to disk at each
transaction commit
- 2 - written to disk at each commit, flush one a
second

41. InnoDB
• innodb_log_buffer_size
- Amount of data written to ib_logfile
- The larger, the less disk I/O
• innodb_log_file_size
- The larger the file, the more InnoDB can do
- < 5.5 the larger, the longer crash recovery time
• innodb_double_write_buffer
- data integrity checking
• innodb_file_per_table
- data and indexes written to their own .ibd file

42. InnoDB
• Efficiency:
innodb_buffer_pool_reads /
innodb_buffer_pool_read_requests

43. Examine, Test
and Measure
“Dr. Watson I Presume”

44. MySQLD
( quick check )
Quick Check - mysql-tuning-primer
https://launchpad.net/mysql-tuning-primer

45. MySQLD
( in depth 1/11 )
Getting In depth - SHOW GLOBAL STATUS
+--------------------------+------------+
| Variable_name | Value |
+--------------------------+------------+ | Com_replace | 37372 |
| Com_alter_table | 184 | | Com_replace_select | 0 |
| Com_begin | 103281460 | | Com_rollback | 148312594 |
| Com_change_db | 631 | | Com_select | 7164448573 |
| Com_commit | 669825867 | | Com_set_option | 1877614377 |
| Com_create_index | 0 | | Com_show_collations | 130652029 |
| Com_create_table | 354 | | Com_show_fields | 694 |
| Com_delete | 1235248406 | | Com_show_grants | 36 |
| Com_delete_multi | 353434291 | | Com_show_processlist | 33184 |
| Com_drop_table | 359 | | Com_show_slave_hosts | 1 |
| Com_flush | 3 | | Com_show_variables | 130885618 |
| Com_insert | 2725147239 | | Com_show_warnings | 99619 |
| Com_insert_select | 9 | | Com_truncate | 11 |
| Com_kill | 3642 | | Com_unlock_tables | 18 |
| Com_load | 15 | | Com_update | 1348570903 |
| Com_lock_tables | 18 | | Com_update_multi | 8 |
+--------------------------+------------+

46. MySQLD
( in depth 2/11 )
Values with out timeframe are
meaningless.
10:21:08 rderoo@mysql09:mysql [1169]> SHOW GLOBAL STATUS LIKE 'Uptime';
+---------------+----------+
| Variable_name | Value |
+---------------+----------+
| Uptime | 12973903 |
+---------------+----------+
1 row in set (0.00 sec)
That’s about 150 days. :)

47. MySQLD
( in depth 3/11 )
By using mysqladmin we can observe
changes over short periods of time:
$ mysqladmin -u rderoo -p -c 2 -i 10 -r extended-status > review.txt
Output is the same as SHOW GLOBAL STATUS
with two copies appearing in review.txt
the first has values since the last server
restart / FLUSH COUNTERS the second output
are the delta between the first and 10 seconds
later.

48. MySQLD
( in depth 4/11 )
Temporary Tables:
How rderoo@mysql09:event [1185]> SHOW GLOBAL VARIABLES LIKE
17:54:33
big?
'%_table_size';
+---------------------+----------+
| Variable_name | Value |
+---------------------+----------+
| max_heap_table_size | 67108864 |
| tmp_table_size | 67108864 |
+---------------------+----------+
2 rows in set (0.00 sec)
How many?
17:55:09 rderoo@mysql09:event [1186]> SHOW GLOBAL STATUS LIKE '%_tmp
%tables';
+-------------------------+-----------+
| Variable_name | Value |
+-------------------------+-----------+
| Created_tmp_disk_tables | 156 |
| Created_tmp_tables | 278190736 |
+-------------------------+-----------+
2 rows in set (0.00 sec)

49. MySQLD
( in depth 6/11 )
open_table_cache
| Open_tables | 4094 |
| Opened_tables | 12639 |
| Uptime | 12980297 |
• This yields a rate of ~85 table opens per day, a bit high...
20:52:22 rderoo@mysql09:mysql [1190]> SHOW GLOBAL VARIABLES LIKE
'table_cache';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| table_cache | 4096 |
+---------------+-------+
• Increasing the table_cache would be advisable.

50. MySQLD
( in depth 7/11 )
thread_cache_size
• To determine the effectiveness of the thread_cache
as a hit ratio use the following formula:
100-((Threads_created/Connections)*100)
| Connections | 131877233 |
| Threads_created | 20014624 |
100 - ( ( 20014624 / 131877233 ) * 100 ) = 84.8%

51. MySQLD
( in depth 8/11 )
read_buffer_size
• The effectiveness can be found by examining the
Select_scan :
| Select_scan | 304864263 |
• This should be set to a small value globally and
increased on a per session basis
• Increase when performing full table scans

52. MySQLD
( in depth 9/11)
read_rnd_buffer_size
• Used when reading sorted results after a key sort has
taken place
• This should be set to a small value globally and
increased on a per session basis
• Usually should be set ( per session ) no high than 8M

53. MySQLD
( in depth 10/11)
sort_buffer_size
• The effectiveness can be found by examining the
Sort_merge_passes :
| Sort_merge_passes | 7 |
• This should be set to a small value globally and
increased on a per session basis
• Increase GROUP BY, ORDER BY, SELECT
DISTINCT and UNION DISTINCT performance

54. MySQLD
( in depth 11/11 )
join_buffer_size
• The effectiveness can be found by examining the
Select_full_join :
| Select_full_join | 5369 |
• A crutch used when proper indexing is not in place
• Will help in cases of full table scans
• This should be set to a small value globally and
increased on a per session basis
• Increase GROUP BY and ORDER BY performance

55. Tuning Queries
Where the magic happens

56. Tuning your queries is the single biggest
performance improvement you can do to
your server
• efficient resource usage
• results returned quicker
Per QUERY!

57. Where do I start?

58. slow query log

59. Slow query log

60. Slow query log
• A log of all the “slow” queries

61. Slow query log
• A log of all the “slow” queries
• Define “slow”
• took longer then long_query_time
to run AND at least
min_examined_row_limit rows
are looked at
• log_queries_not_using_indexes
• log_slow_admin_statements

62. Turning it on
• Option is slow_query_log
• no argument or value of 1 - enabled
• argument of 0 - disabled (default)

63. Output Types
• Use log_output
• write to a file
• write to a table in the mysql database

64. slow log at startup
• Place the slow_query_log and
optional log_output options in
my.cnf
• log_output acceptable values
• TABLE, FILE, NONE
• Can also use slow_query_log_file
to specify a log file name

65. slow log at runtime
• All the below global variables can be
changed at run time
• log_output
• slow_query_log
• slow_query_log_file
SET GLOBAL slow_query_log = 1;

66. /usr/sbin/mysqld, Version: 5.0.67-0ubuntu6.1-log ((Ubuntu)). started with:
Tcp port: 3306 Unix socket: /var/run/mysqld/mysqld.sock
Time Id Command Argument
# Time: 110328 13:15:05
# User@Host: homer[homer] @ localhost []
# Query_time: 3 Lock_time: 0 Rows_sent: 0 Rows_examined: 4452238
use confs;
select
distinct u.ID as user_id,
t.event_id,
u.username,
u.full_name
from
user u,
talks t,
talk_speaker ts
where
u.ID <> 6198 and
u.ID = ts.speaker_id and
t.ID = ts.talk_id and
t.event_id in (
select
distinct t.event_id
from
talk_speaker ts,
talks t
where
ts.speaker_id = 6198 and
t.ID = ts.talk_id
)
order by rand()
limit 15;

67. slow query log
EXPLAINed

68. There are multiple ways of getting
information out of the slow query log.
• just open it up to see the raw data
• usually more helpful to summarize/
aggregate the raw data

69. Tools for aggregating
the slow log data
• mk-query-digest
• third party
• part of the maatkit utilities
• mysqldumpslow
• comes with MySQL in the bin dir

70. Macintosh-7:joindin-for-lig ligaya$ perl -laF ~/mysql_installs/maatkit/mk-
query-digest mysql-slow.log
# 23s user time, 240ms system time, 17.53M rss, 86.51M vsz
# Current date: Mon May 16 11:08:24 2011
# Hostname: Macintosh-7.local
# Files: mysql-slow.log
# Overall: 73.72k total, 14 unique, 0.02 QPS, 0.04x concurrency __________
# Time range: 2011-03-28 13:15:05 to 2011-05-13 02:33:52
# Attribute total min max avg 95% stddev median
# ============ ======= ======= ======= ======= ======= ======= =======
# Exec time 160130s 0 180s 2s 3s 819ms 2s
# Lock time 11s 0 1s 149us 0 12ms 0
# Rows sent 387.64k 0 255 5.38 14.52 6.86 0
# Rows examine 338.66G 0 5.35M 4.70M 4.93M 387.82k 4.70M
# Query size 30.19M 17 8.35k 429.47 420.77 29.33 420.77
# Profile
# Rank Query ID Response time Calls R/Call Apdx V/M Item
# ==== ================== ================= ===== ======= ==== ===== =====
# 1 0x2AAA5EC420D7E2A2 159878.0000 99.8% 73682 2.1698 0.50 0.12 SELECT
user talks talk_speaker talks
# 3 0x0A2584C03C8614A9 50.0000 0.0% 16 3.1250 0.44 0.87 SELECT
wp_comments
# MISC 0xMISC 202.0000 0.1% 20 10.1000 NS 0.0 <12
ITEMS>

71. Count: 1 Time=180.00s (180s) Lock=0.00s (0s) Rows=1.0 (1), XXXXX[xxxxx]@localhost
SELECT SLEEP(N)
Count: 16 Time=3.12s (50s) Lock=0.00s (0s) Rows=0.9 (14), XXXXX[xxxxx]@localhost
SELECT comment_date_gmt FROM wp_comments WHERE comment_author_IP = 'S' OR
comment_author_email = 'S' ORDER BY comment_date DESC LIMIT N
Count: 5 Time=2.80s (14s) Lock=0.00s (0s) Rows=0.0 (0), XXXXX[xxxxx]@localhost
SELECT comment_ID FROM wp_comments WHERE comment_post_ID = 'S' AND ( comment_author
= 'S' OR comment_author_email = 'S' ) AND comment_content = 'S' LIMIT N
Count: 73682 Time=2.17s (159867s) Lock=0.00s (11s) Rows=5.4 (396223),
XXXXX[xxxxx]@localhost
select
distinct u.ID as user_id,
t.event_id,
u.username,
u.full_name
from
user u,
talks t,
talk_speaker ts
where
u.ID <> N and
u.ID = ts.speaker_id and
t.ID = ts.talk_id and
t.event_id in (
select
distinct t.event_id
from
talk_speaker ts,
talks t
where
ts.speaker_id = N and
t.ID = ts.talk_id
)
order by rand()
limit N

72. Tuning a query
• Tables
• SHOW CREATE TABLE `talks`;
• SHOW TABLE STATUS LIKE
‘talks’;
• Indexes
• SHOW INDEXES FROM ‘talks’;
• EXPLAIN

73. EXPLAIN

74. EXPLAIN Basics
• Syntax: EXPLAIN [EXTENDED]
SELECT select_options
• Displays information from the
optimizer about the query execution
plan
• Works only with SELECT statements

75. EXPLAIN Output
• Each row provides information on one
table
• Output columns:
id key_length
select_type ref
table rows
type Filtered (new to 5.1)
possible_keys Extra
key

76. EXPLAIN Output

77. id
• Select identifier
• Only if there are subqueries, derived
tables or unions is this incremented
• Number reflects the order that the
SELECT/FROM was done in

78. select_type
• Type of SELECT
• SIMPLE
• PRIMARY
• DERIVED
• UNION
• SUBQUERY

79. table
• The table (or alias) that is used
• Read down - these are the order the
tables are JOINed
• DERIVED tables are noted and
numbered

80. type
• Access type
• Various types:
preference access type
BEST system/const
eq_ref
ref
ref_or_null
index_merge
range
index
WORST ALL

81. possible_keys
• List of all possible indexes (or NULL)
that optimizer considered using

82. key
• Index used for the query or NULL
• Look at key and possible_keys to
consider your index strategy

83. key_len
• shows number of bytes MySQL will use
from the index
• Possible that only part of the index will
be used
NOTE: a character set may use more then
one byte per character

84. ref
• Very different then the access type ‘ref’
• Show which columns or a constant
within the index will be used for the
access

85. rows
• Number of rows MySQL expects to find
based on statistics
• statistics can be updated with
ANALYZE TABLE

86. filtered
• New in 5.1.12
• Estimated number of rows filtered by
the table condition
• Shows up if you use EXPLAIN
EXTENDED

87. Extra
• All the other stuff
• Using index - using a covering index
• Using filesort - sorting in a temporary
table
• Using temporary - a temporary table
was made.
• Using where - filtering outside the
storage engine.

88. Query Tuning
Example

89. Query we are working
with:
EXPLAIN select distinct u.ID as user_id,
t.event_id,
u.username,
u.full_name
from user u,
talks t,
talk_speaker ts
where u.ID <> 11945 and
u.ID = ts.speaker_id and
t.ID = ts.talk_id and
t.event_id in (
select distinct t.event_id
from talk_speaker ts,
talks t
where ts.speaker_id = 11945 and
t.ID = ts.talk_id
)
order by rand()
limit 15G

90. Full EXPLAIN plan
*************************** 1. row ***************************
id: 1
select_type: PRIMARY
table: t
type: index
possible_keys: PRIMARY
key: idx_event
key_len: 9
ref: NULL
rows: 3119
Extra: Using where; Using index; Using temporary; Using filesort
*************************** 2. row ***************************
id: 1
select_type: PRIMARY
table: ts
type: ref
possible_keys: talk_id
key: talk_id
key_len: 5
ref: joindin.t.ID
rows: 1
Extra: Using where
*************************** 3. row ***************************
id: 1
select_type: PRIMARY
table: u
type: eq_ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: joindin.ts.speaker_id
rows: 1
Extra:

91. Full EXPLAIN plan
(con’t)
*************************** 4. row ***************************
id: 2
select_type: DEPENDENT SUBQUERY
table: t
type: ref
possible_keys: PRIMARY,idx_event
key: idx_event
key_len: 5
ref: func
rows: 24
Extra: Using where; Using index; Using temporary
*************************** 5. row ***************************
id: 2
select_type: DEPENDENT SUBQUERY
table: ts
type: ref
possible_keys: talk_id
key: talk_id
key_len: 5
ref: joindin.t.ID
rows: 1
Extra: Using where
5 rows in set (0.00 sec)

92. What was wrong
with that query
plan?

93. Hint
EXPLAIN select distinct u.ID as user_id, *************************** 4. row ***********************
t.event_id, id: 2
u.username, select_type: DEPENDENT SUBQUERY
u.full_name table: t
from user u, type: ref
talks t, possible_keys: PRIMARY,idx_event
talk_speaker ts key: idx_event
where u.ID <> 11945 and key_len: 5
u.ID = ts.speaker_id and ref: func
t.ID = ts.talk_id and rows: 24
t.event_id in ( Extra: Using where; Using index; Using temporary
select distinct t.event_id *************************** 5. row **********************
from talk_speaker ts, id: 2
talks t select_type: DEPENDENT SUBQUERY
where ts.speaker_id = 11945 table: ts
and type: ref
t.ID = ts.talk_id possible_keys: talk_id
) key: talk_id
order by rand() key_len: 5
limit 15G ref: joindin.t.ID
rows: 1
Extra: Using where
5 rows in set (0.00 sec)

94. Solution
1:
SELECT DISTINCT u.ID as user_id,
t.event_id,
u.username,
u.full_name
FROM user u
JOIN talk_speaker ts
ON u.ID = ts.speaker_id
JOIN talks t
ON t.ID = ts.talk_id
JOIN
-- find the events the speaker has talks in
(
SELECT t1.event_id
FROM talk_speaker ts1
JOIN talks t1
ON t1.ID = ts1.talk_id
WHERE ts1.speaker_id = 11945
) as e
ON t.event_id = e.event_id
WHERE u.ID <> 11945
LIMIT 15G

95. Solution
1:
SELECT DISTINCT u.ID as user_id,
t.event_id,
u.username,
u.full_name
FROM user u
JOIN talk_speaker ts
ON u.ID = ts.speaker_id
JOIN talks t
ON t.ID = ts.talk_id
JOIN
-- find the events the speaker has talks in
(
SELECT t1.event_id
FROM talk_speaker ts1
JOIN talks t1
ON t1.ID = ts1.talk_id
WHERE ts1.speaker_id = 11945
) as e
ON t.event_id = e.event_id
WHERE u.ID <> 11945
LIMIT 15G

96. Solution
1:
SELECT DISTINCT u.ID as user_id,
t.event_id,
u.username,
u.full_name
FROM user u
JOIN talk_speaker ts
ON u.ID = ts.speaker_id
JOIN talks t
ON t.ID = ts.talk_id
JOIN
-- find the events the speaker has talks in
(
SELECT t1.event_id
FROM talk_speaker ts1
JOIN talks t1
ON t1.ID = ts1.talk_id
WHERE ts1.speaker_id = 11945
) as e
ON t.event_id = e.event_id
WHERE u.ID <> 11945
LIMIT 15G

97. Solution
1: reminder:
CREATE TABLE `talk_speaker` (
SELECT DISTINCT u.ID as user_id, `talk_id` int(11) DEFAULT NULL,
t.event_id, `speaker_name` varchar(200) DEFAULT NULL,
u.username, `ID` int(11) NOT NULL AUTO_INCREMENT,
u.full_name `speaker_id` int(11) DEFAULT NULL,
FROM user u `status` varchar(10) DEFAULT NULL,
JOIN talk_speaker ts PRIMARY KEY (`ID`),
ON u.ID = ts.speaker_id KEY `talk_id` (`talk_id`) USING BTREE
JOIN talks t ) ENGINE=MyISAM AUTO_INCREMENT=3763
ON t.ID = ts.talk_id DEFAULT CHARSET=utf8;
JOIN
-- find the events the speaker has talks in
(
SELECT t1.event_id
FROM talk_speaker ts1
JOIN talks t1
ON t1.ID = ts1.talk_id
WHERE ts1.speaker_id = 11945
) as e
ON t.event_id = e.event_id
WHERE u.ID <> 11945
LIMIT 15G

98. Solution
1: reminder:
CREATE TABLE `talk_speaker` (
SELECT DISTINCT u.ID as user_id, `talk_id` int(11) DEFAULT NULL,
t.event_id, `speaker_name` varchar(200) DEFAULT NULL,
u.username, `ID` int(11) NOT NULL AUTO_INCREMENT,
u.full_name `speaker_id` int(11) DEFAULT NULL,
FROM user u `status` varchar(10) DEFAULT NULL,
JOIN talk_speaker ts PRIMARY KEY (`ID`),
ON u.ID = ts.speaker_id KEY `talk_id` (`talk_id`) USING BTREE
JOIN talks t ) ENGINE=MyISAM AUTO_INCREMENT=3763
ON t.ID = ts.talk_id DEFAULT CHARSET=utf8;
JOIN
-- find the events the speaker has talks in
(
SELECT t1.event_id 2:
FROM talk_speaker ts1 ALTER TABLE `talk_speaker`
JOIN talks t1 ADD INDEX (speaker_id);
ON t1.ID = ts1.talk_id
WHERE ts1.speaker_id = 11945 OR
) as e ALTER TABLE `talk_speaker`
ON t.event_id = e.event_id ADD INDEX (speaker_id, talk_id);
WHERE u.ID <> 11945
LIMIT 15G

99. Q&A

100. Thank you
http://joind.in/3420