I have explained different ways to optimize the MySQL database.

1. MySQL
Performance
Tuning
Anurag Srivastava

2. Where to Analyze ?
● Development to catch bad queries early
● Production to catch good queries starting to go bad

3. Query
Performance
always shows it
● Workload Changes
● Data Size
● Changing Query Plans
● Upgrading MySQL Version
● Changing MySQL Settings
● Hardware Changes

4. Queries
The IN clause in MySQL is very fast!
● Select ... Where idx IN(1,23,345,456)
Keep column alone on left side of condition
● Select ... Where func(idx) = 20 [index ignored]
● Select .. Where idx = otherfunc(20) [may use index]
Avoid % at the start of LIKE on an index
● Select ... Where idx LIKE(‘ABC%’) can use index
● Select ... Where idx LIKE(‘%XYZ’) must do full table scan

5. Queries
Use EXPLAIN to see the execution plan for the query
● Check the index usage
● Check rows scanned
If column used in ORDER BY clause are indexed they help with
performance.
Try to convert <> operator to = operator
● = operator increases chances of index usage
Avoid using SELECT *
● Forces full table scan
● Wastes network bandwidth

6. Index
● Database structure that can be
associated to a table
● Indexes are usually used to search
table faster
● Without indexes it would be needed
to scan the whole table
● Table may have multiple indexes

7. When to use Indexes
● Do not start using indexes before you need them
● Only large tables need indexes
● Large table: Multiply the record count by the field's
length
● Monitor the slow query logs

8. What is the
slow query
log ?
● Show information of queries that take long time
to execute
● Enable with these lines to your server my.cnf
file
log_slow_queries = 1;
slow_query_log_file = <some file name>;
● Use mysqldumpslow command to get
summaries of slow queries

9. Index
● In MyISAM data pointers point to physical
offset in the data file
○ All indexes are essentially equivalent
● In Innodb
○ PRIMARY KEY (Explicit or Implicit) - stores data in leaf
pages of the index, not pointer
○ Secondary Indexes - store primary key as data pointer.

10. Index - Tricky with Multiple Columns
Index(A,B,C) - Order of columns matters
Will use Index for lookup (All listed keyparts)
● A>5
● A=2 AND B>6
● A=2 AND B=6 AND C=7
● A=5 AND B IN (2,3) AND C>5
Will not use Index
● B>5
● B=6 AND C=7
Will Use Part of Index
● A>5 AND B=4 - range on first column, only use this key part
● A=5 AND B>6 AND C=2 - range on second column, use 2 parts

11. Index-join
Indexes speed up joins
● SELECT X.A, Y.B FROM X,Y WHERE X.C = ‘FL’ and Y.A
= X.A ;
The Filter is on column C of table X
● Table X needs an index on column C for the filter
Table Y is joined to table X by column A
● Table Y needs an index on column A

12. Index-sort
select * from players order by score desc limit 10
● Will use index on score column
● Without index MySQL will do filesort (external sort) which
is very expensive)
Often combined with using index for lookup
Select * from players where country = “INDIA” order by score
desc limit 10
● Best served by Index on (country, score)

13. Don't want
MySQL
filesort?
By default, MySQL sorts all GROUP BY col1, col2,
… queries as if you specified ORDER BY col1,
col2, … in the query as well.
If a query includes GROUP BY but you want to
avoid the overhead of sorting the result, you can
suppress sorting by specifying ORDER BY NULL.
SELECT a, COUNT(*) FROM bar GROUP BY a ORDER BY NULL;

14. Index - multi column efficient sorting
KEY(A,B)
Will use Index for sorting
● ORDER BY A - Sorting by leading column
● A=5 ORDER BY B - EQ filtering by 1st and sorting by 2nd
● ORDER BY A DESC, B DESC - Sorting by 2 columns in same order
● A>5 ORDER BY A - Range and sorting on the 1st column
Will NOT use Index for sorting
● ORDER BY B - Sorting by second column in the index
● A>5 ORDER BY B - Range on first column, sorting by second
● A IN (1,2) ORDER BY B - In-Range on first column

15. Covering
Index
● Applies to index use for specific query, not type
of index.
Reading Index ONLY and not accessing the data
SELECT status from orders where
customer_id = 123
KEY(customer_id, status)
● Index is typically smaller than data
● Access is a lot more sequential

16. Overhead of
Indexing ● Indexes are costly. Do not add more than you
need.
○ In most cases extending index is better that adding
new one
● Writes
○ Updating indexes is often major cost of database writes
● Reads
○ Wasted space on disk and in memory.
○ Additional overhead during query optimization

17. Tools ● Explain
● Performance_Schema (MySQL 5.5+)
● MySQL sys Schema (MySQL 5.7.7+)
● pt-query-digest (Percona Toolkit)
● pt-index-usage (Percona Toolkit)
● mysqltuner

18. Explain
EXPLAIN is one of the most powerful
tools at your disposal for understanding
and optimizing troublesome MySQL
queries

19. Explain
Cost: 239*4145*1 = 990655

20. Explain
Alter table City add index c1 (Name)
Alter table Country add index c2 (Name)

21. Explain
Old Cost: 239*4145*1 = 990655 New Cost: 1*1*1 = 1

22. Performance Schema
Performance Schema is a mechanism to give user an
insight of what is happening behind the scene when
MySQL server is running.

23. Performance
Schema
● Introduced in MySQL 5.5
● New storage engine : Performance
Schema
● New Database : Performance Schema
● SQL user interface

24. Performance Schema - Instruments
● Name of the monitored activity
● Tree like structure separated by ‘/’.
● Left to right: More generic to more specific.
statement/sql/create_table
statement/sql/select
● 1000+ instruments in MySQL 5.7

25. Performance Schema - Instruments
TABLE SETUP_INSTRUMENTS
NAME ENABLED TIMED
statement/sql/select YES YES
statement/sql/create_table YES NO
statement/com/Create DB NO NO
…... ... ...

26. Performance Schema - use cases 1
● Multiple queries running for long on MySQL Server
● Few long running query (taking lots of time)
● No idea which one
● No idea why
● What to do ?

27. Performance Schema - use cases 1
SELECT digest_text , count_star, avg_timer_wait FROM
events_statements_summary_by_digest ORDER BY avg_timer_wait DESC
LIMIT 1G;

28. Performance Schema - use cases 2
Statement giving errors (or warning)
SELECT DIGEST_TEXT, SCHEMA_NAME, COUNT_STAR, SUM_ERRORS, SUM_WARNINGS FROM
performance_schema.event_statements_summary_by_digest WHERE SUM_ERRORS >0 ORDER BY
SUM_ERRORS DESC limit 1G;
select digest_text, schema_name, count_star, sum_errors, sum_warnings, first_seen, last_seen from
performance_schema.events_statements_summary_by_digest where digest =
'47d2c74d34a16ad5d2193a3706374c3e' G;

29. Performance Schema - use cases 2

30. Performance Schema - use cases 3
Problem Statement
● Multithreaded environment
● Session stuck
● Why
● What to do ??

31. Performance Schema - use cases 3
Diagnosis
● What t1 is waiting
select * from events_waits_current where thread_id = t1;
Say t1 is waiting for mutex1 (column: object_instance_begin)
● Lets see who has taken this mutex1
select * from mutex_instances where object_instance_begin = mutex1;
So thread t2 is holding mutex1 (column: locked_by_thread_id)
● Find out what thread t2 is waiting for
select * from events_waits_current where thread_id = t2;

32. Mysql sys Schema
● A set of objects that helps DBAs and developers interpret data collected by
the Performance Schema.
● Views that summarize Performance Schema data into more easily
understandable form.

33. Mysql sys Schema
Find out all unused indexes:
select * from sys.schema_unused_indexes;
Queries with errors or warnings:
select * from sys.statements_with_errors_or_warnings;

34. Pt-query-digest
● Command Line tool from Percona
Toolkit
● Process Log Manually
● Powerful Filtering
● Storing History

35. pt-query-digest

36. pt-query-digest
# 3.4s user time, 20ms system time, 37.16M rss, 99.46M vsz
# Current date: Wed Mar 29 13:58:09 2017
# Hostname: KELLGGNLPTP0129
# Files: /home/user/Downloads/SlaveDB-slow.log
# Overall: 7.25k total, 57 unique, 0.01 QPS, 0.80x concurrency ___________
# Time range: 2017-02-22 12:17:48 to 2017-03-02 12:15:43
# Attribute total min max avg 95% stddev median
# ============ ======= ======= ======= ======= ======= ======= =======
# Exec time 551459s 10s 83865s 76s 167s 1479s 19s
# Lock time 10s 0 6s 1ms 568us 65ms 348us
# Rows sent 56.58M 0 1.40M 7.99k 9.33k 25.57k 4.71k
# Rows examine 67.63G 0 1.10G 9.55M 46.53M 18.98M 3.68M
# Rows affecte 0 0 0 0 0 0 0
# Bytes sent 8.41G 0 43.24M 1.19M 2.49M 1.72M 1.03M
# Query size 13.89M 29 22.27k 1.96k 2.89k 874.90 2.06k

37. pt-query-digest
TOP QUERIES
# Profile
# Rank Query ID Response time Calls R/Call V/M Item
# ==== ================== ================= ===== ========== ===== =======
# 1 0x478608453385DE52 245460.6392 44.5% 8 30682.5799 35... SELECT stats_family_geo_mapping
beneficiary_mother users vhnd_pw_details delivery_details cronstats_child_immunization death_details
# 2 0x178ACC79136F8827 150300.4869 27.3% 906 165.8946 0.17 SELECT checklist_answer_summary
checklist_answers
# 3 0x621A20DE43CAD79C 53892.1029 9.8% 1625 33.1644 0.10 SELECT beneficiaries beneficiary_childs
Infant_details delivery_details users
# 4 0x18AD3F125971674C 19376.2565 3.5% 1368 14.1639 0.19 SELECT beneficiaries beneficiary_childs
Infant_details delivery_details users districts blocks sub_centers
# 5 0xA5AF83E712E06BB9 14995.0588 2.7% 940 15.9522 0.74 SELECT beneficiaries families users districts
blocks sub_centers villages
# 6 0x774412AFA6FD952C 10953.0720 2.0% 411 26.6498 1.11 SELECT stats_family_geo_mapping users
beneficiaries beneficiary_mother vhnd_pw_details delivery_details
# 7 0x8DA9B283102403CC 10652.1384 1.9% 8 1331.5173 0.75 SELECT delivery_details
stats_mother_hbnc_count families districts blocks users sub_centers stats_pnc_first_visit

38. pt-query-digest
QUERY DETAILS
# Query 1: 0.00 QPS, 0.43x concurrency, ID 0x478608453385DE52 at byte 9080661
# This item is included in the report because it matches --limit.
# Scores: V/M = 35585.94
# Time range: 2017-02-22 12:17:48 to 2017-03-01 01:10:30
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 0 8
# Exec time 44 245461s 29s 83865s 30683s 81750s 33043s 39323s
# Lock time 1 115ms 200us 112ms 14ms 110ms 36ms 366us
# Rows sent 0 6 0 1 0.75 0.99 0.43 0.99
# Rows examine 0 22.81M 1.47M 3.56M 2.85M 3.50M 841.46k 3.42M
# Rows affecte 0 0 0 0 0 0 0 0
# Bytes sent 0 414 0 69 51.75 65.89 28.53 65.89
# Query size 0 6.38k 817 817 817 817 0 817

39. pt-index-usage
● Read queries from a log and analyze how they use indexes.
● This tool connects to a MySQL database server, reads through a query log,
and uses EXPLAIN to ask MySQL how it will use each query.
● When it is finished, it prints out a report on indexes that the queries didn’t
use.
sudo pt-index-usage --ask-pass ~/Downloads/SlaveDB-slow.log

40. pt-index-usage
ALTER TABLE `msehat`.`sub_centers` DROP KEY `fk_village`; -- type:non-unique
ALTER TABLE `msehat`.`users` DROP KEY `block`, DROP KEY `device_imei`, DROP KEY
`fk_district_id`, DROP KEY `fk_villaged_id`, DROP KEY `idx_device_imei_secondary`, DROP KEY
`idx_parent_id`, DROP KEY `policy_id`, DROP KEY `sub_center`; -- type:non-unique
user@KELLGGNLPTP0129:/var/lib$ sudo pt-index-usage --help
pt-index-usage reads queries from logs and analyzes how they use indexes. For
more details, please use the --help option, or try 'perldoc
/usr/bin/pt-index-usage' for complete documentation.

41. MySQLTuner
MySQLTuner is a script written in Perl that allows you to
review a MySQL installation quickly and make
adjustments to increase performance and stability. The
current configuration variables and status data is
retrieved and presented in a brief format along with
some basic performance suggestions.

42. MySQLTuner
>> MySQLTuner 1.6.9 - Major Hayden <major@mhtx.net>
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering
[OK] Logged in using credentials from debian maintenance account.
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.7.13-0ubuntu0.16.04.2
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM
+MyISAM +PERFORMANCE_SCHEMA
[--] Data in InnoDB tables: 160M (Tables: 379)
[--] Data in MyISAM tables: 4M (Tables: 39)
[!!] Total fragmented tables: 180
-------- Security Recommendations -------------------------------------------
[OK] There are no anonymous accounts for any database users

43. Efficiency
Right Queries
Executed Most
Efficiently

44. Practical
Metrics How many rows are being scanned vs Sent ?
How much IO are we doing ?
Is there avoidable overhead ?
● Filesorts ?
● TMP Tables ?

45. Areas of Improvement
Architecture
Hardware
Schema and Queries
MySQL version
MySQL Configuration

46. Thank You