Information about optimizing the queries in MySQL

1. Practical MySQL Practical MySQL
Performance Performance
OptimizationOptimization
Marian HackMan MarinovMarian HackMan Marinov
<mm@siteground.com><mm@siteground.com>
Chief System ArchitectChief System Architect
SiteGroundSiteGround

2. Who am I?Who am I?Who am I?Who am I?
❖ Chief System Architect of Siteground.com
❖ Sysadmin since 1996
❖ Organizer of OpenFest, BG Perl Workshops,
LUG-BG and similar :)
❖ Teaching Network Security and Linux System
Administration at Sofia University

3. ❖ MySQL 5.7
❖ Or its equivalent from MariaDB or Percona
Requirements...Requirements...

4. ❖ Why am I making this presentation?
❖ Where is the information coming from?
MySQL Optimization Documentation
❖ I have skipped the SQL examples as they
made the presentation too big ☹
DISCLAMERDISCLAMER

5. ❖ Manual monitoring
❖ Using Monyog
MonitoringMonitoring

6. How MySQL uses indexesHow MySQL uses indexes
❖ What type of indexes are there?

7. How MySQL uses indexesHow MySQL uses indexes
❖ What type of indexes are there?
 B-Tree
 Hash
FULLTEXT
Spatial

8. ❖ What type of indexes are there?
❖ Which is used for what ?
 hash
  used for = or <=>
  can not be used in ORDER BY
  only whole keys can be used for search
How MySQL uses indexesHow MySQL uses indexes

9. ❖ What type of indexes are there?
❖ Which is used for what ?
 hash
How MySQL uses indexesHow MySQL uses indexes

10. ❖ What type of indexes are there?
❖ Which is used for what ?
 b-tree
  can be used for =, >, >=, <, <=, or
BETWEEN comparisons
  can be used with LIKE, only if it doesn't start
with %
  can not be used in the form
column1 LIKE column2
  can limit the size of the index (prefix)
How MySQL uses indexesHow MySQL uses indexes

11. ❖ When is index NOT used
How MySQL uses indexesHow MySQL uses indexes

12. ❖ When is index NOT used
when the index does not cover all the
columns in the where clause
How MySQL uses indexesHow MySQL uses indexes

13. ❖ When is index NOT used
when the index does not cover all the
columns in the where clause
when the types don't match: integer used
for string index(column)
How MySQL uses indexesHow MySQL uses indexes

14. ❖ When is index NOT used
when the index does not cover all the
columns in the where clause
when the types don't match: integer used
for string index(column)
when != is used
How MySQL uses indexesHow MySQL uses indexes

15. ❖ When is index NOT used
when the index does not cover all the
columns in the where clause
when the types don't match: integer used
for string index(column)
when != is used
when you have a multicolumn index but
the column you are referencing is not the
first one and you don't have a separate index
How MySQL uses indexesHow MySQL uses indexes

16. ❖ When is index NOT used
 when you have a Btree index but you use
distinct on the column
How MySQL uses indexesHow MySQL uses indexes

17. ❖ When is index NOT used
 when you have a Btree index but you use
distinct on the column
 when you have a single column indexes
but you are actually referencing multiple
columns in your WHERE/JOIN
How MySQL uses indexesHow MySQL uses indexes

18. ❖ When is index NOT used
 when you have a Btree index but you use
distinct on the column
 when you have a single column indexes
but you are actually referencing multiple
columns in your WHERE/JOIN
 when LIKE is used with % in the begining
e.g. LIKE '%name'
How MySQL uses indexesHow MySQL uses indexes

19. ❖ When is index NOT used
 when you have a Btree index but you use
distinct on the column
 when you have a single column indexes
but you are actually referencing multiple
columns in your WHERE/JOIN
 when LIKE is used with % in the begining
e.g. LIKE '%name'
 order of columns in indexes MATTERS
How MySQL uses indexesHow MySQL uses indexes

20. 1410864 rows
1 row in set (10.46 sec)
1 row in set (8.33 sec)
1 row in set (6.66 sec)
Index hintsIndex hints

21. ❖ IGNORE INDEX
Index hintsIndex hints

22. ❖ IGNORE INDEX
USE INDEX❖
Index hintsIndex hints

23. ❖ IGNORE INDEX
❖ USE INDEX
❖ FORCE INDEX
Index hintsIndex hints

24. ❖ "Waiting for query cache lock"
Abusing query cacheAbusing query cache

25. ❖ "Waiting for query cache lock"
❖ SELECT *,NOW() FROM TABLE
Abusing query cacheAbusing query cache

26. ❖ "Waiting for query cache lock"
❖ SELECT *,NOW() FROM TABLE
❖ SQL_NO_CACHE
Abusing query cacheAbusing query cache

27. ❖ "Waiting for query cache lock"
❖ SELECT *,NOW() FROM TABLE
❖ SQL_NO_CACHE
❖ SET SESSION query_cache_type = OFF;
Abusing query cacheAbusing query cache

28. ❖ "Waiting for query cache lock"
❖ SELECT *,NOW() FROM TABLE
❖ SQL_NO_CACHE
❖ SET SESSION query_cache_type = OFF;
❖ Аs of MySQL 5.6.8 query cache is now
disabled by default
Abusing query cacheAbusing query cache

29. ❖ Large result sets hammer the network...
Abusing query cacheAbusing query cache

30. ❖ Large result sets hammer the network...
❖ For caching large results move the cache,
closer to the application by caching the results
in ProxySQL
Abusing query cacheAbusing query cache

31. ❖ InooDB uses only a single index per query
Optimizing InnoDB queriesOptimizing InnoDB queries

32. ❖ InooDB uses only a single index per query
❖ If an indexed column cannot contain any
NULL values, declare it as NOT NULL
Optimizing InnoDB queriesOptimizing InnoDB queries

33. ❖ InooDB uses only a single index per query
❖ If an indexed column cannot contain any
NULL values, declare it as NOT NULL
❖ START TRANSACTION READ ONLY
Optimizing InnoDB queriesOptimizing InnoDB queries

34. ❖ use ANALYZE TABLE or myisamchk
--analyze
Optimizing MyISAM queriesOptimizing MyISAM queries

35. ❖ use ANALYZE TABLE or myisamchk
--analyze
❖ myisamchk --sort-index --sort-records=1
Optimizing MyISAM queriesOptimizing MyISAM queries

36. ❖ use ANALYZE TABLE or myisamchk
--analyze
❖ myisamchk --sort-index --sort-records=1
❖ avoid complex SELECT queries on
MyISAM tables that are updated frequently
Optimizing MyISAM queriesOptimizing MyISAM queries

37. ❖ use ANALYZE TABLE or myisamchk
--analyze
❖ myisamchk --sort-index --sort-records=1
❖ avoid complex SELECT queries on
MyISAM tables that are updated frequently
❖ MyISAM supports concurrent inserts
Optimizing MyISAM queriesOptimizing MyISAM queries

38. ❖ use ANALYZE TABLE or myisamchk
--analyze
❖ myisamchk --sort-index --sort-records=1
❖ avoid complex SELECT queries on MyISAM
tables that are updated frequently
❖ MyISAM supports concurrent inserts
concurrent_insert = 0 (disabled)
concurrent_insert = 1 (auto)
concurrent_insert = 1 (always, even with
deleted rows)
Optimizing MyISAM queriesOptimizing MyISAM queries

39. ❖ use ANALYZE TABLE or myisamchk
--analyze
❖ myisamchk --sort-index --sort-records=1
❖ avoid complex SELECT queries on
MyISAM tables that are updated frequently
❖ MyISAM supports concurrent inserts
❖ try to avoid VARCHAR, BLOB, and TEXT in
MyISAM tables
Optimizing MyISAM queriesOptimizing MyISAM queries

40. ❖ use ANALYZE TABLE or myisamchk
--analyze
❖ myisamchk --sort-index --sort-records=1
❖ avoid complex SELECT queries on
MyISAM tables that are updated frequently
❖ MyISAM supports concurrent inserts
❖ try to avoid VARCHAR, BLOB, and TEXT in
MyISAM tables
❖ Don't split large(columns) tables
Optimizing MyISAM queriesOptimizing MyISAM queries

41. ❖ use ANALYZE TABLE or myisamchk
--analyze
❖ myisamchk --sort-index --sort-records=1
❖ avoid complex SELECT queries on MyISAM
tables that are updated frequently
❖ MyISAM supports concurrent inserts
❖ try to avoid VARCHAR, BLOB, and TEXT in
MyISAM tables
❖ Don't split large(columns) tables
❖ use DELAY_KEY_WRITE=1
Optimizing MyISAM queriesOptimizing MyISAM queries

42. mysql> SELECT @@optimizer_switchG
*************************** 1. row ***************************
@@optimizer_switch: index_merge=on,index_merge_union=on,
index_merge_sort_union=on,
index_merge_intersection=on,
engine_condition_pushdown=on,
index_condition_pushdown=on,
mrr=on,mrr_cost_based=on,
block_nested_loop=on,batched_key_access=off,
materialization=on,semijoin=on,loosescan=on,
firstmatch=on,duplicateweedout=on,
subquery_materialization_cost_based=on,
use_index_extensions=on,
condition_fanout_filter=on,derived_merge=on
Controlling the OptimizerControlling the Optimizer
MySQL 5.7
only

43. ❖ Merging multiple index scans on a SINGLE
table
index_merge=off (default on)
index_merge_union=on
index_merge_sort_union=on
index_merge_intersection=on
low selectivity indexes are very slow with
intersect
Controlling the OptimizerControlling the Optimizer

44. ❖ Selecting different merge algorithms per
table (only for 5.7)
Batched Key Access join /*+ BKA(t1) */
Block Nested-Loop join /*+ BNL(t1, t2) */
Controlling the OptimizerControlling the Optimizer

45. ❖ Optimizing subqueries
Controlling the OptimizerControlling the Optimizer

46. ❖ Optimizing subqueries
Materialization strategy
Controlling the OptimizerControlling the Optimizer

47. ❖ Optimizing subqueries
Materialization strategy
Exists strategy
Controlling the OptimizerControlling the Optimizer

48. ❖ Optimizing subqueries
Materialization strategy
Exists strategy
  (usually preferred when the sub queries
can produce NULL results)
Controlling the OptimizerControlling the Optimizer

49. ❖ Optimizing subqueries
Materialization strategy
Exists strategy
  (usually preferred when the sub queries
can produce NULL results)
subquery_materialization_cost_based
Controlling the OptimizerControlling the Optimizer

50. ❖ use SQL_SMALL_RESUL
❖ If all columns in the index are numeric MySQL will
read only the INDEX
❖ Avoid large amounts of values in IN statements
 values in an IN() list count as a predicates
combined with OR
use eq_range_index_dive_limit to control the index
dives in range comparisons(IN) 5.7
range optimization can't be used with NOT IN()
 /*+ NO_RANGE_OPTIMIZATION(t4 PRIMARY) */
❖ Index Condition Pushdown 5.7
General optimizationGeneral optimization

51. ❖ Avoid full table scans
--max-seeks-for-key=1000
SSD or NVMe
General optimizationGeneral optimization

52. # Turn tracing on (it's off by default):
SET optimizer_trace="enabled=on";
SELECT ...; # your query here
SELECT * FROM
INFORMATION_SCHEMA.OPTIMIZER_TRACE;
# possibly more queries...
# When done with tracing, disable it:
SET optimizer_trace="enabled=off";
Using the tracerUsing the tracer

53. "filesort_summary": {
"rows": 100,
"examined_rows": 100,
"number_of_tmp_files": 0,
"sort_buffer_size": 25192,
"sort_mode": "<sort_key,
packed_additional_fields>"
}
Using the tracerUsing the tracer

54. ❖ Consider using INSERT with multiple VALUES
lists
❖ Updates on tables with many indexes may
become slow
❖ Updates with selects in them may block
because of query cache lock contention
❖ Deletes also suffer from the above problems
Changing dataChanging data

55. ❖ Splitting less frequently used columns from
large(column) tables using foreign keys
Optimizing architectureOptimizing architecture

56. ACID RainACID Rain
Concurrency-Related Attacks onConcurrency-Related Attacks on
Database-Backed Web ApplicationsDatabase-Backed Web Applications
ACID RainACID Rain
Concurrency-Related Attacks onConcurrency-Related Attacks on
Database-Backed Web ApplicationsDatabase-Backed Web Applications
Marian HackMan Marinov
<mm@siteground.com>

57. THANK YOUTHANK YOUTHANK YOUTHANK YOU
Marian HackMan Marinov
<mm@siteground.com>