MySQL 8 (a huge leap forward), indexing capabilities, execution plan enhancements, optimizer improvements, and many other current query tweak features are covered in the slides.

1. Modern Query Optimisation
Features In MySQL 8
Karthik P R
CEO, Mydbops
Apr 24, 2021
Mydbops Webinar

2. Interested in Open Source Database technologies
11 Years of Experience with MySQL
Ex-Yahoo!
Tech Speaker/ Blogger
CEO Mydbops
Karthik P R
About Me

3. Database
Consulting
Services
Managed
Database
Services
Focuses on Top Opensource database MySQL,MariaDB,
MongoDB and PostgreSQL ON Premises and Cloud
Mydbops Services

4. MySQL 8 (A Major Leap)
Indexing features
Execution Plan Improvements
Optimiser Imporvements
Others Improvements
Agenda

5. MySQL 8 is a Major Leap

6. MySQL 8 is a Major Leap
MySQL 8
InnoDB Performance Enhancement
Faster Parallel Replication
Security Enhancements
Operational Improvements ( Shell,Clone Plugin )
CTE , Window Function and Lateral Derived tables

7. Indexing Features

8. Indexing Features
Functional Index
Descending Index
Invisible Index

9. Functional Index

10. Functional Index
MySQL do Support Functional Index 8.0.13
A functional key part of queries can be indexed
JSON based Queries benefits a lot

11. Functional Index ( Before )
EXPLAIN SELECT VERSION,TILE FROM NODES_BACKUP WHERE DATE(TIMESTAMP)='2021-04-04';
+----+-------------+--------------+------------+------+---------------+------+---------+------+----------+----------+-------------+
| ID | SELECT_TYPE | TABLE | PARTITIONS | TYPE | POSSIBLE_KEYS | KEY | KEY_LEN | REF | ROWS | FILTERED | EXTRA |
+----+-------------+--------------+------------+------+---------------+------+---------+------+----------+----------+-------------+
| 1 | SIMPLE | NODES_BACKUP | NULL | ALL | NULL | NULL | NULL | NULL | 47378283 | 100.00 | USING WHERE |
+----+-------------+--------------+------------+------+---------------+------+---------+------+----------+----------+-------------+
ROWS SCANNED : 47M
TYPE : FULL TABLE SCAN

12. Functional Index ( Possible Solutions )
Query Rewrites
Application Rewrites
Rewriter Plugin
ProxySQL Rewrites
Virtual Column ( Again needs a Rewrite )

13. Functional Index ( After )
CREATE INDEX IDX_FUN_DATE ON NODES_BACKUP((DATE(TIMESTAMP)))
QUERY OK, 0 ROWS AFFECTED (5 MIN 30.38 SEC)
RECORDS: 0 DUPLICATES: 0 WARNINGS: 0
SELECT VERSION,TILE FROM NODES_BACKUP WHERE DATE(TIMESTAMP)='2021-04-04';
+----+-------------+--------------+------------+------+---------------+--------------+---------+-------+------+----------+-------+
| ID | SELECT_TYPE | TABLE | PARTITIONS | TYPE | POSSIBLE_KEYS | KEY | KEY_LEN | REF | ROWS | FILTERED | EXTRA |
+----+-------------+--------------+------------+------+---------------+--------------+---------+-------+------+----------+-------+
| 1 | SIMPLE | NODES_BACKUP | NULL | REF | IDX_FUN_DATE | IDX_FUN_DATE | 4 | CONST | 423 | 100.00 | NULL |
+----+-------------+--------------+------------+------+---------------+--------------+---------+-------+------+----------+-------+
1 ROW IN SET, 1 WARNING (0.01 SEC)

14. Descending Index

15. Descending Index
MySQL do Support Functional Index 8.0
B+Tree Indexing ( InnoDB )
Index are generally Forward Scan
Queries with different sorting orders are benefited a lot
ASC / DESC Keywords must be used while indexing
Idenitify the queries for optimisation via "Backward index scan" in Extra
filed.

16. Descending Index
EXPLAIN SELECT EMAIL,DESCRIPTION FROM USERS ORDER BY CREATION_TIME DESC ,DISPLAY_NAME DESC LIMIT 10;
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+----------------+
| ID | SELECT_TYPE | TABLE | PARTITIONS | TYPE | POSSIBLE_KEYS | KEY | KEY_LEN | REF | ROWS | FILTERED | EXTRA |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+----------------+
| 1 | SIMPLE | USERS | NULL | ALL | NULL | NULL | NULL | NULL | 3941 | 100.00 | USING FILESORT |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+----------------+
CREATE INDEX IDX_CT_DN ON USERS(CREATION_TIME,DISPLAY_NAME);
QUERY OK, 0 ROWS AFFECTED (0.03 SEC)
EXPLAIN SELECT EMAIL,DESCRIPTION FROM USERS ORDER BY CREATION_TIME DESC ,DISPLAY_NAME DESC LIMIT 10;
+----+-------------+-------+------------+-------+---------------+-----------+---------+------+------+----------+---------------------+
| ID | SELECT_TYPE | TABLE | PARTITIONS | TYPE | POSSIBLE_KEYS | KEY | KEY_LEN | REF | ROWS | FILTERED | EXTRA |
+----+-------------+-------+------------+-------+---------------+-----------+---------+------+------+----------+---------------------+
| 1 | SIMPLE | USERS | NULL | INDEX | NULL | IDX_CT_DN | 772 | NULL | 10 | 100.00 | BACKWARD INDEX SCAN |
+----+-------------+-------+------------+-------+---------------+-----------+---------+------+------+----------+---------------------+

17. Descending Index
Query uses the Composite Index ( Creation Time , Display name )
No ordering is mentioned in while Indexing
Query uses Backward Scan ( Explain Plan )
But Forwards scan will have 15% Performance gain

18. Descending Index
CREATE INDEX IDX_CT_DN_DESC ON USERS(CREATION_TIME DESC,DISPLAY_NAME DESC);
QUERY OK, 0 ROWS AFFECTED (0.03 SEC)
RECORDS: 0 DUPLICATES: 0 WARNINGS: 0
EXPLAIN SELECT EMAIL,DESCRIPTION FROM USERS ORDER BY CREATION_TIME DESC ,DISPLAY_NAME DESC LIMIT 10;
+----+-------------+-------+------------+-------+---------------+----------------+---------+------+------+----------+-------+
| ID | SELECT_TYPE | TABLE | PARTITIONS | TYPE | POSSIBLE_KEYS | KEY | KEY_LEN | REF | ROWS | FILTERED | EXTRA |
+----+-------------+-------+------------+-------+---------------+----------------+---------+------+------+----------+-------+
| 1 | SIMPLE | USERS | NULL | INDEX | NULL | IDX_CT_DN_DESC | 772 | NULL | 10 | 100.00 | NULL |
+----+-------------+-------+------------+-------+---------------+----------------+---------+------+------+----------+-------+

19. Descending Index ( Limitation )
Only Supports InnoDB Engine
Full text is not supported
They are not Buffered in Change Buffer

20. Invisible Index

21. Indexing for Performance ( Process followed Before )
Find the Query for optimisation
Create the index ( Affects Optimiser )
Evaluate the Query Performane
Retain the Index if it benefits queries
Drop the Index if it is not much benefit

22. Indexing for Performance ( After )
Note : Can be followed for dropping an unused index too.
Find the Query for optimisation
Create the index using Invisible as Keyword
Turn on Switch 'use_invisible-indexes=on' ( session )
Evaluate the Query Performane now.
Make the index Visible if it benefits.
Drop the Index if it is not much benefit.

23. Execution Plan Improvements

24. Execution Plan Improvements
Explain Format in Tree
Explain Analyze

25. Explain Format=Tree

26. Explain Format=Tree
Explain Format in Tree was introduced in MySQL 8.0.16
Shows Query Plans and Cost estimations
Easy understanding on internal operations
Identation helps understanding the execution orders

27. Explain ( Traditional / Tabular )
explain SELECT 'node' as type, node_id as id FROM node_tags WHERE k='bridge' and v='memorial' UNION SELECT 'way' as type, way_id as id FROM
way_tags WHERE k='lanes' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='tunnel' and
v='Barkingside';
+----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+
| 1 | PRIMARY | node_tags | NULL | ALL | NULL | NULL | NULL | NULL | 822783 | 1.00 | Using where |
| 2 | UNION | way_tags | NULL | ALL | NULL | NULL | NULL | NULL | 1347360 | 1.00 | Using where |
| 3 | UNION | relation_tags | NULL | ALL | NULL | NULL | NULL | NULL | 62797 | 1.00 | Using where |
|NULL| UNION RESULT | <union1,2,3> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | NULL | Using temporary |
+----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+

28. Explain Format=Tree
explain format=tree SELECT 'node' as type, node_id as id FROM node_tags WHERE k='bridge' and v='memorial' UNION SELECT 'way' as type, way_id
as id FROM
way_tags WHERE k='lanes' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='tunnel' and
v='Barkingside'G
*************************** 1. row ***************************
EXPLAIN: -> Table scan on <union temporary> (cost=0.01..281.61 rows=22329)
-> Union materialize with deduplication (cost=227556.45..227838.05 rows=22329)
-> Filter: ((node_tags.k = 'bridge') and (node_tags.v = 'memorial')) (cost=83072.55 rows=8228)
-> Table scan on node_tags (cost=83072.55 rows=822783)
-> Filter: ((way_tags.k = 'lanes') and (way_tags.v = 'cafe')) (cost=135899.00 rows=13474)
-> Table scan on way_tags (cost=135899.00 rows=1347360)
-> Filter: ((relation_tags.k = 'tunnel') and (relation_tags.v = 'Barkingside')) (cost=6351.95 rows=628)
-> Table scan on relation_tags (cost=6351.95 rows=62797)

29. Explain Analyze

30. Explain Analyze
Explain Analyse was introduced in MySQL 8.0.18
Actual Time spent on each Iterator.
Time to fetch first row
Time to fetch all rows
Better than Optimiser Trace

31. Explain Analyze
explain analyze SELECT 'node' as type, node_id as id FROM node_tags WHERE k='bridge' and v='memorial' UNION SELECT 'way' as type, way_id as
id FROM way_tags WHERE k='lanes' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='tunnel' and
v='Barkingside'G
*************************** 1. row ***************************
EXPLAIN: -> Table scan on <union temporary> (cost=0.01..281.61 rows=22329) (actual time=0.002..0.002 rows=0 loops=1)
-> Union materialize with deduplication (cost=227556.45..227838.05 rows=22329) (actual time=636.728..636.728 rows=0 loops=1)
-> Filter: ((node_tags.k = 'bridge') and (node_tags.v = 'memorial')) (cost=83072.55 rows=8228) (actual time=233.393..233.393 rows=0
loops=1)
-> Table scan on node_tags (cost=83072.55 rows=822783) (actual time=0.039..182.413 rows=833423 loops=1)
-> Filter: ((way_tags.k = 'lanes') and (way_tags.v = 'cafe')) (cost=135899.00 rows=13474) (actual time=385.278..385.278 rows=0
loops=1)
-> Table scan on way_tags (cost=135899.00 rows=1347360) (actual time=0.039..299.565 rows=1343477 loops=1)
-> Filter: ((relation_tags.k = 'tunnel') and (relation_tags.v = 'Barkingside')) (cost=6351.95 rows=628) (actual time=18.046..18.046
rows=0 loops=1)
-> Table scan on relation_tags (cost=6351.95 rows=62797) (actual time=0.039..14.101 rows=63220 loops=1)

32. Explain Analyze
Explain Tree ( Cost ) Analyze (time in ms)
Primary 83072 233
UNION 135899 385
UNION 6352 18
UNION Result 227556 636
Note : Analyze result is 15-30% slower than actual time

33. Optimiser Improvements

34. Optimiser Improvements
Optimiser is the brain of any RDBMS , better algorithms and better statistics will
make its intelligence better.
Histogram
Hash Joins

35. Histogram

36. Histogram
Syntax :
analyze table table_name update/drop histogram with N buckets ;
The Data distribution is not uniform
Histogram helps in Better stat to DB.
MySQL has Histogram from 8.0.3 ( before GA )
Histogram can be applied for single column / multi columns.

37. Histogram
Title
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
0 1​ 2​
​
series1
Count value
1 1 0
2 3874 1
3 116 2
Data Distribution

38. Histogram (Before)
Optimizer Guesstimate the row filtering is common

39. Histogram (After)
Row filtering has improved post Histogram

40. Histogram
Histograms are better than indexes at cases.
Lesser maintenance over head.
Controlled by condition_fanout_filter.
1024 is the maximum buckets allowed.
Histogram stats for table can be visualised on
COLUMN_STATISTICS ( IS table ).

41. Hash Join

42. Optimiser Improvements

43. Hash Join
MySQL Support Hash Join from 8.0.18
Improved further in 8.0.20 ( Left Join )
HASH Join algorithm help in better Join Optimisation than BNL.
Equi Joins are much benefitted
BNL Support was removed from MySQL 8.0.20

44. Hash Join
In memory Hash table
Join Buffer Size Prevents over flow
Optimiser switch hash_join=ON (can't be disabled )

45. Other Optimisation
Perfer_orderding_index
sub_query_to-derived
CTE ( Common Table expression )
Lateral Derived Tables
Window Function
Switches

46. Reach Us : Info@mydbops.com
Thank You