Performance Schema in MySQL (Danil Zburivsky)

Performance Schema in
MySQL
Danil Zburivsky
MySQL DBA and Team Lead at Pythian

Tuesday, October 23, 12

About myself
• MySQL DBA and Team
Lead at Pythian

• Managing dozens of
customers and
thousands of MySQL
servers

• http://www.pythian.com/
news/author/zburivsky/

• @zburivsky


Debugging MySQL
Performance


MySQL is a great database, but
instrumentation sucks

Slow query log SHOW ENGINE
INNODB STATUS

OS STATS
SHOW
PROCESSLIST


I think there is a problem with
database...


database...
• Identify timeframe (use trending tools)


database...
• Collect data during incident


database...
• Collect more data


database...
• Collect as much data as you can!


database...
• Try to make sense of it


database...
• Tune settings, SQL, hardware. Did it help?


database...
• Tune settings, SQL, hardware. Did it help?
• I think there is a problem with database...


Tools that might help you

• pt-stalk (http://www.percona.com/doc/
percona-toolkit/2.1/pt-stalk.html)
• innotop (http://code.google.com/p/
innotop/)
• Or you write your own stuff


What do “Big Boys” have?
• Oracle: SQL-traces,
kernel events timings,
tons of books on
performance tuning

• SQL Server: SQL
proﬁler, Data
Management Views,
Extended Events,tons
of books on
performance tuning


Performance schema in
MySQL 5.5


The Idea

• Executing a query breaks down to
hundreds of smaller tasks
• There are background tasks as well
• We want to instrument it all to know
where server is spending time


Implementation

• Instrumentation: measuring when event
begins and ends
• Implemented in MySQL code on server
end storage engine level
• Can be enabled/disabled or customized


Consumers

SELECT * FROM setup_consumers;
+----------------------------------------------+---------+
| NAME | ENABLED |
+----------------------------------------------+---------+
| events_waits_current | YES |
| events_waits_history | YES |
| events_waits_history_long | YES |
| events_waits_summary_by_thread_by_event_name | YES |
| events_waits_summary_by_event_name | YES |
| events_waits_summary_by_instance | YES |
| file_summary_by_event_name | YES |
| file_summary_by_instance | YES |
+----------------------------------------------+---------+


What are top wait events in
my server?
SELECT COUNT_STAR, SUM_TIMER_WAIT, AVG_TIMER_WAIT
FROM events_waits_summary_by_instance
ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;
+---------------------------------------------+------------+-------------------+----------------+
| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |
+---------------------------------------------+-----------------------+---------------+---------|
| wait/io/file/innodb/innodb_data_file | 3347631 | 13547848609831118 | 4046995803 |
| wait/synch/rwlock/innodb/index_tree_rw_lock | 69535441 | 2000129499580014 | 28764173 |
| wait/synch/mutex/innodb/buf_pool_mutex | 800909600 | 433331246983734 | 541048 |
| wait/synch/mutex/innodb/log_sys_mutex | 181356697 | 420610626124392 | 2319245 |
| wait/synch/rwlock/innodb/btr_search_latch | 192529196 | 161285917556962 | 837721 |
| wait/synch/mutex/sql/LOCK_open | 285022231 | 154822982324406 | 543196 |
| wait/synch/mutex/innodb/kernel_mutex | 403820715 | 130291049763820 | 322645 |
| wait/synch/mutex/mysys/THR_LOCK::mutex | 285017708 | 121061567338910 | 424751 |
+---------------------------------------------+------------+-------------------+----------------+


innodb_buffer_pool_instances=10

SELECT * FROM events_waits_summary_by_instance
ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;
+-----------------------------------------------+------------+-------------------+----------------+
| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |
+-----------------------------------------------+------------+-------------------+----------------+
| wait/synch/mutex/innodb/trx_doublewrite_mutex | 6432834 | 592011487365760 | 92029654 |
| wait/synch/mutex/innodb/log_sys_mutex | 183016201 | 459928639055248 | 2513048 |
| wait/synch/rwlock/innodb/btr_search_latch | 190363676 | 183251774563990 | 962640 |
| wait/synch/mutex/sql/LOCK_open | 285022522 | 144911560909258 | 508421 |
| wait/synch/mutex/mysys/THR_LOCK::mutex | 285018012 | 130317830152708 | 457226 |
| wait/synch/mutex/innodb/kernel_mutex | 401734769 | 126815854421630 | 315670 |
+-----------------------------------------------+------------+-------------------+----------------+

buf_pool_mutex is no longer in top 10 wait events


Which ﬁles are being accessed
the most?
mysql> SELECT SUM_NUMBER_OF_BYTES_READ, SUM_NUMBER_OF_BYTES_WRITE
FROM file_summary_by_instance
ORDER BY SUM_NUMBER_OF_BYTES_READ+SUM_NUMBER_OF_BYTES_WRITE DESC LIMIT 10;
+------------------------------+--------------------------+---------------------------+
| FILE_NAME |SUM_NUMBER_OF_BYTES_READ | SUM_NUMBER_OF_BYTES_WRITE |
+------------------------------+--------------------------+---------------------------+
| /var/lib/mysql/ibdata1 | 166920192 | 363649155072 |
| /var/lib/mysql/WIR.ibd | 180310720512 | 114136367104 |
| /var/lib/mysql/EMIWPR.ibd | 51803815936 | 71776468992 |
| /var/lib/mysql/EWAPR.ibd | 38596755456 | 54104932352 |
| /var/lib/mysql/OFPR.ibd | 24171593728 | 33964572672 |
| /var/lib/mysql/OFIR.ibd | 38002688000 | 14265303040 |
| /var/lib/mysql/EQPO.ibd | 12718882816 | 21407596544 |
| /var/lib/mysql/FIR.ibd | 19646103552 | 10337140736 |
| /var/lib/mysql/EIR.ibd | 14708899840 | 12814794752 |
| /var/lib/mysql/FPR.ibd | 6360350720 | 9764585472 |
+------------------------------+--------------------------+---------------------------+


Threads table
(performance_schema) > SELECT * FROM threads;
+-----------+----------------+----------------------------------------+
| THREAD_ID | PROCESSLIST_ID | NAME |
+-----------+----------------+----------------------------------------+
| 0 | 0 | thread/sql/main |
| 12 | 0 | thread/innodb/srv_lock_timeout_thread |
| 14 | 0 | thread/innodb/srv_monitor_thread |
| 16 | 0 | thread/innodb/srv_purge_thread |
| 13 | 0 | thread/innodb/srv_error_monitor_thread |
| 2 | 0 | thread/innodb/io_handler_thread |
| 25 | 8 | thread/sql/one_connection |
| 15 | 0 | thread/innodb/srv_master_thread |
| 17 | 0 | thread/sql/signal_handler |
+-----------+----------------+----------------------------------------+


Average wait per thread (I)

SELECT e.THREAD_ID, e.EVENT_NAME, MAX(e.AVG_TIMER_WAIT)
FROM events_waits_summary_by_thread_by_event_name e
LEFT JOIN threads t ON t.THREAD_ID = e.THREAD_ID
WHERE event_name LIKE 'wait/synch/mutex/innodb/%'
AND t.NAME='thread/sql/one_connection'
AND AVG_TIMER_WAIT > 0
GROUP BY t.thread_id, e.EVENT_NAME
ORDER BY t.thread_id, MAX(e.AVG_TIMER_WAIT)


Average wait per thread (II)
+-----------+-------------------------------------------------------+-------------------------+
| THREAD_ID | EVENT_NAME | MAX(AVG_TIMER_WAIT) |
+-----------+-------------------------------------------------------+-------------------------+
| 27 | wait/synch/mutex/innodb/mutex_list_mutex | 49463 |
| 27 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83247 |
.....
.....
| 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 115681 |
| 27 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 168537 |
| 27 | wait/synch/mutex/innodb/ibuf_mutex | 282400 |
| 27 | wait/synch/mutex/innodb/log_sys_mutex | 1690694 |
----------------------------------------------------------------------------------------------
| 28 | wait/synch/mutex/innodb/flush_list_mutex | 52375 |
| 28 | wait/synch/mutex/innodb/log_flush_order_mutex | 62040 |
.....
.....
| 28 | wait/synch/mutex/innodb/autoinc_mutex | 63012 |
| 28 | wait/synch/mutex/innodb/dict_sys_mutex | 181203 |
| 28 | wait/synch/mutex/innodb/innobase_share_mutex | 239514 |
| 28 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 266684 |


Average wait per thread (III)
innodb_change_buffering = none [Don’t try this at home! ]
+-----------+--------------------------------------------------+-------------------------+
| THREAD_ID | EVENT_NAME | MAX(ebt.AVG_TIMER_WAIT) |
+-----------+--------------------------------------------------+-------------------------+
.....
.....
| 27 | wait/synch/mutex/innodb/buf_pool_mutex | 101598 |
| 27 | wait/synch/mutex/innodb/kernel_mutex | 148332 |
-----------------------------------------------------------------------------------------
....
....
| 28 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83545 |
| 28 | wait/synch/mutex/innodb/buf_pool_mutex | 101365 |
| 28 | wait/synch/mutex/innodb/kernel_mutex | 149135 |


Picoseconds, shmicosecond! ps_helper to
rescue!

• http://www.markleith.co.uk/ps_helper/
• Great examples of using
performance_schema
• Useful tools for converting time, bytes,
paths into human readable format


Example:
top_global_io_consumers_by_bytes_usage
(ps_helper) > SELECT event_name, total_read, avg_read, total_written, avg_written,avg_latency
FROM top_global_io_consumers_by_bytes_usage;
+-------------------------+------------+-----------+---------------+-------------+-------------+
| event_name | total_read | avg_read | total_written | avg_written | avg_latency |
+-------------------------+------------+-----------+---------------+-------------+-------------+
| innodb/innodb_data_file | 10.76 GiB | 16.00 KiB | 16.05 GiB | 30.90 KiB | 10.96 ms |
| innodb/innodb_log_file | 68.00 KiB | 11.33 KiB | 1.03 GiB | 369.66 KiB | 2.33 ms |
| myisam/dfile | 197.79 KiB | 8.60 KiB | 187.04 KiB | 5.84 KiB | 9.89 µs |
| sql/FRM | 142.28 KiB | 491 bytes | 4.50 KiB | 242 bytes | 3.66 µs |
| sql/ERRMSG | 43.68 KiB | 14.56 KiB | 0 bytes | 0 bytes | 15.42 µs |
| myisam/kfile | 13.22 KiB | 226 bytes | 5.96 KiB | 24 bytes | 8.26 µs |
| mysys/charset | 17.83 KiB | 17.83 KiB | 0 bytes | 0 bytes | 10.68 µs |
| sql/partition | 1.06 KiB | 32 bytes | 0 bytes | 0 bytes | 4.38 µs |
| sql/pid | 0 bytes | 0 bytes | 6 bytes | 6 bytes | 34.78 µs |
| sql/global_ddl_log | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 3.69 µs |
| archive/data | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.35 µs |
| sql/dbopt | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.55 µs |
| sql/casetest | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 28.08 µs |
+-------------------------+------------+-----------+---------------+-------------+-------------+


No free lunch: PFS overhead

• Instrumentation doesn’t come for free
• In CPU-bound workloads overhead can be
~15-20%
• In IO-bound — 5%-8%
• Signiﬁcant improvements in 5.6


Performance schema in
MySQL 5.6


What’s new in 5.6?

• performance_schema enabled by default!
• Less overhead: 5%-10% for CPU-bound
workload
• Statements, Stages, Actors and Objects


Actors: ﬁlter events by user

(performance_schema) > SELECT * FROM setup_actors;
+------+------+------+
| HOST | USER | ROLE |
+------+------+------+
| % | % | % |
+------+------+------+
1 row in set (0.00 sec)


Objects: ﬁlter out events by
database/table
(performance_schema) > SELECT * FROM setup_objects;
+-------------+--------------------+-------------+---------+-------+
| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |
+-------------+--------------------+-------------+---------+-------+
| TABLE | mysql | % | NO | NO |
| TABLE | performance_schema | % | NO | NO |
| TABLE | information_schema | % | NO | NO |
| TABLE | % | % | YES | YES |
+-------------+--------------------+-------------+---------+-------+


Statements
SELECT * FROM events_statements_summary_by_digestG

*************************** 4. row ***************************
DIGEST: f60b0866d7cbd78d39caf5fb52f43bfb
DIGEST_TEXT: SELECT * FROM `table_io_waits_summary_by_index_usage` ORDER BY
`SUM_TIMER_WAIT` DESC LIMIT ?
COUNT_STAR: 1
SUM_TIMER_WAIT: 511539000
MIN_TIMER_WAIT: 511539000
AVG_TIMER_WAIT: 511539000
MAX_TIMER_WAIT: 511539000
SUM_LOCK_TIME: 72000000
SUM_ERRORS: 0
SUM_WARNINGS: 0
SUM_ROWS_AFFECTED: 0
SUM_ROWS_SENT: 5
SUM_ROWS_EXAMINED: 91
SUM_CREATED_TMP_DISK_TABLES: 0
SUM_CREATED_TMP_TABLES: 0
SUM_SELECT_FULL_JOIN: 0
SUM_SELECT_FULL_RANGE_JOIN: 0
SUM_SELECT_RANGE: 0
SUM_SELECT_RANGE_CHECK: 0
SUM_SELECT_SCAN: 1
SUM_SORT_MERGE_PASSES: 0
SUM_SORT_RANGE: 0
SUM_SORT_ROWS: 5
SUM_SORT_SCAN: 1
SUM_NO_INDEX_USED: 1
SUM_NO_GOOD_INDEX_USED: 0
FIRST_SEEN: 2012-10-16 21:50:13
LAST_SEEN: 2012-10-16 21:50:13


Summary. Pros.

• Provides insight into internal server metric
• Flexible
• Relational model allows to build your own
views on top of this data


Summary. Cons.

• Steep learning curve: not very well
documented
• Overhead for CPU-bound loads can be
signiﬁcant


Thank you!


Performance Schema in MySQL (Danil Zburivsky)

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Performance Schema in MySQL (Danil Zburivsky)

Ähnlich wie Performance Schema in MySQL (Danil Zburivsky) (20)

Mehr von Ontico

Mehr von Ontico (20)

Performance Schema in MySQL (Danil Zburivsky)