1. PERFORMANCE_SCHEMA
in MySQL 5.6:
The Good, The Bad and The Ugly
Valerii Kravchuk, Principal Support Engineer
valerii.kravchuk@percona.com

2. Who am I?
Valerii (aka Valeriy) Kravchuk (you may call me “Larry”):
●
●
●
●
●
●
MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005 - 2012
○ Bugs Verification Team all this time
○ Support issues related to bugs, InnoDB, performance and more
○ Trainings (mostly informal) for new team members
○ All kinds of decision making committees…
Principal Support Engineer in Percona, 2012 - ...
○ Doing more or less the same as before, but better (I hope)...
○ Plus I speak and write about MySQL in public now!
http://mysqlentomologist.blogspot.com - my blog about MySQL bugs
https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about
MySQL (mostly bugs…)
http://bugs.mysql.com - my personal playground
Yes, I am Orthodox, in many senses...
www.percona.com

3. What is this session about?
“The Good, the Bad and the Ugly (Italian title:
Il buono, il brutto, il cattivo) is a 1966 Italian epic
Spaghetti western film directed by Sergio Leone,
starring Clint Eastwood, Lee Van Cleef, and Eli
Wallach in the title roles respectively.
...
It is the third film in the Dollars Trilogy following
A Fistful of Dollars (1964) and For a Few Dollars
More (1965). The plot revolves around three
gunslingers competing to find a fortune in buried
Confederate gold amid the violent chaos of
gunfights, hangings, American Civil War battles
and prison camps.”
http://en.wikipedia.org/wiki/The_Good,_the_Bad_and_the_Ugly
www.percona.com

4. So, what is this session really about?
●
●
●
●
●
How to find the root causes of MySQL server performance problems?
○ http://method-r.com/faq/35-what-is-method-r
○ Where time is spent?
Proper instrumentation of the code, is it really important?
○ http://tkyte.blogspot.com/2005/06/instrumentation.html
○ http://en.wikipedia.org/wiki/Observer_effect_%28physics%29
OS-level debug, trace and profiling tools, aren’t they allow to pinpoint any
performance problem in MySQL?
Instrumentation in MySQL - PERFORMANCE_SCHEMA
○ History and engineers involved
○ Features in MySQL 5.6
○ How to use it?
○ Any alternatives in MySQL world?
○ Sources of additional information and tools
The Good, The Bad and The Ugly in PERFORMANCE_SCHEMA
www.percona.com

5. Method R - common sense approach
1. Identify the most important task (the bottleneck)
2. Measure its response time (R) in detail
3. Optimize that response time in the most (economically)
efficient way
4. Repeat until your system is (economically) optimal
Percona also tries to apply common sense:
http://www.mysqlperformanceblog.com/2008/11/24/how-percona-does-a-mysql-performance-audit/
http://www.mysqlperformanceblog.com/2011/07/27/whats-required-to-tune-mysql/
http://www.mysqlperformanceblog.com/2011/05/05/the-two-even-more-fundamental-performance-metrics/
www.percona.com

6. Instrumentation
“Code instrumentation, what is it? To me, it is the fine art of
making approximately every other line of your developed
code “debug” of some sort. Trace information. With
timestamps. And meaningful information.” - Tom Kyte
“In science, the term observer effect refers to changes that
the act of observation will make on a phenomenon being
observed. This is often the result of instruments that, by
necessity, alter the state of what they measure in some
manner.” - Wikipedia
www.percona.com

7. Trace files and debug printouts?
●
●
●
●
Readily available in MySQL for years...
○ http://dev.mysql.com/doc/refman/5.6/en/making-trace-files.html
○ http://dev.mysql.com/doc/refman/5.6/en/dbug-package.html
○ … --debug=d:t:i:o,/tmp/mysqld.trace
○ -debug binaries only
Additional tracing features are available in MySQL 5.6 for non-debug
binaries also (for optimizer decision making, for example)
○ http://dev.mysql.com/doc/internals/en/tracing-example.html
Still far behind Oracle’s SQL Trace and trace events in general
○ http://www.juliandyke.com/Diagnostics/Events/EventReference.html
Let’s try to produce trace while test case for Bug #68079 is running:
mysql> set session debug='d:t:i:o,/tmp/mysqld.trace';
Query OK, 0 rows affected (0.00 sec)
www.percona.com

8. Content of the trace file...
...
T@9
: | | | | THD::enter_stage: /home/openxs/bzr2/percona-5.6/Percona-Server-5.6.10-alpha60.
2/sql/sql_parse.cc:5383
T@9
: | | | | >PROFILING::status_change
T@9
: | | | | <PROFILING::status_change 374
T@9
: | | | | >close_thread_tables
T@9
: | | | | <close_thread_tables 1469
T@9
: | | | | >stmt_causes_implicit_commit
T@9
: | | | | <stmt_causes_implicit_commit 279
T@9
: | | | | >MDL_context::release_transactional_locks
T@9
: | | | | | >MDL_context::release_locks_stored_before
T@9
: | | | | | <MDL_context::release_locks_stored_before 2753
T@9
: | | | | | >MDL_context::release_locks_stored_before
T@9
: | | | | | <MDL_context::release_locks_stored_before 2753
T@9
: | | | | <MDL_context::release_transactional_locks 2908
T@9
: | | | <mysql_execute_command 5431
T@9
: | | | >~opt_trace_start
T@9
: | | | <~opt_trace_start 230
T@9
: | | | THD::enter_stage: /home/openxs/bzr2/percona-5.6/Percona-Server-5.6.10-alpha60.
2/sql/sql_parse.cc:6676
T@9
: | | | >PROFILING::status_change
T@9
: | | | <PROFILING::status_change 374
...
●
●
●
Debug printouts/traces/assertions/error log, is this enough?
Think about concurrency!
Too intrusive, changes behavior, serializes and wastes resources...
www.percona.com

9. Example of the problem to solve: Bug #68079
http://bugs.mysql.com/bug.php?id=68079
CREATE TABLE `incident` (
`sys_id` char(32) NOT NULL DEFAULT '',
`category` varchar(40) DEFAULT NULL,
PRIMARY KEY (`sys_id`),
KEY `incident_category` (`category`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
insert into incident(sys_id) values (rand()*1000000);
insert into incident(sys_id) select rand()*1000000 from incident; -- 13 times
CREATE TABLE `task` (
`sys_id` char(32) NOT NULL DEFAULT '',
`u_root_cause` char(32) DEFAULT NULL,
`u_business_impact_description` mediumtext,
`u_business_impact_category` mediumtext,
PRIMARY KEY (`sys_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
insert
insert
insert
update
into task(sys_id) values(rand()*1000000);
into task(sys_id) select rand()*1000000 from task; -- 13 times
into task(sys_id) select sys_id from incident limit 100;
incident set category=rand()*100000;
www.percona.com

10. Example of the problem to solve: Bug #68079
These queries scale well:
select count(*), category from incident group by category;
select * from task where sys_id=rand()*1000000;
This one does not scale even to 10 threads on 12 cores:
select count(*), category from task inner join incident on task.sys_id=incident.sys_id
group by incident.category;
mysql> explain select count(*), category from task inner join incident on task.
sys_id=incident.sys_id group by incident.category;
| 1 | SIMPLE
|
NULL
| 1 | SIMPLE
|
test.incident.sys_id
incident | index | PRIMARY
| 8210 | Using index |
task
| eq_ref | PRIMARY
|
1 | Using index |
| incident_category | 123
|
| PRIMARY
|
| 96
www.percona.com

11. Example of the problem to solve: Bug #68079
mysqlslap -uroot --iterations=10 --concurrency=N --create-schema=test --no-drop --numberof-queries=1000 --query='select count(*), category from task inner join incident on task.
sys_id=incident.sys_id group by incident.category'
where N in (1, 2, 4, 8, 10, 12, 24 …
2*number_of_cores … 1024 … 8192)
Results:
Benchmark
Average number of
Minimum number of
Maximum number of
Number of clients
Average number of
…
Benchmark
Average number of
Minimum number of
Maximum number of
Number of clients
Average number of
seconds
seconds
seconds
running
queries
to run all queries: 8.512 seconds
to run all queries: 8.248 seconds
to run all queries: 8.844 seconds
queries: 8
per client: 125
seconds
seconds
seconds
running
queries
to run all queries: 9.399 seconds
to run all queries: 9.137 seconds
to run all queries: 9.749 seconds
queries: 10
per client: 100
www.percona.com

12. Example of the problem to solve: Bug #68079
Simple (?) join does NOT scale even to 10 threads...
● Why does this happen?
● Where time is spent?
● Is there any way to tune MySQL server to improve
scalability for this query?
● Maybe we need different hardware?
● Any changes at schema level that may help?
● How to get information to answer the questions above?
● This is a performance tuning task to check tools,
approaches and code improvement ideas!
www.percona.com

13. Sources of information on performance
What sources of information and tools we can try to use?
●
●
●
●
●
●
●
●
●
●
●
●
show engine innodb statusG
show engine innodb mutex;
InnoDB-related tables in the INFORMATION_SCHEMA (?)
show global status like ‘innodb%’ (?)
userstat (Percona Server and others) (?)
show profiles
Poor man’s profiler (pt-pmp)
perf
oprofile
DTrace (?)
Maybe… PERFORMANCE_SCHEMA?
Discussion of the need for PERFORMANCE_SCHEMA and alternative
approaches - hence all the above
www.percona.com

14. SHOW ENGINE INNODB STATUS
●
Do you know how to read that?
○
○
○
●
http://dev.mysql.com/doc/refman/5.6/en/innodb-monitors.html - fine manual
http://www.mysqlperformanceblog.com/2006/07/17/show-innodb-status-walk-through/ - the
best thing ever since Worcestershire sauce
http://www.percona.com/doc/percona-server/5.6/diagnostics/innodb_show_status.html - just
better in Percona Server
Not very useful for Bug #68079 on my QuadCore:
Per second averages calculated from the last 36 seconds
----------------BACKGROUND THREAD
----------------srv_master_thread loops: 121 srv_active, 0 srv_shutdown, 2473 srv_idle
srv_master_thread log flush and writes: 2594
---------SEMAPHORES
---------OS WAIT ARRAY INFO: reservation count 9477
OS WAIT ARRAY INFO: signal count 3703
Mutex spin waits 28946906, rounds 25108917, OS waits 8475
RW-shared spins 209, rounds 5999, OS waits 182
RW-excl spins 0, rounds 9, OS waits 0
Spin rounds per wait: 0.87 mutex, 28.70 RW-shared, 9.00 RW-excl
www.percona.com

15. SHOW ENGINE INNODB STATUS
...
Purge done for trx's n:o < 2384 undo n:o < 0 state: running but idle
History list length 20
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 143558, not started
MySQL thread id 1, OS thread handle 0x7f01cc057700, query id 10678 localhost msandbox init
show engine innodb status
...
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
285 OS file reads, 49 OS file writes, 49 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.06 writes/s, 0.06 fsyncs/s
Hash table size 276671, node heap has 0 buffer(s)
0.00 hash searches/s, 142759.09 non-hash searches/s
...
Log sequence number 4732500
...
Last checkpoint at 4732480
0 pending log writes, 0 pending chkp writes
50 log i/o's done, 0.06 log i/o's/second
...
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
...
Main thread process no. 5202, id 139645286401792, state: sleeping
Number of rows inserted 0, updated 0, deleted 0, read 89167561
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 144695.09 reads/s
www.percona.com

16. SHOW ENGINE INNODB MUTEX
●
●
http://dev.mysql.com/doc/refman/5.6/en/show-engine.html
-debug builds give more information
○ count - how many times the mutex was requested
○ spin_waits - how many times the spinlock had to run
○ spin_rounds - the number of spinlock rounds
○ os_waits - indicates the number of operating system waits. The mutex
was not unlocked during the spinlock and it was necessary to yield to
the operating system and wait. You have only this value in nondebug binaries.
○ os_wait_times - indicates the amount of time (in ms) spent in operating
system waits, if the timed_mutexes system variable is 1 (ON). If
timed_mutexes is 0 (OFF, default), timing is disabled, so os_wait_times
is 0.
| InnoDB | rw_lock_mutexes
| count=0, spin_waits=0, spin_rounds=0,
os_waits=0, os_yields=0, os_wait_times=0 |
www.percona.com

17. SHOW ENGINE INNODB MUTEX
●
Looks like we have something immediately useful here for Bug #68079:
mysql [localhost] {msandbox} (test) > show engine innodb mutex;
+--------+-------------------------+----------------+
| Type
| Name
| Status
|
+--------+-------------------------+----------------+
| InnoDB | log0log.cc:737
| os_waits=2
|
| InnoDB | buf0buf.cc:1242
| os_waits=2
|
| InnoDB | combined buf0buf.cc:992 | os_waits=66475 |
| InnoDB | dict0dict.cc:896
| os_waits=1
|
| InnoDB | log0log.cc:799
| os_waits=16 |
| InnoDB | combined buf0buf.cc:993 | os_waits=1
|
+--------+-------------------------+----------------+
6 rows in set (0.02 sec)
vi +992 /home/openxs/bzr2/mysql-5.6/storage/innobase/buf/buf0buf.cc
mutex_create(PFS_NOT_INSTRUMENTED, &block->mutex, SYNC_BUF_BLOCK);
●
●
Hardly is maintained properly in MySQL 5.6. Check my Bug #70819
Percona Server shows mutex name, not file and line number…
| InnoDB | &buf_pool->LRU_list_mutex
| InnoDB | &fil_system->mutex
| InnoDB | combined &block->mutex
| os_waits=13
| os_waits=5073
| os_waits=162396
www.percona.com

18. INFORMATION_SCHEMA?
InnoDB-related tables in the INFORMATION_SCHEMA:
●
●
●
http://dev.mysql.com/doc/refman/5.6/en/innodb-i_s-tables.html - there are
many...
But as we know from INNODB STATUS it’s not about I/O or locks, it seems
they do not add much useful for the case of Bug #68079…
If only INNODB_METRICS - http://dev.mysql.com/doc/refman/5.6/en/innodbmetrics-table.html:
| NAME
| SUBSYSTEM
| COUNT
| MAX_COUNT
| MIN_COUNT
| AVG_COUNT
...
| STATUS
| TYPE
| COMMENT
|
|
|
|
|
|
varchar(193)
varchar(193)
bigint(21)
bigint(21)
bigint(21)
double
|
|
|
|
|
|
NO
NO
NO
YES
YES
YES
| varchar(193) | NO
| varchar(193) | NO
| varchar(193) | NO
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
0
NULL
NULL
NULL
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
www.percona.com

19. INNODB_METRICS...
mysql> select name, count from information_schema.innodb_metrics where
status='enabled' and type='status_counter';
+---------------------------------+-----------+
| name
| count
|
+---------------------------------+-----------+
...
| buffer_pool_reads
|
271 |
| buffer_pool_read_requests
| 336246061 |
...
| buffer_pages_read
|
270 |
| buffer_data_reads
|
6623232 |
| buffer_data_written
|
141312 |
| os_data_reads
|
285 |
| os_data_writes
|
89 |
| os_data_fsyncs
|
89 |
| os_log_bytes_written
|
40960 |
| os_log_fsyncs
|
83 |
...
| innodb_rwlock_s_spin_waits
|
210 |
| innodb_rwlock_x_spin_waits
|
0 |
| innodb_rwlock_s_spin_rounds
|
6029 |
| innodb_rwlock_x_spin_rounds
|
9 |
| innodb_rwlock_s_os_waits
|
183 |
| innodb_rwlock_x_os_waits
|
0 |
...
www.percona.com

20. SHOW GLOBAL STATUS LIKE ‘innodb%’
●
●
pt-mext helps - http://www.percona.com/doc/percona-toolkit/2.1/pt-mext.html
Let’s monitor the counters (51 starts with Innodb_) over 10 seconds:
[openxs@chief msb_5_6_14]$ pt-mext -r -- /home/openxs/5.6.14/bin/mysqladmin -socket=/tmp/mysql_sandbox5614.sock -uroot ext -i10 -c3 >/tmp/mext.txt
Innodb_buffer_pool_read_requests
...
Innodb_buffer_pool_write_requests
Innodb_data_fsyncs
...
Innodb_data_read
Innodb_data_reads
Innodb_data_writes
...
Innodb_data_written
...
Innodb_log_writes
Innodb_num_open_files
Innodb_os_log_fsyncs
...
Innodb_rows_read
14726570
686556718
160
176
3
3
6623232
285
176
0
0
3
250368
1536
160
7
166
342861007
3
0
3
7353994
www.percona.com

21. Percona Server and userstat?
http://www.percona.com/doc/percona-server/5.6/diagnostics/user_stats.html:
●
●
●
●
●
●
Introduced in Google patch
Additional tables in INFORMATION_SCHEMA:
○ CLIENT_STATISTICS
○ THREAD_STATISTICS
○ USER_STATISTICS
○ TABLE_STATISTICS
○ INDEX_STATISTICS
Easy to configure:
○ http://dom.as/2013/04/17/on-performance-schemas/
Low performance impact we say:
○ http://www.mysqlperformanceblog.com/2012/06/02/how-expensive-is-user_statistics/
Hardly this can help with mutexes or waits…
But some says it adds overhead:
○
“and then, once userstat=1 the regression is horrible.. over 30%... “ - guess who says that and why?
www.percona.com

22. SHOW PROFILE
●
●
●
http://bugs.mysql.com/bug.php?id=24795 - SHOW PROFILE contribution by
Jeremy Cole (MySQL 5.0)
http://dev.mysql.com/doc/refman/5.6/en/show-profile.html - details…
Let’s try to profile the query while test for Bug #68079 is running:
mysql [localhost] {msandbox} (test) > set profiling=1;
Query OK, 0 rows affected, 1 warning (0.02 sec)
mysql [localhost] {msandbox} (test) > show warningsG
*************************** 1. row ***************************
Level: Warning
Code: 1287
Message: '@@profiling' is deprecated and will be removed in a future release.
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (test) > select count(*), category from task inner join
incident on task.sys_id=incident.sys_id group by incident.category;
...
103 rows in set (0.08 sec)
mysql [localhost] {msandbox} (test) > show profiles;
mysql [localhost] {msandbox} (test) > show profile for query 3;
www.percona.com

23. SHOW PROFILE
●
Profile of the query while test for Bug #68079 is running:
| starting
| 0.000063 |
| checking permissions | 0.000004 |
| checking permissions | 0.000004 |
| Opening tables
| 0.000092 |
| init
| 0.000023 |
| System lock
| 0.000007 |
| optimizing
| 0.000009 |
| statistics
| 0.000037 |
| preparing
| 0.000015 |
| Creating tmp table
| 0.000023 |
| Sorting result
| 0.000003 |
| executing
| 0.000002 |
| Sending data
| 0.081722 |
| Creating sort index | 0.000141 |
| end
| 0.000005 |
| removing tmp table
| 0.000008 |
| end
| 0.000004 |
| query end
| 0.000008 |
| closing tables
| 0.000013 |
| freeing items
| 0.000553 |
| cleaning up
| 0.000017 |
+----------------------+----------+
21 rows in set, 1 warning (0.00 sec)
www.percona.com

24. Poor man’s profiler (gdb tricks)
●
●
●
●
http://poormansprofiler.org/
http://www.percona.com/doc/percona-toolkit/2.1/pt-pmp.html
http://www.mysqlperformanceblog.com/2011/12/02/three-ways-that-thepoor-mans-profiler-can-hurt-mysql/
○ mysqld freezes for the duration of the process
○ They say mysqld can crash (?)
○ They say mysqld can be left in an unstable state (?)
I’ve tried recently on QuadCore with test from bug #68079, not impressed:
10 libaio::??,os_aio_linux_collect,os_aio_linux_handle,fil_aio_wait,io_handler_thread,start_thread,clone
2 __memcpy_ssse3,store_key_field::copy_inner,copy,cp_buffer_from_ref,cmp_buffer_with_ref,join_read_key,
sub_select,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_execute_select,mysql_select,
handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,
do_handle_one_connection,handle_one_connection,
pfs_spawn_thread,start_thread,clone
1 update_ref_and_keys,make_join_statistics,JOIN::optimize,mysql_execute_select,mysql_select,
handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,
do_handle_one_connection,handle_one_connection,
pfs_spawn_thread,start_thread,clone
www.percona.com

25. Poor man’s profiler - Domas rulez!
●
●
●
●
So, I was not impressed (found nothing enlightening), but it depends:
○ “Usually pmp (poor man's profiler, making stacktrace snapshots)
producing more relevant information than performance schema reports.”
Google for something like site:bugs.mysql.com Mark Callaghan pmp:
○ http://bugs.mysql.com/bug.php?id=58037 - fixed in 5.5+ with MDL
○ http://bugs.mysql.com/bug.php?id=57021 - it was used to find a problem
in Facebook’s patch…
○ http://bugs.mysql.com/bug.php?id=61545 - still “Verified”, commit stalls
when mysql_binlog_send does disk read
If you are on Linux and don’t run MySQL with real profiler, try use pt-pmp
when server stalls, hangs or seems overloaded and unusable
Key to success for all kinds of profilers - open source of MySQL server!
www.percona.com

26. Let’s try real profiler - perf
●
●
●
https://perf.wiki.kernel.org/index.php/Main_Page:
○ “Among others, it provides per task, per CPU and per-workload
counters, sampling on top of these and source code event annotation”
http://bugs.mysql.com/bug.php?id=69236 - Single-threaded workloads can
be much slower on 5.6 than on 4.0. perf was used to prove that!
Sample session while test for Bug #68079 was running:
[openxs@chief ~]$ perf record -p 3310 sleep 20
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 2.006 MB perf.data (~87654 samples) ]
[openxs@chief ~]$ perf report
...
12.76% mysqld mysqld
12.32% mysqld mysqld
10.39% mysqld mysqld
[.] btr_cur_search_to_nth_level(d
[.] buf_page_get_gen(unsigned lon
[.] my_strnncollsp_utf8
www.percona.com

27. perf output continued...
●
Sample session while test for Bug #68079 was running:
...
7.28%
6.68%
5.76%
4.56%
4.22%
2.87%
2.79%
2.68%
2.33%
1.97%
1.86%
1.83%
1.81%
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
rec_get_offsets_func(unsigned
mtr_memo_slot_release_func(mt
page_cur_search_with_match(bu
cmp_dtuple_rec_with_match_low
row_search_for_mysql(unsigned
ut_delay(unsigned long)
pfs_mutex_enter_func(ib_mutex
innobase_mysql_cmp(int, unsig
cmp_whole_field(unsigned long
row_mysql_store_col_in_innoba
get_internal_charset
mutex_spin_wait(ib_mutex_t*,
pfs_mutex_exit_func(ib_mutex_
...
●
●
This was on QuadCore with 4 concurrent threads, MySQL 5.6.14
Now tell me that PERFORMANCE_SCHEMA impact with default settings is
low on MySQL 5.6...
www.percona.com

28. Let’s try another profiler - oprofile
●
●
http://www.fromdual.ch/mysql-oprofile:
○ Nice summary from Oli on how to use it!
Sample session while test for Bug #68079 was running:
[root@chief openxs]# opreport --demangle=smart --symbols --long-filenames --merge tgid
/home/openxs/5.6.14/bin/mysqld | head -n 20
...
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00
(Unhalted core cycles) count 100000
samples %
image name
symbol name
1208437 12.5180 /home/openxs/5.6.14/bin/mysqld btr_cur_search_to_nth_level
(dict_index_t*, unsigned long, dtuple_t const*, unsigned long, unsigned long, btr_cur_t*,
unsigned long, char const*, unsigned long, mtr_t*)
1151258 11.9257 /home/openxs/5.6.14/bin/mysqld buf_page_get_gen(unsigned long, unsigned
long, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned
long, mtr_t*)
1115969 11.5601 /home/openxs/5.6.14/bin/mysqld my_strnncollsp_utf8
...
242608
2.5131 /home/openxs/5.6.14/bin/mysqld pfs_mutex_enter_func(ib_mutex_t*, char
const*, unsigned long)
...
142685
1.4780 /lib64/libc-2.13.so
__memcpy_ssse3
132334
1.3708 /home/openxs/5.6.14/bin/mysqld pfs_mutex_exit_func(ib_mutex_t*)
...
www.percona.com

29. DTrace? One day, maybe...
●
I wish I remember how to use it and have MacOS X or Solaris, or FreeBSD
9.x (on some good hardware) at hand…
●
but I don’t. So, I had not tried to use DTrace to find out what’s wrong with
that Bug #68079.
●
Some useful references, still, if you ever plan to try:
○
○
http://dom.as/tag/dtrace/ - check what Domas used to do with DTrace
○
http://cdn.oreillystatic.com/en/assets/1/event/21/Deep-inspecting%20MySQL%20with%
20DTrace%20Presentation.pdf - from one of User Conferences...
○
●
http://dev.mysql.com/doc/refman/5.6/en/dba-dtrace-server.html - it’s documented in the
manual, including static probes added
http://dev.mysql.com/tech-resources/articles/getting_started_dtrace_saha.html - “native” Sun
engineers also contributed
I really miss Sun...
www.percona.com

30. Time to try PERFORMANCE_SCHEMA
●
●
●
●
We know functions where most of the time was spent (profiling)
We know that some mutex waits were long enough to force OS wait (switch
to other thread)
We need more details, specifically time spent waiting… So, time to use
PERFORMANCE_SCHEMA it seems
Sample output while test for Bug #68079 was running, naive attempt (the
time is in picoseconds):
mysql> select event_name, count_star, sum_timer_wait from performance_schema.
events_waits_summary_global_by_event_name where event_name like
'wait/synch/mutex/innodb%' and count_star > 0 order by sum_timer_wait desc limit 5;
+--------------------------------------------+------------+----------------+
| event_name
| count_star | sum_timer_wait |
+--------------------------------------------+------------+----------------+
| wait/synch/mutex/innodb/os_mutex
|
71754726 | 11380485535846 |
| wait/synch/mutex/innodb/trx_sys_mutex
|
60057 |
4191727883 |
| wait/synch/mutex/innodb/buf_pool_mutex
|
8655 |
468303388 |
| wait/synch/mutex/innodb/mutex_list_mutex
|
10310 |
427385546 |
| wait/synch/mutex/innodb/rw_lock_list_mutex |
8304 |
316946749 |
+--------------------------------------------+------------+----------------+
5 rows in set (0.00 sec)
www.percona.com

31. Basic ideas and terms
PERFORMANCE_SCHEMA (aka PS, P_S, PFS):
● Inspired by Oracle Wait Interface
● Separate database and storage engine
● Records run time statistics via built-in instrumentation points
● Designed to let us get the answers to the following questions:
○ Where was time spent?
○ Who has used most of resources/time?
○ What was executed to to use these?
○ When it was executed?
● Appeared in MySQL 5.5, improved and ON by default in MySQL 5.6, and is
documented and explained in details already:
○ http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html
● Implemented by Marc Alff (http://marcalff.blogspot.com/)
● Deal with instruments (including wait events), consumers, objects (tables for
now) that are monitored, threads (some of them background) that run and
actors (users) that execute SQL statements (in stages).
www.percona.com

32. So, how to use it?
●
●
The amount of information in the manual is huge, it’s not easy to start using P_S
○ I wish it would be as easy as setting TIMED_STATISTICS to TRUE with ALTER
SYSTEM + few queries to V$SESSTATS, V$SYSSTATS or V$SESSION_WAIT
Marc Alff explains in http://marcalff.blogspot.com/2013/04/on-configuringperformance-schema.html:
○ “Define your goals” - this is clear, we have waits and we need to find out what
waits are most important based on time wasted (or waited)
○ “Define what to instrument” - we need to instrument InnoDB waits mutex waits
and internal lock waits, but how to do this? Details next, but:
■ "wait/synch/mutex/%", for mutexes
■ "wait/synch/rwlock/%", for read write locks
○ “Define how much detail to collect” - well, we need counts and time spent for
all InnoDB-related waits
○ “Provide sizing data” - autosizing based on max_connections?
| performance_schema_max_mutex_instances
| performance_schema_max_rwlock_instances
○
| 15600 |
| 9000 |
“Monitor sizing problems” - memory used vs dropped statistics and impact
www.percona.com

33. How instruments are named?
●
●
For those who want solutions, not complains or comments...
http://dev.mysql.com/doc/refman/5.6/en/performance-schema-instrumentnaming.html:
○ "wait/io/file/%" - file io
○ "wait/io/socket/%" - network io
○ "wait/io/table/%" - table io
○ "wait/lock/table/%" - table locks
○ "wait/synch/cond/%" - conditions
○ "wait/synch/mutex/%" - mutexes
○ "wait/synch/rwlock/%" - read write locks
○ "stage/%" - stages of statement execution (see show profile)
○ "idle" - clients idle time.
○ "statement/com/%" - commands in the client/server protocol
○ "statement/sql/%" - SQL queries (and some more statement/…)
Hint: bold means “new in MySQL 5.6”, blue color means enabled by default
www.percona.com

34. Setup for Bug #68079 (waits/…)
●
●
http://mysqlentomologist.blogspot.com/2013/01/how-to-use-performanceschemato-check.html - my experience back at MySQL 5.6 RC stage...
To set up InnoDB mutex waits monitoring via PERFORMANCE_SCHEMA in
general with MySQL 5.6.x you should do at least the following:
1. Start MySQL server with all mutex related instruments enabled at startup
(performance_schema=ON by default on recent 5.6.x), like this:
mysqld_safe --performance_schema_instrument='wait/synch/mutex/innodb/%=on' &
2.
Connect to server and set up proper consumers explicitly, like this:
UPDATE performance_schema.setup_consumers SET enabled = 'YES'
WHERE name like 'events_waits%';
3.
4.
Run your problematic load.
Check waits using whatever tables you need
www.percona.com

35. Summary of my setup experience...
●
●
●
●
It's great to have Oracle engineers reading your Facebook posts and
replying (especially in a real time). Thank you, Mark Leith!
PERFORMANCE_SCHEMA in MySQL 5.6 can be used to study InnoDB
mutex waits
MySQL 5.6 manual is not clear enough about steps and details on
instruments involved to do the study explained above
http://dev.mysql.com/doc/refman/5.6/en/performance-schema-filtering.html:
“Modifications to any of these tables affect monitoring immediately, with the exception of
setup_actors. Modifications to setup_actors affect only foreground threads created
thereafter.”
●
●
Now compare to reality, complain in public, report a bug:
○ http://bugs.mysql.com/bug.php?id=68097 - “Manual does not explain
that some P_S instruments must be enabled at startup”
Get a reply from Dimitri Kravtchuk:
“All instrumentation is dynamic EXCEPT mutexes which are created mainly on the MySQL server
startup, so to get their instrumentation initialized you have to enable it on startup”
www.percona.com

36. Not that clear and easy to use?
www.percona.com

37. ps_helper To the Rescue!
ps_helper:
●
●
●
●
●
●
●
Developed by Mark Leith, MySQL expert @ Oracle
○ worked in MySQL Support team since 2005
○ senior software development manager @Oracle now (MEM, maybe MEB)
○ good Oracle RDBMS background
Easier to use comparing to “raw” PERFORMANCE_SCHEMA
Ready to use collection of views, stored routines and...
Recently even an easy way to create ps, P_S, PFS or whatever synonym you like!
git clone https://github.com/MarkLeith/dbahelper.git dbahelper - if you prefer...
Installed in a separate schema (ps_helper)
IO/latency/waits/statement digests
Links:
● http://www.markleith.co.uk/ps_helper/
● https://github.com/MarkLeith/dbahelper/archive/master.zip
● https://www.slideshare.net/Leithal/performance-schema-andpshelper
www.percona.com

38. ps_helper: basics
Install:
openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot test < ps_helper_56.sql
Check:
mysql> show tables;
+-------------------------------------------------+
| Tables_in_ps_helper
|
+-------------------------------------------------+
| _digest_95th_percentile_by_avg_us
|
| _digest_avg_latency_by_avg_us
|
| check_lost_instrumentation
|
...
53 rows in set (0,00 sec)
mysql> select routine_name from information_schema.routines where
routine_schema='ps_helper';
+-------------------------------+
| routine_name
|
+-------------------------------+
| analyze_statement_digest
|
...
20 rows in set (0,00 sec)
www.percona.com

39. ps_helper: why is it useful?
Have some real fun:
openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot ps_helper <
procedures/create_synonym_db.sql
Warning: Using a password on the command line interface can be insecure.
mysql> call create_synonym_db('performance_schema', 'ps');
Query OK, 0 rows affected (2,35 sec)
mysql> use ps
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+----------------------------------------------------+
| Tables_in_ps
|
+----------------------------------------------------+
| accounts
|
...
Use ps_helper.format_time(SUM(sum_timer_wait)):
● Check Bug #70252 to find out what may happen if you don’t use it
● Check https://www.slideshare.net/Leithal/performance-schema-andpshelper
www.percona.com

40. Some history behind...
● “The Good, The Bad and The Ugly”:
“The film's title has entered the English language as an idiomatic expression. Typically used when describing
something thoroughly, the respective phrases refer to upsides, downsides and the parts that could, or should have
been done better, but were not.” - Wikipedia
● “For Three Men The Civil War Wasn’t Hell. It Was Practice!”
○
○
Guess who are these men, in case of PERFORMANCE_SCHEMA…
Maybe more than three…
● Read the WorkLogs:
○
○
○
http://dev.mysql.com/worklog/task/?id=2360 - “Why we're doing this”
http://dev.mysql.com/worklog/task/?id=4674 - “Setup for Actors”
http://dev.mysql.com/worklog/task/?id=4513 - “Implement it…”
● Where the name comes from?
“Proposals for the schema name were DYNAMIC (but that is a reserved word in standard SQL),
PERFORMANCE_SCHEMA (Jim Winstead's suggestion), PERFORMANCE (Peter's preference), and DBTrace
because it sounds like DTrace. There was a vote (see dev-private thread "WL#2360 Vote: schema name and
table name"). The winner was PERFORMANCE_SCHEMA.” - Peter Gulutzan
www.percona.com

41. PERFORMANCE_SCHEMA in MySQL 5.6
Useful reading (and names to remember):
●
http://marcalff.blogspot.com/2013/04/on-configuring-performance-schema.html - article from Mark
Alff on how to configure PERFORMANCE_SCHEMA
●
http://dev.mysql.com/doc/refman/5.6/en/performance-schema-configuration.html - RTFM
●
https://www.facebook.com/notes/mysql-at-facebook/my-mysql-is-faster-than-yourmysql/10151250402570933 - influence of P_S in 5.6.10 and... 5.1.63 or 5.1.52+FB patch are still
fastest in many cases
●
http://dimitrik.free.fr/blog/archives/2013/07/mysql-performance-why-performance-schemaoverhead.html - overhead explained and measured by Dimitri Kravtchuk:
●
http://www.percona.com/webinars/using-mysql-56-performance-schema-troubleshoot-typicalworkload-bottlenecks - webinar by Peter Zaitsev (May 15, 2013)
●
http://mysql.wisborg.dk/category/mysql/performance_schema/ - useful posts from Jesper Krogh
on the topic. He had created yet another set of tools to make out life easier, ps_tools: http://mysql.
wisborg.dk/2013/10/10/mysql-connect-2013-ps_tools/
●
http://mysqlblog.fivefarmers.com/tag/performance_schema/ - practical use of
PERFORMANCE_SCHEMA, by Todd Farmer
www.percona.com

42. PERFORMANCE_SCHEMA - The Good
●
●
●
●
●
●
●
●
Monitoring is available on all platforms supported by MySQL
Activating the Performance Schema causes no (few) changes in server behavior
○ No persistent on-disk storage
○ Memory allocation at server startup
○ Processing happens mostly at events retrieval
○ No new keywords or threads added
Overhead is small comparing to some other ways to monitor
Problems/failures in P_S code should not prevent server from working normally
It is easy to add new instrumentation points (they say)
Under active development:
○ Very few bugs - 4 active in MySQL 5.6 (of 47) as of November 1, 2013
○ Bug #70018 (PFS Overhead on frequent connect/disconnect) fixed in 5.6.15!
○ Even more features in MySQL 5.7: check Bug #69527 (MDL locks!)
Used by Oracle internally a lot for real things (MEM, performance improvements)
As it’s based on proper ideas, eventually it’s going to become an ultimate solution
for MySQL performance monitoring (and replace everything else)
www.percona.com

43. Let me produce some flame...
www.percona.com

44. PERFORMANCE_SCHEMA - The Bad
● Performance overhead in default configuration is notable in
some cases (10-30%, see bug reports and 3rd party studies)
● Additional memory use is notable (see Bug #68514):
○
“Why is this not a bug? On my x86-64 with 5.6.12 when
performance_schema=1 then the extra RAM is:
max_connections * 196kb” - Mark Callaghan
● Mutex and rw-lock waits instrumentation must be enabled at
server startup
● How complete is the instrumentation?
● There are bugs and missing parts in the manual
● There are bugs in implementation
● Is this kind of instrumentation really good? (SQL, no trace)
● Any critical posts and even some bug reports are sometimes
considered as a kind of a personal insult: "godfather" case...
www.percona.com

45. Bugs in P_S and fine manual
Bug #68413 - Facebook: “performance_schema overhead is at least 10%”
Bug #69782 - Old files not being removed from performance_schema.file_instances
Bug #70628 - Wrong instrumentation interface for mysql_cond_timedwait
Bug #70697 - Probably not a bug, but indication that manual for SHOW ENGINE
PERFORMANCE_SCHEMA STATUS is still not clear
Bug #68574 - “No instrumentation for InnoDB files in P_S on windows” - same with
5.6.14, by the way...
Bug #68785 - new features (connection attributes) work well together (or not)...
Bug #66589 - still not fixed (do not use mysqlhotcopy). One of the last bugs I verified in
Oracle...
Bug #68514 - beware of memory usage impact with P_S
Bug #68097 - still some details to document in the manual clearly
www.percona.com

46. PERFORMANCE_SCHEMA - The Ugly
●
●
●
●
●
●
Names: schema name, long table names… you know whom to blame (and thank
for ps_helper). I wish we have public synonyms in MySQL, so I’d create
V$SYSSTATS (Bug #68855). You can create views, but that’s overhead, no?
Why additional views and procedures (ps_helper) are not there by default?
Some important things and storage engines are not properly instrumented yet,
MDL locks, for example… ups, seems fixed in 5.7.3
Heuristics to control size (read Bug #68514 again)… less than obvious and I’d
prefer more control
Make up your mind on what should be in P_S vs I_S vs SHOW statements. Think
about SHOW PROCESSLIST vs I_S.PROCESSLIST vs P_S.THREADS
Want to know what else users care about? Check feature requests for P_S (11
“Verified” as of November 2, 2013), some of them are limitations “by design”:
○
○
○
Bug #69881 - “Try to compute elapsed time for an event which has not completed” - Todd Farmer
Bug #66515 - “I have a problem with PS instrumentation that can only be enabled at server start.” Mark Callaghan
Bug #70383 - “Implement a SELECT-syntax/statement as a synonym for SHOW ENGINE
performance_schema STATUS” - Peter Laursen
www.percona.com

47. Do we have a hope?
www.percona.com

48. Do you still care about Bug #68079?
“mysql-trunk (a.k.a 5.7.3): Scales up to 32 threads reasonably well, the HW has
48 logical CPUs” - Sunny Bains, October 18, 2013
“There is some confusion here, there are two issues in this bug report. One is the
page mutex and the other is the page re-lock. For this query we acquire it in S
mode all the time. If we remove the page mutex overhead then this S mode
rwlock becomes the next bottleneck. However, this doesn't mean that the original
problem is not fixed, it simply means that the fix requires further work that is more
involved, that's all.“ - Sunny Bains, October 30, 2013
“These same numbers can be achieved by using "PARTITION BY KEY()
PARTITIONS N". By using partitions you can reduce the contention on the block
mutex.” - Sunny Bains, October 30, 2013
●
●
Does Oracle care about MySQL?
Was PERFORMANCE_SCHEMA used to find the bottlenecks?
www.percona.com

49. Thank you!
Questions and Answers
Please, report bugs to:
http://bugs.mysql.com
Use “Affects Me” button!
www.percona.com