[INSIGHT OUT 2011] A17 mastering my sql performance tuning(ronald bradford)
1. Title
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
2. Title
Mastering MySQL
Performance Tuning
Ronald Bradford
http://ronaldbradford.com
Insight Out DB Showcase
October 2011
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
3. OBJECTIVE
Identify key MySQL instrumentation
Detail the MySQL terms to know and use
Provide references to further materials
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
4. ABOUT AUTHOR
Ronald BRADFORD
12 years with MySQL / 22 years with RDBMS
Senior Consultant at MySQL Inc (06-08)
Consultant for Oracle Corporation (96-99)
7 years presenting MySQL content
All time top MySQL blogger
Available NOW
Published author for consulting
Oracle ACE Director
http://RonaldBradford.com
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
5. MySQL
Instrumentation
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
6. Key sources
Configuration
my.cnf (e.g. /etc/my.cnf, /etc/mysql/my.cnf)
SHOW [SESSION|GLOBAL] VARIABLES
INFORMATION_SCHEMA tables
Status
SHOW [SESSION|GLOBAL] STATUS
INFORMATION_SCHEMA tables
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
7. STATUS
SHOW STATUS EXAMPLE
INFORMATION_SCHEMA.STATUS Example
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
8. Key Sources
Running Threads
SHOW FULL PROCESSLIST
INFORMATION_SCHEMA.PROCESSLIST
Storage Engines
SHOW ENGINE INNODB STATUS
Performance Schema
PERFORMANCE_SCHEMA
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
9. PROCESSLIST
PROCESSLIST EXAMPLE
INFORMATION_SCHEMA.PROCESSLIST Example
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
10. KEY areas
System performance
SQL performance
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
11. Know your system
bottleneck
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
12. BOTTLENECK
Memory
Disk
CPU
Network
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
13. MEMORY
Basic configuration (Global use)
Basic configuration (Thread use)
Memory Tables
Thread Temporary tables
http://dev.mysql.com/doc/refman/5.5/en/memory-use.html
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
14. Memory
Global Buffer configuration
innodb_buffer_pool_size
key_buffer_size
query_cache_size
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
15. InnoDB Size
EXAMPLE
innodb_buffer_pool_size
Was 1 GB, should be 300M+
AWS small instance with 1.7G RAM
mysql> source sql/innodb_size.sql
+-------------------+----------+---------+----------+
| title | total_mb | data_mb | index_mb |
+-------------------+----------+---------+----------+
| Total InnoDB Size | 278 | 171 | 107 |
+-------------------+----------+---------+----------+
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
16. InnoDB Size
EXAMPLE
innodb_buffer_pool_size
Was 1 GB, should be 300M+
AWS small instance with 1.7G RAM
mysql> source sql/innodb_size.sql
+-------------------+----------+---------+----------+
| title | total_mb | data_mb | index_mb |
+-------------------+----------+---------+----------+
| Total InnoDB Size | 278 | 171 | 107 |
+-------------------+----------+---------+----------+
Rule of 75-80% of RAM is WRONG!
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
17. MyISAM Size
EXAMPLE
key_buffer_size
For MyISAM Indexes ONLY
Was default in RDS, should be 700M+
Github
mysql> source sql/myisam_size.sql
+-------------------+----------+---------+----------+
| title | total_mb | data_mb | index_mb |
+-------------------+----------+---------+----------+
| Total MyISAM Size | 6140 | 5492 | 648 |
+-------------------+----------+---------+----------+
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
18. MyISAM Size
EXAMPLE
key_buffer_size
For MyISAM Indexes ONLY
Was default in RDS, should be 700M+
Github
mysql> source sql/myisam_size.sql
+-------------------+----------+---------+----------+
| title | total_mb | data_mb | index_mb |
+-------------------+----------+---------+----------+
| Total MyISAM Size | 6140 | 5492 | 648 |
+-------------------+----------+---------+----------+
Needs 1M min for all InnoDB system
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
19. memory
Thread buffer configuration
read_buffer_size
read_rnd_buffer_size
sort_buffer_size
join_buffer_size
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
20. memory
In Memory Tables
max_heap_table_size
Internal Temporary Tables
MIN(tmp_table_size,max_heap_table_size)
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
21. Memory
mysqld memory
Unbounded
Linux process list (ps)
$ $ ps -eopid,fname,vsz,rss,user,command | grep -e "RSS" -e "mysql"
PID COMMAND VSZ RSS USER COMMAND
13628 grep 3352 820 ubuntu grep -e RSS -e mysql
28082 mysqld 631076 405660 mysql /usr/sbin/mysqld
$ ps -eopid,fname,vsz,rss,user,command | grep " mysqld " | grep -v grep
| awk '{print $3,$4}'
631076 405660
http://ronaldbradford.com/blog/are-you-monitoring-rss-vsz-2009-03-08/
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
22. MEMORY
PROCESSLIST
Copying to tmp table
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
23. Disk
fsync configuration
flush_log_at_trx_commit
sync_binlog
innodb_file_per_table
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
24. MySQL DISK
Status Variables
mysql> show global status like 'innodb%write%';
+-----------------------------------+-----------+ Volumes
| Innodb_buffer_pool_write_requests | 821053284 |
| Innodb_data_pending_writes | 0 |
| Innodb_data_writes | 198400090 |
| Innodb_dblwr_writes | 547847 |
| Innodb_log_write_requests | 64860531 |
| Innodb_log_writes | 188609653 |
| Innodb_os_log_pending_writes | 0 |
+-----------------------------------+-----------+
mysql> show global status like '%sync%';
+------------------------------+---------+ True Costs
| Innodb_data_fsyncs | 6835646 |
| Innodb_data_pending_fsyncs | 0 |
| Innodb_os_log_fsyncs | 5681432 |
| Innodb_os_log_pending_fsyncs | 0 |
+------------------------------+---------+
mysql> show global status like 'created_tmp_disk_tables';
+-------------------------+-------+
| Created_tmp_disk_tables | 87 |
+-------------------------+-------+
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
25. CPU
Query sorting (ORDER BY/GROUP BY)
PROCESSLIST
Sorting result
EXPLAIN
Using Filesort
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
26. NETWORK
STATUS
Bytes_sent
Bytes_received
PROCESSLIST
Sending data
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
27. SQL
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
28. SQL performance
Throughput
Scalability is about increasing
throughout
Latency
Performance is about improving
latency
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
29. THROUgHPUT
Status
Performance Schema
TCP/IP
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
30. MySQL STATUS
EXAMPLE
$ status_now.sh
--------------------------+------------+
| Variable_name | Value |
+--------------------------+------------+
| Bytes_received | 142721 |
| Bytes_sent | 85170 | Temporary disk table
| Com_admin_commands | 160 | writes per second
| Com_delete | 4 |
| Com_insert | 12 |
| Com_select | 331 |
| Com_show_status | 1 |
| Com_update | 5 |
| Created_tmp_disk_tables | 52 |
| Created_tmp_tables | 53 |
...
| Qcache_inserts | 331 |
| Qcache_hits | 382 |
| Qcache_lowmem_prunes | 319 |
| Qcache_free_memory | -8600 |
...
| Uptime | 1 |
+--------------------------+------------+
https://github.com/ronaldbradford/EffectiveMySQL
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
31. MySQL STATUS
EXAMPLE
$ status_now.sh What is happening this second?
--------------------------+------------+
| Variable_name | Value |
+--------------------------+------------+
| Bytes_received | 142721 |
| Bytes_sent | 85170 | Temporary disk table
| Com_admin_commands | 160 | writes per second
| Com_delete | 4 |
| Com_insert | 12 |
| Com_select | 331 |
| Com_show_status | 1 |
| Com_update | 5 |
| Created_tmp_disk_tables | 52 |
| Created_tmp_tables | 53 |
...
| Qcache_inserts | 331 |
| Qcache_hits | 382 |
| Qcache_lowmem_prunes | 319 |
| Qcache_free_memory | -8600 |
...
| Uptime | 1 |
+--------------------------+------------+
https://github.com/ronaldbradford/EffectiveMySQL
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
32. TCP/IP
TCP/IP Example
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
33. Latency
EXPLAIN
SHOW CREATE TABLE
SHOW INDEXES
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
34. EXPLAin
EXPLAIN Example
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
35. LATENCY
Optimizer Trace (Since 5.6.3)
Shows all optimizer actions
Rows statistics
Tree pruning
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
36. OPTIMIZER TRACE
Optimizer Trace Example
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
37. latency
performance_schema
Introduced in 5.5
Much improved 5.6.3
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
38. Performance schema
Performance Schema example
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
39. throughput
Repeating queries
Redundant queries
Unnecessary complex queries
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
40. throughput
MySQL Query Cache
Great for high read
Poor for read/write
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
41. The art of doing more
starts with doing less
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
42. MYSQL STATUS
EXAMPLE
====================================================================================================
Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval
====================================================================================================
Variable Delta/Percentage Per Second Total
Why are you doing crazy
====================================================================================================
Statement Activity
things Mr Framework?
====================================================================================================
SELECT: 16,042 267.37 8,177,050 (46.03%)
INSERT: 5,838 97.30 1,826,616 (10.28%)
UPDATE: 1,109 18.48 738,546 (4.16%)
DELETE: 2,018 33.63 1,374,983 (7.74%)
REPLACE: 0 0.00 0 (0.00%)
INSERT ... SELECT: 0 0.00 27 (0.00%)
REPLACE ... SELECT: 0 0.00 0 (0.00%)
Multi UPDATE: 0 0.00 0 (0.00%)
Multi DELETE: 0 0.00 0 (0.00%)
COMMIT: 5,708 95.13 2,161,232 (12.17%)
ROLLBACK: 5,746 95.77 3,485,828 (19.62%)
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
43. Row Processing
EXAMPLE
SELECT name FROM firms WHERE id=727;
SELECT name FROM firms WHERE id=758;
SELECT name FROM firms WHERE id=857;
SELECT name FROM firms WHERE id=740; SELECT id, name
SELECT name FROM firms WHERE id=849; FROM firms
SELECT name FROM firms WHERE id=839; WHERE id IN (723, 727, 740, 758, 812, 829, 839,
SELECT name FROM firms WHERE id=847; 847, 849, 857, 867, 868);
SELECT name FROM firms WHERE id=867;
SELECT name FROM firms WHERE id=829;
SELECT name FROM firms WHERE id=812;
SELECT name FROM firms WHERE id=868;
SELECT name FROM firms WHERE id=723;
http://ronaldbradford.com/blog/optimizing-sql-performance-the-art-of-elimination-2010-07-08/
http://ronaldbradford.com/blog/simple-lessons-in-improving-scalability-2011-02-16/
http://ronaldbradford.com/blog/the-rat-and-the-cat-2006-08-24/
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
44. Row Processing
EXAMPLE
SELECT name FROM firms WHERE id=727; Classic N+1 problem
SELECT name FROM firms WHERE id=758;
SELECT name FROM firms WHERE id=857;
SELECT name FROM firms WHERE id=740; SELECT id, name
SELECT name FROM firms WHERE id=849; FROM firms
SELECT name FROM firms WHERE id=839; WHERE id IN (723, 727, 740, 758, 812, 829, 839,
SELECT name FROM firms WHERE id=847; 847, 849, 857, 867, 868);
SELECT name FROM firms WHERE id=867;
SELECT name FROM firms WHERE id=829;
SELECT name FROM firms WHERE id=812;
SELECT name FROM firms WHERE id=868;
SELECT name FROM firms WHERE id=723;
http://ronaldbradford.com/blog/optimizing-sql-performance-the-art-of-elimination-2010-07-08/
http://ronaldbradford.com/blog/simple-lessons-in-improving-scalability-2011-02-16/
http://ronaldbradford.com/blog/the-rat-and-the-cat-2006-08-24/
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
45. Row Processing
EXAMPLE
SET PROFILING=1;
SELECT … 4X longer processing for
SHOW PROFILES; every page load
+----------+------------+---------------------------------------------------------
| Query_ID | Duration | Query
+----------+------------+---------------------------------------------------------
| 1 | 0.00030400 | SELECT name FROM firms WHERE id=727
SELECT 'Sum Individual Queries' AS txt,SUM(DURATI
| 2 | 0.00014400 | SELECT name FROM firms WHERE id=758
INFORMATION_SCHEMA.PROFILING WHERE QUERY_ID BETWE
| 3 | 0.00014300 | SELECT name FROM firms WHERE id=857
UNION
| 4 | 0.00014000 | SELECT name FROM firms WHERE id=740Query',SUM(DURATION) FROM INFORM
SELECT 'Combined
| 5 | 0.00012300 | SELECT name FROM firms WHERE 13;
QUERY_ID = id=849
| 6 | 0.00012200 | SELECT name FROM firms WHERE id=839
+------------------------+------------+
| 7 | 0.00011600 | SELECT name FROM firms WHERE id=847
| txt | total_time |
| 8 | 0.00014300 | SELECT name FROM firms WHERE id=867
+------------------------+------------+
| 9 | 0.00013900 | SELECT name FROM firms WHERE id=829Queries |
| Sum Individual 0.001311 |
| 10 | 0.00014000 | SELECT name FROM firms WHERE Query
| Combined id=812 | 0.000311 |
| 11 | 0.00012800 | SELECT name FROM firms WHERE id=868
+------------------------+------------+
| 12 | 0.00011700 | SELECT name FROM firms WHERE id=723
| 13 | 0.00031100 | SELECT id, name FROM firms WHERE id IN (723 ...
+----------+------------+---------------------------------------------------------
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
46. Row Processing
EXAMPLE
SET PROFILING=1;
SELECT … 4X longer processing for
SHOW PROFILES; every page load
+----------+------------+---------------------------------------------------------
| Query_ID | Duration | Query
+----------+------------+---------------------------------------------------------
| 1 | 0.00030400 | SELECT name FROM firms WHERE id=727
SELECT 'Sum Individual Queries' AS txt,SUM(DURATI
| 2 | 0.00014400 | SELECT name FROM firms WHERE id=758
INFORMATION_SCHEMA.PROFILING WHERE QUERY_ID BETWE
| 3 | 0.00014300 | SELECT name FROM firms WHERE id=857
UNION
| 4 | 0.00014000 | SELECT name FROM firms WHERE id=740Query',SUM(DURATION) FROM INFORM
SELECT 'Combined
| 5 | 0.00012300 | SELECT name FROM firms WHERE 13;
QUERY_ID = id=849
| 6 | 0.00012200 | SELECT name FROM firms WHERE id=839
+------------------------+------------+
| 7 | 0.00011600 | SELECT name FROM firms WHERE id=847
| txt | total_time |
| 8 | 0.00014300 | SELECT name FROM firms WHERE id=867
+------------------------+------------+
| 9 | 0.00013900 | SELECT name FROM firms WHERE id=829Queries |
| Sum Individual 0.001311 |
| 10 | 0.00014000 | SELECT name FROM firms WHERE Query
| Combined id=812 | 0.000311 |
| 11 | 0.00012800 | SELECT name FROM firms WHERE id=868
+------------------------+------------+
| 12 | 0.00011700 | SELECT name FROM firms WHERE id=723
| 13 | 0.00031100 | SELECT id, name FROM firms WHERE id IN (723 ...
+----------+------------+---------------------------------------------------------
Instant DB Scalability
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
47. resources
Many principles are straight forward
for experienced Oracle resoures
Open Source projects
Less skilled workforce
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
48. Architecture
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
49. ARCHITECTure
Storage Engines
Reducing SQL
Monitoring
Replication
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
50. replication
Weakest Link
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
51. conclusion
MySQL is a relational database
Internals are very different
Instrumentation is improving in new
versions 5.5 GA 5.6 DMR
Planet MySQL http://planet.mysql.com
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
52. Available NOW
http://effectivemysql.com/book/optimizing-sql-statements/
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011
53. Ronald Bradford
http://effectiveMySQL.com
EffectiveMySQL.com - Its all about Performance and Scalability
Wednesday, October 19, 2011