This presentation goes through performance tuning basics in MySQL Cluster.
It also covers the new parameters and status variables of MySQL Cluster 7.2 to determine issues with e.g disk data performance and query (join) performance.
Handwritten Text Recognition for manuscripts and early printed texts
Â
Conference slides: MySQL Cluster Performance Tuning
1. Performance Tuning of MySQL Cluster
Santa Clara, April 2012
Johan Andersson
Severalnines AB
johan@severalnines.com
Cell +46 73 073 60 99
Copyright Severalnines 2012
2. 2
Agenda
ï€ Scaling and Partitioning
ï€ Designing a Scalable System
ï€ Insert Performance Tuning
ï€ Query Tuning
ï€ Random tricks
ï€ Disk Data Tuning
Copyright Severalnines 2012
3. 3
Here is ...
Access Layer
App App
Server Server
MYSQL MYSQL
STORAGE LAYER
DATA DATA
NODE NODE
P0 P1
Node group 0
Copyright Severalnines 2012
4. 4
It can scale linearly ...
Access Layer
App App App App App App App App
Server Server Server Server Server Server Server Server
MYSQL MYSQL MYSQL MYSQL MYSQL NDBAPI NDBAPI NDBAPI
STORAGE LAYER STORAGE LAYER STORAGE LAYER
DATA DATA DATA DATA DATA DATA
NODE NODE NODE NODE NODE NODE
P0 P1 P2 P3 P4 P5
Node group 0 Node group 1 Node group 2
Copyright Severalnines 2012
5. if you find the bottlenecks
ï€ A lot of CPU is used on the data nodes
ï€ Probably a lot of large index scans and full table scans are used.
ï€ A lot of CPU is used on the mysql servers
ï€ Probably a lot of GROUP BY/DISTINCT or aggregate functions.
ï€ Hardly no CPU is used on either mysql or data nodes
ï€ Probably low load
ï€ Time is spent on network (a lot of âping pongâ to satisfy a request).
ï€ System is running slow in general
ï€ Disks (io util), queries, swap (should never happen)
Copyright Severalnines 2012
6. and if you know how
ï€ Adding mysqlds â trivial â if the mysqld is the bottleneck
ï€ BUT! Adding data nodes
ï€ More data nodes does not automatically give better performance
â
Latency may increase for a single query
â
Total throughout will be improved
ï€ How to get both?
Copyright Severalnines 2012
7. 7
Designing a
Scalable System
ï€ Define the most typical Use Cases
ï€ List all my friends, session management etc etc.
ï€ Optimize everything for the typical use case
ï€ Keep it simple
ï€ Complex access patterns does not scale
ï€ Simple access patterns do ( Primay key and Partitioned Index Scans )
ï€ Note! There is no parameter in config.ini that affects performance â only availability.
ï€ Everything is about the Schema and the Queries.
ï€ Tune the mysql servers (sort buffers etc) as you would for innodb.
Copyright Severalnines 2012
8. 8
Simple Access
ï€ PRIMARY KEY lookups are HASH lookup O(1)
ï€ INDEX searches a T-tree and takes O(log n) time.
ï€ In 7.2 JOINs are ok, but in 7.1 you should try to avoid
them.
Copyright Severalnines 2012
9. 9
Data Access
Access Layer
App App App App App App App App
Server Server Server Server Server Server Server Server
MYSQL MYSQL MYSQL MYSQL MYSQL NDBAPI NDBAPI NDBAPI
STORAGE LAYER
DATA DATA DATA DATA DATA DATA
NODE NODE NODE NODE NODE NODE
P0 P1 P2 P3 P4 P5
Node group 0 Node group 1 Node group 2
Copyright Severalnines 2012
10. 10
Data Access
ï€ One Request can hit all Partitions
ï€ Sub-optimal and won't scale
Copyright Severalnines 2012
11. 11
Data Access
Access Layer
App App App App App App App App
Server Server Server Server Server Server Server Server
MYSQL MYSQL MYSQL MYSQL MYSQL NDBAPI NDBAPI NDBAPI
STORAGE LAYER
DATA DATA DATA DATA DATA DATA
NODE NODE NODE NODE NODE NODE
P0 P1 P2 P3 P4 P5
Node group 0 Node group 1 Node group 2
Copyright Severalnines 2012
12. 12
Data Access
ï€ One Request hits one partition
ï€ Scales!
ï€ The number of Partitions (data nodes) does not matter!
ï€ Partitioning is important!
Copyright Severalnines 2012
13. 13
Partitioning
ï€ MySQL Cluster auto-partitions based on the Primary Key
ï€ Data is spread randomly
ï€ If possible better to Partition on a part of the Primary Key
ï€ CREATE TABLE user_friends(
userid,
friendid ,
somedata,
PRIMARY KEY (userid, friendid)) PARTITION BY KEY(userid)
ï€ All records with userid=X will be stored in the same partition!
ï€ Ultra important for MySQL Cluster 7.2 and Fast JOINs.
Copyright Severalnines 2012
14. 14
Partitioning
ï€ EXPLAIN PARTITIONS <query>
ï€ Tells you what partitions you touch.
ï€ Also verify with:
mysql> show global status like 'ndb_pruned_scan_countâ;
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| Ndb_pruned_scan_count | 1 |
+-----------------------+-------+
ï€ Increases when Partition Pruning could be used.
Copyright Severalnines 2012
15. 15
Insert Performance
ï€ Scaling Inserts
ï€ Option 1) Batch INSERTS if you can
â An insert batch of 10 records will perform 10x faster than 10 single
inserts!
â INSERT INTO t1 VALUES (<record1>), (<record2>), âŠ,(<recordN>)
ï€ Option 2) Many threads (parallelism)
ï€ Or a combo of both
ï€ Dumpfiles or LOAD DATA INFILEs
ï€ Chunk them up and load in parallel on several mysqlds
Copyright Severalnines 2012
16. 16
Insert Performance
ï€ INSERTs in a table with AUTO_INCREMENT
ï€ MySQL Server query Data nodes for an auto_increment
â The mysqld can hold a range of autoincs (cache)
â Before an INSERT, and autoinc must be fetched from either Data node
(slow) or on the cache (fast)
ï€ ndb_autoincremet_prefetch_sz sets the cache size and it affects insert perf:
â ndb_autoincrement_prefetch_sz=1: 1211.91TPS
â ndb_autoincrement_prefetch_sz=256: 3471.71TPS
â ndb_autoincrement_prefetch_sz=1024: 3659.52TPS
Copyright Severalnines 2012
17. 17
Insert Performance
ï€ ndb_batch_size can also be important with LOAD DATA INFILE or
dumps
ï€ SET GLOBAL|SESSION NDB_BATCH_SIZE = 16M
ï€ You may get LongMessageBuffer Overload
â Increase it in config.ini to 32M or 48M
ï€ Also REDO logs may get overloaded if your disks are too slow and/or
the REDO is too small.
Copyright Severalnines 2012
18. 18
Query Performance
ï€ Queries needs to be tuned as âusualâ:
ï€ Slow query / general log
ï€ From a monitoring system (like ClusterControl)
ï€ + a methodology
Copyright Severalnines 2012
19. Query Performance
ï€ Slow query log
ï€ set global slow_query_log=1;
ï€ set global long_query_time=0.01;
ï€ set global log_queries_not_using_indexes=1;
ï€ General log (if you donât get enough info in the Slow Query Log)
ï€ Activate for a very short period of time (30-60seconds) â intrusive
ï€ Can fill up disk very fast â make sure you turn it off.
ï€ set global general_log=1;
ï€ Use Severalnines ClusterControl
ï€ Includes a Cluster-wide Query Monitor.
ï€ Query frequency, EXPLAINs, lock time etc.
ï€ Performance Monitor and Manager.
Copyright Severalnines 2012
20. Data Types
ï€ BLOB/TEXT columns are stored in an external hidden table.
ï€ First 255B are stored inline in main table
ï€ Reading a BLOB/TEXT requires two reads
ï€ One for reading the Main table + reading from hidden table
ï€ Change to VARBINARY/VARCHAR if:
ï€ Your BLOB/TEXTs can fit within an 8052B record
ï€ (record size is currently 8052 Bytes)
ï€ Reading/writing VARCHAR/VARBINARY is less expensive
Note 1: BLOB/TEXT are also more expensive in Innodb as BLOB/TEXT data is not
inlined with the table. Thus, two disk seeks are needed to read a BLOB.
Note 2: Store images, movies etc outside the database on the filesystem.
Copyright Severalnines 2012
21. Data Types
ï€ Example
CREATE TABLE `t1_blob` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`data1` blob,
`data2` blob,
PRIMARY KEY (`id`)
)ENGINE=ndbcluster
ï€ Performance (8 threads, one mysqld, two data nodes):
ï€ data1 and data2 as BLOBs: 5844TPS
ï€ data1 and data2 as VARBINARY: 19206TPS
ï€ ~3x
Copyright Severalnines 2012
22. Denormalize
ï€ Tables sharing the same PRIMARY KEY can be denormalized.
ï€ Table T1: <UID, SOME_DATA>
ï€ Table T2: <UID, SOME_OTHER_DATA
ï€ SELECT * from T1,T2 WHERE T1.UID=T2.UID and T2.UID=1 requires
two roundtrips.
ï€ Starting with MySQL Cluster 7.2 only one roundtrip is needed,.
ï€ Denormalize
ï€ Table T12: <UID,SOME_DATA, SOME_OTHER_DATA>
ï€ Improvement: 2X in throughput
Copyright Severalnines 2012
23. Query Tuning < 7.2
ï€ Don't trust the OPTIMIZER in MySQL Cluster 7.1 and earlier
ï€ Statistics gathering is non-existing
ï€ Optimizer thinks there are only 10 rows to examine in each table!
ï€ You have to do a lot of
ï€ FORCE INDEX / STRAIGH_JOIN to get queries run the way you
want.
Copyright Severalnines 2012
24. Query Tuning < 7.2
ï€ Classic example: if you have two similar indexes:
ï€ index(a)
ï€ index(a,ts)
on the following table
CREATE TABLE `t1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`a` bigint(20) DEFAULT NULL,
`ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_t1_a` (`a`),
KEY `idx_t1_a_ts` (`a`,`ts`)) ENGINE=ndbcluster DEFAULT CHARSET=latin1
Copyright Severalnines 2012
25. Query Tuning < 7.2
mysql> explain select * from t1 where a=2 and ts='2011-10-05 15:32:11';
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
| 1 | SIMPLE | t1 | ref | idx_t1_a,idx_t1_a_ts | idx_t1_a | 9 | const | 10 | Using where |
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
ï€ Use FORCE INDEX(..) ...
mysql> explain select * from t1 FORCE INDEX (idx_t1_a_ts) where a=2 and ts='2011-10-05
15:32:11;
+| 1 | SIMPLE | t1 | ref | idx_t1_a_ts | idx_t1_a_ts | 13 | const,const | 10 | Using where |
1 row in set (0.00 sec)
ï€ ..to ensure the correct index is picked!
ï€ The difference can be 1 record read instead of any number of
records!
Copyright Severalnines 2012
26. 26
Query Tuning in 7.2
ï€ ANALYZE TABLE
ï€ Must be performed periodically to rebuild index stats
ï€ EXPLAIN EXTENDED/PARTITIONS
ï€ Make sure the explain show âChild of JOIN pushed downâ
ï€ This means that the Fast JOIN of NDB could be used
ï€ SHOW WARNINGS;
â Shows why a Query was not pushed down.
Copyright Severalnines 2012
27. Ndb_cluster_connection_pool
ï€ Problem:
ï€ A Sendbuffer on the connection between mysqld and the data nodes is protected
by a Mutex.
ï€ Connection threads in MySQL must acquire Mutex and the put data in SendBuffer.
ï€ Many threads gives more contention on the mutex
ï€ Must scale out with many MySQL Servers.
ï€ Workaround:
ï€ Ndb_cluster_connection_pool (in my.cnf) creates more connections from one
mysqld to the data nodes
ï€ Threads load balance on the connections gives less contention on mutex which in
turn gives increased scalabilty
ï€ Less MySQL Servers needed to drive load!
ï€ www.severalnines.com/cluster-configurator allows you to specify the connection
pool.
ï€ >70 % improvement.
Copyright Severalnines 2012
28. Ndb_cluster_connection_pool
ï€ Gives atleast 70% better performance and a MySQL Server that
can scale beyond four database connections.
ï€ Set Ndb_cluster_connection_pool=2x<CPU cores>
ï€ It is a good starting point
ï€ One free [mysqld] slot is required in config.ini for each
Ndb_cluster_connection.
ï€ 4 mysql servers,each with Ndb_cluster_connection_pool=8 requires 32
[mysqld] in config.ini
Copyright Severalnines 2012
29. Disk Data Tuning
ï€ Disk Data Tables
ï€ Un-indexed columns â tablespace on disk
ï€ Indexed columns â DataMemory
ï€ DiskPageBufferMemory (DPBM) â LRU page cache
ï€ Like innodb_buffer_pool
ï€ Should be big as possible
ï€ If data not in DPBM DiskPage
ï€ Go to TS and fetch (Slow) IndexMemory DataMemory Buffer
Memory
ï€ If data is DPBM
REDO LOG
ï€ Return page (faster) UNDO LOG
Tablespace
Copyright Severalnines 2012
30. Disk Data Tuning
ï€ DiskPageBufferMemory
â Hit ratio (derived from ndbinfo.diskpagebuffer):
â 1000*page_requests_direct_return/
(page_requests_direct_return +
page_requests_wait_io+
page_requests_wait_queue)
â 998 is good (like in innodb).
â DiskPageBufferMemory=2048M is a good start
DiskPage
IndexMemory DataMemory Buffer
Memory
REDO LOG
UNDO LOG
Tablespace
Copyright Severalnines 2012
31. Disk Data Tuning
ï€ UNDO LOG
â Always overseen but can be extended overtime
â Set it to 50% of the REDO log size :
â 0.5 x NoOfFragmentLogFiles x FragmentLogFileSize
ï€ Undo buffer (specd in CREATE LOGFILE GROUP)
â 32M to 64M (like the RedoBuffer)
â SharedGlobalMemory=512M
DiskPage
IndexMemory DataMemory Buffer
Memory
REDO LOG
UNDO LOG
Tablespace
Copyright Severalnines 2012
32. More on Cluster
ï€ Severalnines Forum
â http://support.severalnines.com/forums/20323398-mysql-cluster
ï€ Johan Andersson @ blogspot
â http://johanandersson.blogspot.com
ï€ Configuration and Deployment
â http://www.severalnines.com/cluster-configurator
â ~20 min to deploy a 4 node cluster (288 seconds is the
World Record)
ï€ Self-training
â http://severalnines.com/mysql-cluster-training
Copyright Severalnines 2012