Percona Live 2014 presentation https://www.percona.com/live/mysql-conference-2014/sessions/xtradb-56-key-performance-algorithms

1. XtraDB 5.6
Key Performance Algorithms
Laurynas Biveinis
Alexey Stroganov
2014-04-02

2. Agenda
• Background: InnoDB and XtraDB in 5.5
• XtraDB 5.6 mk I: InnoDB with trx descriptors
• XtraDB 5.6 mk II: 5.5 performance patches
revisited
• XtraDB 5.6 mk III: flushing rework
• XtraDB 5.6: the rest and benchmarks
2

3. 5.5
• InnoDB 5.5 & XtraDB 5.5
3

4. InnoDB 5.5: Buffer Pool 4
Flush list: …
LRU list: …
Free list: …
Page hash:
flush list mutex
buffer pool mutex
Accessed on:
Page dirtying. Master thread
checkpoint age flush
Page access. Query thread flush
to make space in buffer pool
Flush to make space in buffer pool
Page lookup

5. MySQL solution for the contention 5
Flush list 1: …
LRU list 1: …
Free list 1: …
Page hash 1:
flush list mutex 1
buffer pool mutex 1
Flush list 2: …
LRU list 2: …
Free list 2: …
Page hash 2:
flush list mutex 2
buffer pool mutex 2

6. XtraDB BP mutex split solution 6
Flush list: …
LRU list: …
Free list: …
Page hash:
flush list mutex
LRU list mutex
free list mutex
page hash latch

7. The solutions may be combined 7
Flush list: …
LRU list 1: …
Free list 1: …
Page hash 1:
Flush list: …
LRU list 2: …
Free list 2: …
Page hash 2:

8. InnoDB 5.5: trx list 8
1. TRX_ACTIVE
kernel_mutex:
2. TRX_ACTIVE
3. TRX_COMMITTED_IN_MEMORY
trx_sys->trx_list:
…
Create
Read
View
list
scan
view->trx_ids:
1
2
…

9. XtraDB 5.5: trx descriptors 9
kernel_mutex:
trx_sys->descriptorsCreate
Read
view memcpy
view->descriptors:
1 2 … 1 2
• http://bugs.mysql.com/bug.php?id=49169
• http://bit.ly/pstrx1
• http://bit.ly/pstrxdesc2

10. InnoDB 5.5: AHI 10
4
2 3
. 7
1 5 64 8 97
. .
t1
d
b c
. g
a e fd h jg
. .
t2
t1.3
t1.5
t1.7
t2.c
t2.g
btr_search_sys->
hash_index
btr_search_latch:

11. XtraDB 5.5: AHI partitions 11
4
2 3
. 7
1 5 64 8 97
. .
t1
d
b c
. g
a e fd h jg
. .
t2
t1.3
t1.5
t1.7
t2.c
t2.g
btr_search_sys->
hash_tables[0]
btr_search_latch[0]:
btr_search_latch[1]:
btr_search_sys->
hash_tables[1]

12. XtraDB 5.6 mk I
• Can we drop the scalability patches?
– InnoDB 5.6 introduced cleaner thread, http://bit.ly/ibclean
– What it does:
●
Flush LRU tail, somehow, innodb_lru_scan_depth
●
Flush flush list, adaptively, innodb_io_capacity[_max]
●
Sleep until 1 s completes
●
Goto 1
– http://bit.ly/markexplainslru
●
We knew we could not drop the trx descriptor patch
12

13. XtraDB 5.6 mk I: InnoDB / trx desc 13

14. XtraDB 5.6 mk I: InnoDB / trx desc 14

15. XtraDB 5.6 mk I: InnoDB / trx desc 15
• Why XtraDB 5.6 mk I scales worse than XtraDB 5.5?
– AHI contention
●
OK, should have ported it too at once, no
changes there in 5.6
●
http://bugs.mysql.com/bug.php?id=62018
– Buffer pool mutex contention
●
No problem, we have a patch, right?

16. XtraDB 5.6 mk II: 5.5 patches
• Ported the AHI partition patch
– Improved it during the port.
http://bugs.mysql.com/bug.php?id=70216
– Worked as expected
• Ported the buffer pool mutex split
–Attempted to improve it during the port too
–InnoDB 5.6 split off page hash latch
–And it did not help much/at all
16

17. BP Access Concurrency Analysis, 5.5 17
master threadtime
flush list flush
flush list flush
flush list flush
query thread 1
Make page young
Make page young
LRU list flush
query thread 2
LRU list flush
Make page young
Make page young

18. BP Access Concurrency Analysis, 5.6 18
cleaner threadtime
LRU list flush
flush list flush
LRU list flush
query thread 1
Make page young
Make page young
LRU list flush
query thread 2
LRU list flush
Make page young
Make page young
flush list flush

19. XtraDB 5.6 mk III: flushing rework
• Issue 1: cleaner runs the flushing, but does the cleaner itself
run?
• One of the reasons innodb_thread_concurrency helps...
19
100% CPU run time share
few query threads cleaner
many query threads c

20. XtraDB 5.6 mk III: flushing rework
• Issue 1 solution: increase the cleaner
priority
20
100% CPU run time share
few query threads cleaner
many query threads cleaner

21. XtraDB 5.6 mk III: flushing rework
• setpriority()
• --innodb_sched_priority_cleaner=n
21
100% CPU run time share
few query threads cleaner
many query threads cleaner

22. XtraDB 5.6 mk III: flushing rework 22

23. XtraDB 5.6 mk III: flushing rework
• Issue 2: What happens in the case of an empty
free list in some instance?
• Query thread:
– Lock (buffer pool|free list) mutex
– If free list empty, unlock the mutex, do
something (*), goto 1
– * - query thread eviction / LRU flush attempt,
sleep before looping
23

24. XtraDB 5.6 mk III: flushing rework
• Issue 2. Still no free page, so given
enough query threads...
– Thread 1 waiting for the mutex
– Thread 2 has the mutex, checks
for empty free list
– Thread n waiting for the mutex
24

25. XtraDB 5.6 mk III: flushing rework
• Issue 2. Cleaner produced a free page!
– Thread 1 waiting for the mutex
– Thread 2 has the mutex, checks for
empty free list
– Thread n waiting for the mutex
– Cleaner thread waiting for the mutex
25

26. XtraDB 5.6 mk III: flushing rework
• Issue 2. Cleaner produced a free page, so what?
– Thread 2 unlocked the mutex
– Thundering herd of 1, 3..n, and cleaner threads,
mutex_signal_object()
– The chance that cleaner thread gets the mutex
is roughly 1 / n
– No mutex, no free page, n likely to
increase
26

27. XtraDB 5.6 mk III: flushing rework
• Issue 2. Solution: obviously, the
cleaner must get the free list mutex
before anybody else
• Adapted the sync array code
• mutex->high_priority_event
• mutex_enter_first()
27

28. XtraDB 5.6 mk III: flushing rework
• Issue 1+2: what applies to cleaner, applies
to other background threads as well
– Under high concurrency, background
threads must have priority over query
threads to keep the server in the steady
state
– State is: checkpoint age, history length, ...
28

29. XtraDB 5.6 mk III: flushing rework
• Issue 3. Why do query threads try to
(LRU|flush list) flush anyway?
• Because the cleaner thread failed to do
so in time
• Is it a good idea to increase mutex
pressure if the cleaner thread is already
in trouble?
29

30. XtraDB 5.6 mk III: flushing rework
• Issue 3 solution: do not let query threads
flush
• --innodb_empty_free_list_algorithm
=backoff
• --innodb_foreground_preflush
=exponential_backoff
• http://bit.ly/noqueryflush
30

31. XtraDB 5.6 mk III: flushing rework 31

32. XtraDB 5.6 mk III: flushing rework
• Issue 4: why do we have empty free lists in the first place?
• mysql -uroot -e'show engine innodb statusG'| grep Free
• Free buffers 1142
• Free buffers 1235
• Free buffers 1288
• Free buffers 318
• Free buffers 279
• Free buffers 938
• Free buffers 0
• Free buffers 406
32
• Free buffers 312
• Free buffers 687
• Free buffers 696
• Free buffers 910
• Free buffers 1029
• Free buffers 993
• Free buffers 537

33. XtraDB 5.6 mk III: flushing rework
• Issue 4: buffer pool instance usage is not uniform!
– Mikael Ronstrom: “First the accesses to the buffer pools
is in no way evenly spread out.”
– http://bit.ly/bpsplit
●
But 5.6 flushing does not recognize that!
– LRU/flush list flush request for x pages across y
instances:
●
Request x/y flush for each instance
●
Assuming uniformity where there is none
33

34. XtraDB 5.6 mk III: flushing rework
• Current issue 4 solution attempt
– “pseudoparallel” flushing: instead of issuing all
requests to each instance before advancing to
the next one, issue requests across instances
– keep flushing an instance in a loop if its free list
nearly empty
– innodb_free_list_lwm, default 100,
UNIV_PERF_DEBUG tuning var
34

35. XtraDB 5.6 mk III: flushing rework
• Proper issue 4 solution: parallel LRU
flushers for each instance
– Future work...
– 5.7 MT flushing is a step to right
direction but it must be ensured
that it does the right thing
35

36. XtraDB 5.6 mk III: flushing rework
• Talking of parallel LRU flushers...
• Busy cleaner thread:
36
flush list flush LRU list flush flush list flush LRU list flush ...
Empty free list danger here
Max checkpoint age danger here

37. XtraDB 5.6 mk III: flushing rework
• Solution is separate flush list and
LRU flush threads
37
flush list flush
LRU list flush
flush list flush
LRU list flush
flush list flush

38. XtraDB 5.6: the rest
• Last but not least:
– Adaptive flushing formula tweak to
maintain higher checkpoint age
– Efficient log block checksums
– http://bugs.mysql.com/bug.php?
id=70453
38

39. XtraDB 5.6 39