The agenda of this talk was to introduce MySQL Replication and then follow it up with Multi-threaded slaves(MTS) support. The presentation introduces Multi threading slaves by database which is a part of MySQL-5.6 as well as multi-threading policy introduced in MySQL-5.7.2. Finally there is a brief coverage of the new replication monitoring tables to monitor MySQL Replication. These tables are part of MySQL Performance Schema.

1. MySQL Replication
and
Multi-threaded Slaves
Shivji Kumar Jha,
Software Developer,
MySQL Replication Team

2. Safe Harbour Statement
The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated into any
contract.
It is not a commitment to deliver any material, code, or functionality, and
should not be relied upon in making purchasing decisions. The development,
release, and timing of any features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.
`2
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

3. Agenda
Why Replication?

Replication Internals- A Simple Introduction

Why Multi-threaded Slaves (MTS)?

Different policies for Multi-threading Slaves

`3

Keeping track of replication threads
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

4. Replication: Copy Changes Master → Slave
 MySQL Master Server
– Changes data
– Sends changes to slave
 MySQL Slave Server
– Receives changes from master
– Applies received changes to database
M
`4
S
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

5. Why Replication? – Scalability
 Read scale-out
M
write clients
`5
S
read clients
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

6. Why Replication? – Scalability
 Read scale-out
M
write clients
`6
S
More
reads?
read clients
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

7. Why Replication? – Scalability
 Read scale-out
M
write clients
`7
S
More
reads?
More
slaves!
read clients
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

8. Why Replication? – Scalability
 Read scale-out
S
M
S
More
reads?
More
slaves!
M
S
S
write clients
write clients
`8
read clients
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
read clients

9. Why Replication? – Redundancy
 If master crashes, promote slave to master
B
A
C
`9
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

10. Why Replication? – Redundancy
 If master crashes, promote slave to master
B
Crash
A
C
`10
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

11. Why Replication? – Redundancy
 If master crashes, promote slave to master
B is the
new master
B
A
C
`11
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

12. Why Replication? – Disaster recovery
C
A
A
B
C
B
Image from
www.ginkgomaps.com
`12
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

13. But how do you copy changes to slave?
`13
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

14. But how do you copy changes to slave?
Are there LOGS floating around?
`14
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

15. Introducing replication LOGS...
`15
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

16. Replication Logs
 MySQL Master Server
M
– Changes data (Writes)
Binary
log
– Saves changes in Binary log
 MySQL Slave Server
– Copies changes to Relay log
– Reads Relay log
– Applies changes.
`16
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Relay
log
S

17. All Changes Written to Binary Log
Client
binary log
A
`17
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

18. All Changes Written to Binary Log
Client
create table t (a int);
binary log
A
`18
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

19. All Changes Written to Binary Log
Client
create table t (a int);
binary log
create...
A
Table t
`19
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. |

20. All Changes Written to Binary Log
Client
create table t (a int);
insert into t values (1);
binary log
create...
A
Table t
`20
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

21. All Changes Written to Binary Log
Client
create table t (a int);
insert into t values (1);
binary log
A
create...
insert...
Table t
1
`21
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

22. Slave Initiates Replication
1. Slave sends
request to start replication
to master
Client
relay log
binary log
B
A
2. Master sends
stream of replication data
to slave
`22
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

23. Slave Copies, Applies Same Changes
Client
create...
binary log
A
`23
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
relay log
B

24. Slave Copies, Applies Same Changes
Client
create...
binary log
relay log
create...
A
Table t
`24
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
B

25. Slave Copies, Applies Same Changes
Client
create...
binary log
relay log
create...
create...
A
Table t
`25
B
Table t
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

26. Slave Copies, Applies Same Changes
Client
create...
insert...
binary log
A
relay log
create...
insert...
create...
Table t
1
`26
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
B
Table t

27. Slave Copies, Applies Same Changes
Client
create...
insert...
binary log
A
relay log
create...
insert...
create...
insert...
Table t
1
`27
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
B
Table t
1

28. OK makes sense !
`28
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

29. But I see clients applying changes at
Master, who applies at Slave?
`29
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

30. But I see clients applying changes at
Master, who applies at Slave?
Do you have some internal
replication threads?
`30
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

31. Introducing replication THREADS...
`31
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

32. Replication Threads (4.0-5.5)
Client
create...
insert...
relay log
binary log
A
create...
insert...
create...
network
insert... SQL THREAD
IO THREAD
Writes into relay log at slave
`32
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
B
Reads from relay log.
Applies transactions at slave

33. Replication Threads (4.0-5.5)
Create... insert...
Client
Client
insert...
More clients on master...
Client
relay log
binary log
A
`33
create...
insert...
Insert...
create...
network
insert... SQL THREAD
IO THREAD
Insert...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
B

34. Replication Threads (4.0-5.5)
Create... insert...
Client
Client
insert...
More clients on master...
Client
Slave starts to lag behind master
relay log
binary log
A
`34
create...
insert...
Insert...
create...
network
insert... SQL THREAD
IO THREAD
Insert...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
B

35. Replication Threads (4.0-5.5)
Create... insert...
Client
Client
insert...
More clients on master...
Client
Slave starts to lag behind master
relay log
binary log
A
create...
insert...
Insert...
create...
network
insert... SQL THREAD
IO THREAD
Insert...
Table t
1
2
3
`35
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
B
Table t
1
2

36. Lets Parallelize
slave as well...
`36
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

37. Lets Parallelize
slave as well...
Introducing parallelization by
Database
`37
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

38. Replication Threads (5.6+)
create...
Client
insert... insert... Reads & assigns to worker(s)
Client
Client
w
create..
O
insert... R
K
COORDINATOR
E
insert..
THREAD
R
S
binary log
A
`38
create(db1)
insert(db2)
insert(db3)
network
IO THREAD
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
db1
B db2
db3
create(db1)
insert(db2)
insert(db3) Worker threads apply
concurrently on diff dbs
relay log

39. Replication Threads (5.6+)
Setting up Parallelization by database
mysql> STOP SLAVE;
mysql> SET GLOBAL slave_parallel_workers=1;
mysql> START SLAVE;
`39
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

40. Replication Threads (5.6+)
More databases? More workers!
mysql> STOP SLAVE;
mysql> SET GLOBAL slave_parallel_workers=2;
mysql> START SLAVE;
`40
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

41. What if I have a
a loaded
Database?
Image credits: http://www.easetechnology.co.uk/ask-an-expert/
`41
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

42. Lets Parallelize slave
somewhat similar to master
`42
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

43. Lets Parallelize slave
somewhat similar to master
5.7.2
Introducing parallelization by
Master concurrency
`43
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

44. Replication Threads (4.0-5.5)
Create... insert...
Client
Client
insert...
More clients on master...
Client
Slave starts to lag behind master
relay log
binary log
A
Table t
1
2
3
`44
create...
insert...
Insert...
create...
network
insert... SQL THREAD
IO THREAD
Insert...
REVISITING THE OLD SLIDE !!!
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
B
Table t
1
2

45. Replication Threads (5.7+)
create...
Client
insert... insert... Reads & assigns to worker(s)
Client
Client
COORDINATOR
THREAD
binary log
A
create...
insert...
Insert...
network
IO THREAD
capture parallelization info on master
`45
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
create...
insert...
Insert...
relay log
w
O
R
K
E
R
S
B
db1
db2
Worker threads apply
concurrently

46. Replication Threads (5.7+)
create...
Client
insert... insert... Reads & assigns to worker(s)
Client
Client
COORDINATOR
THREAD
binary log
A
create...
insert...
Insert...
network
IO THREAD
Save parallelization info in binary log
`46
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
create...
insert...
Insert...
relay log
w
O
R
K
E
R
S
B
db1
db2
Worker threads apply
concurrently

47. Replication Threads (5.7+)
create...
Client
insert... insert... Reads & assigns to worker(s)
Client
Client
COORDINATOR
THREAD
binary log
A
create...
insert...
Insert...
network
IO THREAD
Send parallelization info to slave
`47
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
create...
insert...
Insert...
relay log
w
O
R
K
E
R
S
B
db1
db2
Worker threads apply
concurrently

48. Replication Threads (5.7+)
create...
Client
insert... insert... Reads & assigns to worker(s)
Client
Client
COORDINATOR
THREAD
binary log
A
create...
insert...
Insert...
network
IO THREAD
Use parallelization info to assign
`48
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
create...
insert...
Insert...
relay log
w
O
R
K
E
R
S
B
db1
db2
Worker threads apply
concurrently

49. Replication Threads (5.7+)


`49
Leverage parallelization information obtained from the execution on the
master:
—
Transactions that reach the prepare phase on the same data snapshot
are non-contending;
—
Write to the binary log on which snapshot id each transaction prepared;

This identifies the commit parent of each transaction.
—
The commit parent is stepped every time transactions commit.
Meanwhile, at the slave:
—
Transactions with the same commit parent can be executed in parallel;
—
Commit sequence at the slave may not be the same as that on the
master, but it is still a correct execution history.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. |

50. Higher Slave Throughput: Intra-Schema MTS
Concurrent Execution History
on the Master
T1
T2
T3
Time
Execution
Commit
Prepare
Commit
`50
commit_parent++
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. |
Prepare

51. Higher Slave Throughput: Intra-Schema MTS
Concurrent Execution History
on the Master
Parallel
on the Slave.
T1
T2
T3
Time
Execution
Commit
Prepare
Commit
`51
commit_parent++
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. |
Prepare
Not parallel
on the slave.

52. Replication Threads (5.7+)
Setting up Parallelization by master concurrency
mysql> STOP SLAVE;
mysql> SET GLOBAL slave_parallel_type=[ 'logical_clock' | 'database' ];
mysql> SET GLOBAL slave_parallel_workers=3;
mysql> START SLAVE;
database=>parallelize by database
logical_clock=>parallelize by master concurrency
`52
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

53. Too many threads
here and there!
How do I monitor?
Knew that was coming!
Check out replication
Performance_Schema
`53
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

54. Performance Schema (5.5+)

Inspect internal execution of the server at runtime

Exposed within performance_schema database

Records various run time statistics via in-built instrumentation points

Tracks latency for various events:
File I/O, Mutexes, Read/Write Locks, Table I/O, Table Locks,
Stages, Statements, Idle etc..

`54
Its a long growing list
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

55. Replication Performance Schema Tables (5.7+)
replication
5.7.2
execution
connection
configuration
status
configuration
Coordinator thread / SQL thread
`55
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
status
Worker thread(s)

56. The REPLICATION P_S Tables

We have six performance schema tables for replication:
5.7.2
(MySQL-5.7.2):

replication_connection_configuration

replication_connection_status (IO thread status)

replication_execute_configuration

replication_execute_status

replication_execute_status_by_coordinator

replication_execute_status_by_worker
Tables for monitoring MTS
`56
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

57. Monitoring The Coordinator Thread
5.7.2
mysql> select * from performance_schema.replication_execute_status_by_coordinatorG
*************************** 1. row ***************************
THREAD_ID: 13
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
`57
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
No error in thread

58. Monitoring The Coordinator Thread
5.7.2
mysql> select * from performance_schema.replication_execute_status_by_coordinatorG
*************************** 1. row ***************************
THREAD_ID: 13
SERVICE_STATE: ON
Oops! There was an error
LAST_ERROR_NUMBER: 1146
LAST_ERROR_MESSAGE:...'Table 'test.t' doesn't exist'...
LAST_ERROR_TIMESTAMP: 2013-11-04 13:37:23
`58
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

59. Monitoring Worker Threads
5.7.2
mysql> SET GLOBAL slave_parallel_workers= 2;
mysql> select * from performance_schema.replication_execute_status_by_workerG
*************************** 1. row ***************************
WORKER_ID: 0
THREAD_ID: 16
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION: 2b3e03a6-453f-11e3-8668-0021ccb3570d:3
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
WORKER_ID: 1
THREAD_ID: 17
one row per worker thread
SERVICE_STATE: ON
...
`59
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

60. Monitoring Worker Threads
5.7.2
mysql> SET GLOBAL slave_parallel_workers= 2;
mysql> select * from performance_schema.replication_execute_status_by_workerG
*************************** 1. row ***************************
WORKER_ID: 0
THREAD_ID: 16
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION: 2b3e03a6-453f-11e3-8668-0021ccb3570d:3
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
WORKER_ID: 1
THREAD_ID: 17 The newest transaction this worker is aware
SERVICE_STATE: ON
...
`60
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
of

61. Monitoring Worker Threads
5.7.2
mysql> SET GLOBAL slave_parallel_workers= 2;
mysql> select * from performance_schema.replication_execute_status_by_workerG
*************************** 1. row ***************************
WORKER_ID: 0
THREAD_ID: 16
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION: 2b3e03a6-453f-11e3-8668-0021ccb3570d:3
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
No error in this worker
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
WORKER_ID: 1
THREAD_ID: 17
SERVICE_STATE: ON
...
`61
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
thread

62. 
A lot more to explore in MySQL-5.6, keep
reading...

MySQL-5.7 is our current development
branch.
We want your valuable feedback.

Suggest Features, report bugs, contribute
patches.

`62


We Want
Your
Feedback
MySQL-5.6 is our latest GA release.
Help make MySQL-5.7 even better!
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

63. 

Replication Logs

Replication Threads

Summary
Introduction to MySQL Replication
Multi-threaded Slave (MTS)
– Need for Parallelization
– Parallelize by database
– Parallelize by master concurrency

Keeping track of replication threads
– Performance Schema
– Replication info. in Performance Schema
`63
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

64. Read More
About
MySQL
Replication
`64
Find MySQL Official Documentation at
http://dev.mysql.com/doc/refman/5.7/en/replication.html
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

65. 
Read More
About
MultiThreaded
Slaves
`65
Parallelization by database
- Luis's blog
- Andrei's blog

Parallelization by master concurrency
- Rohit's blog

Replication Performance Schema
- Official MySQL documentation
- Shiv's blog
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

66. Questions!
Email: shivji.jha@oracle.com
Blog: shivjijha.blogspot.in
`66
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |