This document provides an overview of MySQL replication including: - Replication enables data from a master database to be replicated to one or more slave databases. - Binary logs contain all writes and schema changes on the master which are used by slaves to replicate data. - Setting up replication involves configuring the master to log binary logs, granting replication privileges, and configuring slaves to connect to the master and read binary logs from the specified position. - Commands like START SLAVE are used to control replication and SHOW SLAVE STATUS displays replication status and lag.

1. Introduction To MySQL Replication
Kenny Gryp <kenny.gryp@percona.com>
Percona Live Washington DC / 2012-01-11

2. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 2

3. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 3

4. What is Replication?
Replication enables data from one MySQL database server
(the master) to be replicated to one or more MySQL
database servers (the slaves)
(Source: http://dev.mysql.com/doc/refman/5.5/en/replication.html)
www.percona.com 4

5. Replication Diagram
APPLICATION
Queries
MASTER SLAVE
... SQL Thread ...
MyISAM MyISAM
INNODB INNODB
IO Thread
BINLOGS RELAY LOGS
www.percona.com 5

6. Replication Diagram
APPLICATION
Queries Queries
MASTER BINLOGS
SLAVE
... SQL Thread ...
MyISAM MyISAM
INNODB INNODB
IO Thread
BINLOGS RELAY LOGS
www.percona.com 6

7. Replication
Happens at MySQL level, not Storage Engine Level (*NDB)
Asynchronous! (Semi-sync available in 5.5)
A server can have only 1 master
IO Thread: Fetches from master
SQL Thread: Executes on slave
Single Threaded Execution (Expect enhancements in 5.6)
Different schema’s are possible between master and slave
(Watch out!!):
different indexes
storage engines
data types
columns
www.percona.com 7

8. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 8

9. Binary Logs
BINLOG.000001 BINLOG.index
BINLOG.000002
BINLOG.000001
BINLOG.000003 BINLOG.000002
BINLOG…. BINLOG.000003
www.percona.com 9

10. Binary Logs
Set of files
Contains all writes and schema changes
!= REDO/Transaction log
Rotated when full (Set max_binlog_size)
Incrementing numbers (000001,000002,000003,...)
Relay Logs are also binary logs
2 Formats:
Statement Based (SBR)
Row Based (RBR, since MySQL 5.1)
Binary file
www.percona.com 10

11. Binary Logs - Info Files
master.info
Contains IO Thread information & connection information
relay.info
Contains SQL Thread information
www.percona.com 11

12. Binary Log - Formats (binlog_format)
Statement Based Replication (SBR):
Writes statements to binary logs, slave executes the stmts
(An update stmt that changes 1 row but reads many will also do
the same on the slave)
Some functions are non-deterministic and cause
inconsistencies:
UUID(), SYSDATE(), FOUND_ROWS(), UPDATE .. LIMIT without
ORDER BY...
Works for NOW(): timestamp included in binary log
More complete list of issues:
http://dev.mysql.com/doc/refman/5.5/en/replication-features.html
www.percona.com 12

13. Binary Log - Formats (binlog_format)
Row Based Replication (RBR, since 5.1):
Write row changes (larger binlogs)
Check Binlog_cache_disk_use, possibly increase
binlog_cache_size
Does not need to parse/execute queries, just make the changes
necessary
Much less/different issues compared to SBR:
http://dev.mysql.com/doc/refman/5.5/en/replication-rbr-
usage.html
Mixed: Combination of both: defaults to SBR, use RBR
when necessary
www.percona.com 13

14. Looking at Binary Log Contents
mysqlbinlog
SHOW BINLOG EVENTS
www.percona.com 14

15. Example SBR
> SHOW GLOBAL VARIABLES LIKE 'binlog_format';
+---------------+-----------+
| Variable_name | Value |
+---------------+-----------+
| binlog_format | STATEMENT |
+---------------+-----------+
1 row in set (0.00 sec)
> CREATE DATABASE replication;
Query OK, 1 row affected (0.14 sec)
> use replication
Database changed
> CREATE TABLE repl (a int) ENGINE=innodb;
Query OK, 0 rows affected (0.25 sec)
> INSERT INTO repl VALUES (1);
Query OK, 1 row affected (0.14 sec)
www.percona.com 15

16. Example SBR - mysqlbinlog
# mysqlbinlog mysql-bin.000193
...
# at 106
#120106 15:19:13 server id 9999 end_log_pos 203 ! Query! thread_id=11
CREATE DATABASE replication
/*!*/;
# at 203
#120106 15:19:32 server id 9999 end_log_pos 312 ! Query! thread_id=11
! exec_time=1! error_code=0
use replication/*!*/;
SET TIMESTAMP=1325859572/*!*/;
CREATE TABLE repl (a INT) ENGINE=innodb
/*!*/;
# at 312
#120106 15:19:55 server id 9999 end_log_pos 387 ! Query! thread_id=11
! exec_time=0! error_code=0
SET TIMESTAMP=1325859595/*!*/;
BEGIN
/*!*/;
# at 387
#120106 15:19:55 server id 9999 end_log_pos 484 ! Query! thread_id=11
! exec_time=0! error_code=0
SET TIMESTAMP=1325859595/*!*/;
INSERT INTO repl VALUES (1)
/*!*/;
# at 484
#120106 15:19:55 server id 9999 end_log_pos 511 ! Xid = 14 www.percona.com 16
COMMIT/*!*/;

17. > SHOW BINLOG EVENTS FROM 106G
*************************** 1. row ***************************
Log_name: mysql-bin.000193
Example SBR - SHOW BINLOG EVENTS
Pos: 106
Event_type: Query
Server_id: 1
End_log_pos: 203
Info: CREATE DATABASE replication
*************************** 2. row ***************************
Log_name: mysql-bin.000193
Pos: 203
Event_type: Query
Server_id: 1
End_log_pos: 312
Info: use `replication`; CREATE TABLE repl (a INT) ENGINE=innodb
*************************** 3. row ***************************
Log_name: mysql-bin.000193
Pos: 312
Event_type: Query
Server_id: 1
End_log_pos: 387
Info: BEGIN
*************************** 4. row ***************************
Log_name: mysql-bin.000193
Pos: 387
Event_type: Query
Server_id: 1
End_log_pos: 484
Info: use `replication`; INSERT INTO repl VALUES (1)
*************************** 5. row ***************************
Log_name: mysql-bin.000193
Pos: 484
Event_type: Xid
Server_id: 1
End_log_pos: 511
Info: COMMIT /* xid=14 */
5 rows in set (0.00 sec) www.percona.com 17

18. Example RBR
> SHOW GLOBAL VARIABLES LIKE 'binlog_format';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| binlog_format | ROW |
+---------------+-------+
1 row in set (0.00 sec)
> CREATE DATABASE replication;
Query OK, 1 row affected (0.14 sec)
> use replication
Database changed
> CREATE TABLE repl (a int) ENGINE=innodb;
Query OK, 0 rows affected (0.25 sec)
> INSERT INTO repl VALUES (1);
Query OK, 1 row affected (0.14 sec)
www.percona.com 18

19. Example RBR - mysqlbinlog
# mysqlbinlog mysql-bin.000193 --start-position=606
...
# at 606
#120106 15:54:54 server id 1 end_log_pos 703 ! Query! thread_id=11
! exec_time=0! error_code=0
CREATE DATABASE replication
/*!*/;
# at 703
#120106 15:55:02 server id 1 end_log_pos 812 ! Query! thread_id=11
! exec_time=0! error_code=0
use replication/*!*/;
SET TIMESTAMP=1325861702/*!*/;
CREATE TABLE repl (a int) ENGINE=innodb
/*!*/;
# at 812
...
# at 937
#120106 15:55:06 server id 1 end_log_pos 937 ! Table_map: `replication`.`repl` mapped to
number 17
#120106 15:55:06 server id 1 end_log_pos 971 ! Write_rows: table id 17 flags: STMT_END_F
BINLOG '
SgsHTxMBAAAAMgAAAKkDAAAAABEAAAAAAAEAC3JlcGxpY2F0aW9uAARyZXBsAAEDAAE=
SgsHTxcBAAAAIgAAAMsDAAAAABEAAAAAAAEAAf/+AQAAAA==
'/*!*/;
# at 971
#120106 15:55:06 server id 1 end_log_pos 998 ! Xid = 34 www.percona.com 19
COMMIT/*!*/;

20. Example RBR -
mysqlbinlog --verbose
# mysqlbinlog mysql-bin.000193 --verbose --verbose
...
# at 937
#120106 15:55:06 server id 1 end_log_pos 937 ! Table_map: `replication`.`repl` mapped to
number 17
#120106 15:55:06 server id 1 end_log_pos 971 ! Write_rows: table id 17 flags: STMT_END_F
BINLOG '
SgsHTxMBAAAAMgAAAKkDAAAAABEAAAAAAAEAC3JlcGxpY2F0aW9uAARyZXBsAAEDAAE=
SgsHTxcBAAAAIgAAAMsDAAAAABEAAAAAAAEAAf/+AQAAAA==
'/*!*/;
### INSERT INTO replication.repl
### SET
### @1=1 /* INT meta=0 nullable=1 is_null=0 */
# at 971
www.percona.com 20

21. > SHOW BINLOG EVENTS FROM 606G
*************************** 1. row ***************************
Log_name: mysql-bin.000193
Example RBR - SHOW BINLOG EVENTS
Pos: 606
Event_type: Query
Server_id: 1
End_log_pos: 703
Info: CREATE DATABASE replication
*************************** 2. row ***************************
Log_name: mysql-bin.000193
Pos: 703
Event_type: Query
Server_id: 1
End_log_pos: 812
Info: use `replication`; CREATE TABLE repl (a int) ENGINE=innodb
...
*************************** 4. row ***************************
Log_name: mysql-bin.000193
Pos: 887
Event_type: Table_map
Server_id: 1
End_log_pos: 937
Info: table_id: 17 (replication.repl)
*************************** 5. row ***************************
Log_name: mysql-bin.000193
Pos: 937
Event_type: Write_rows
Server_id: 1
End_log_pos: 971
Info: table_id: 17 flags: STMT_END_F
*************************** 6. row ***************************
Log_name: mysql-bin.000193
Pos: 971
Event_type: Xid
Server_id: 1
End_log_pos: 998
Info: COMMIT /* xid=34 */ www.percona.com 21
6 rows in set (0.00 sec)

22. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 22

23. Setting up Replication
Prerequisites
Change master/slave MySQL configuration
Configure Replication
Start Replication/Check Status
www.percona.com 23

24. Prerequisites
No way to easily create slave with 1 command
It’s required to Create/Restore consistent backup using
your favorite backup tool
with mysqldump, use --master-data
Binlog file and position from backup should be recorded
File: mysql-bin.000008
Pos: 106
www.percona.com 24

25. Change master/slave Configuration
On the master:
Enable Binary Logging: log-bin=log-bin
Set A Server ID: server-id=1
On the slave:
Set A Server ID: server-id=2
www.percona.com 25

26. Why server-id ?
APPLICATION
Queries Queries
MASTER BINLOGS
SLAVE
events get
... server-id=1 SQL Thread ...
MyISAM MyISAM
INNODB server-id != 2 INNODB
IO Thread
BINLOGS RELAY LOGS
server-id=1 server-id=2
www.percona.com 26

27. Why server-id ?
APPLICATION
Queries Queries
MASTER server-id != 1
RELAY LOGS BINLOGS
SLAVE events get
server-id=2
SQL Thread IO Thread
log_slave_updates log_slave_updates
events get
... server-id=1 SQL Thread ...
MyISAM MyISAM
INNODB server-id != 2 INNODB
IO Thread
BINLOGS RELAY LOGS
server-id=1 server-id=2
www.percona.com 27

28. Why server-id ?
Avoid events to be written more than once
replicate_same_server_id does what it says
www.percona.com 28

29. Configure Replication
On Master, add permissions:
> GRANT REPLICATION SLAVE ON *.* TO
‘repl’@‘slave’ IDENTIFIED BY ‘pass’;
On Slave, configure replication:
> CHANGE MASTER TO MASTER_HOST=‘master’,
MASTER_USER=‘repl’, MASTER_PASSWORD=‘pass’,
MASTER_LOG_FILE=‘mysql-bin.000008’,
MASTER_LOG_POS=106;
www.percona.com 29

30. slave> START SLAVE;
slave> SHOW SLAVE STATUSG
Slave_IO_State: Waiting for master to send event
IO Thread State
Start Replication
Master_Host: master
Master_User: repl
Master_Port: 3306
Connect_Retry:
Master_Log_File:
60
mysql-bin.000008 IO Thread Read Up To
Read_Master_Log_Pos: 254
Relay_Log_File:
Relay_Log_Pos:
relay-bin.000002
399 SQL Thread Read Up To
Relay_Master_Log_File: mysql-bin.000008
Slave_IO_Running:
Slave_SQL_Running:
Yes
Yes Threads Running?
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 254
Relay_Log_Space: 566
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master:
Master_SSL_Verify_Server_Cert:
0
No
How Far Is Slave Behind?
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0 www.percona.com 30
Last_SQL_Error:

31. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 31

32. Commands
SQL Commands
Adminstrative Commands
Diagnostics Commands
Shell Commands
mysqlbinlog
www.percona.com 32

33. Administrative Commands
Rotate binary log: FLUSH BINARY LOGS
Rotate relay log: FLUSH RELAY LOGS
START/STOP SLAVE IO_THREAD/SQL_THREAD
Remove binary logs:
PURGE MASTER LOGS TO ‘mysql-bin.000005’;
PURGE MASTER LOGS
BEFORE ‘2012-01-01 00:00:00’;
Remove all binary logs: RESET MASTER
Remove slave configuration and files: RESET SLAVE
www.percona.com 33

34. Diagnostics Commands
On Master
SHOW MASTER STATUS
SHOW PROCESSLIST
SHOW SLAVE HOSTS
SHOW BINLOG EVENTS
On Slave
SHOW SLAVE STATUS
SHOW PROCESSLIST
www.percona.com 34

35. On Master:SHOW MASTER STATUS
Current binary log file and position:
master> SHOW MASTER STATUSG
*************************** 1. row ***************************
File: mysql-bin.000008
Position: 254
Binlog_Do_DB:
Binlog_Ignore_DB: 1 row in set (0.00 sec)
www.percona.com 35

36. On Master: SHOW PROCESSLIST
Find Connected Slaves using SHOW PROCESSLIST:
master> SHOW PROCESSLISTG
...
*************************** 2. row ***************************
Id: 4
User: repl
Host: localhost:43537
db: NULL
Command: Binlog Dump
Time: 1264
State: Has sent all binlog to slave; waiting for binlog to be
updated
Info: NULL
...
www.percona.com 36

37. On Master: SHOW SLAVE HOSTS
Shows connected slaves with their server-id:
master> SHOW SLAVE HOSTS;
+-----------+-----------+------+-----------+
| Server_id | Host | Port | Master_id |
+-----------+-----------+------+-----------+
| 2 | slave | 3306 | 1 |
+-----------+-----------+------+-----------+
1 row in set (0.00 sec)
Only works when slaves have configured:
report-host, report-user, report-password,
report-port
www.percona.com 37

38. On Slave: SHOW PROCESSLIST
Slave Thread Status:
SLAVE> SHOW PROCESSLISTG
*************************** 1. row ***************************
Id: 5
User: system user
Host:
db: NULL
Command: Connect
Time: 88611
State: Waiting for master to send event
Info: NULL
*************************** 2. row ***************************
Id: 8
User: system user
Host:
db: NULL
Command: Connect
Time: 83
State: Has read all relay log; waiting for the slave I/O thread to update it
Info: NULL
www.percona.com 38

39. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 39

40. Other Common Configuration Options
Filtering: binlog-%, replicate-%
Don’t start replication at start: skip_slave_start
Put relay log events in it’s own binary log:
log_slave_updates
Disallow writing on slaves: read_only
Automatically remove binlogs: expire_logs_days
Change binlog format: binlog_format
www.percona.com 40

41. Filtering on Master
binlog-do-db=..., binlog-ignore-db=...
Warning! Different behavior between SBR and RBR:
SBR: Log all statements executed if the ‘default database’ (use
database) should be binlogged
RBR: Log all row changes of the databases that should be
binlogged
www.percona.com 41

42. Filtering on Slave
replicate-do-db= & replicate-ignore-db=
Warning! Different behavior between SBR and RBR
Similar to binlog-do/ignore-db
replicate-do/ignore-table: Filter on table level
replicate-wild-do/ignore-table: Use wildcards
(%,_) to filter on table level
replicate-rewrite-db=db1->db2
www.percona.com 42

43. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 43

44. Replication Architectures
Master-Slave
Master-Master
Multi Tiered Replication
Circular Replication
www.percona.com 44

45. Master-Slave
Master
Slave
www.percona.com 45

46. Master-Slave
Application
Master
Slave
www.percona.com 45

47. Master-Slave
Application
Master
Slave
www.percona.com 45

48. Master-Slave
Application
Writes/Reads
Master
Slave
www.percona.com 45

49. Master-Slave
Application
Writes/Reads
Master
Slave
www.percona.com 45

50. Master-Slave
Application
Writes/Reads
Master
Reads
Slave
www.percona.com 45

51. Master-Slave
Master
Slave Slave Slave
www.percona.com 46

52. Master-Slave
Application
Master
Slave Slave Slave
www.percona.com 46

53. Master-Slave
Application
Master
Slave Slave Slave
www.percona.com 46

54. Master-Slave
Application Writes/Reads
Master
Slave Slave Slave
www.percona.com 46

55. Master-Slave
Application Writes/Reads
Master
Slave Slave Slave
www.percona.com 46

56. Master-Slave
Application Writes/Reads
Master
Reads
Slave Slave Slave
www.percona.com 46

57. Master-Slave
Application Writes/Reads
Master
Reads
Slave Slave Slave
www.percona.com 46

58. Master-Slave
Application Writes/Reads
Master
Reads
Slave Slave Slave
www.percona.com 46

59. Master-Slave
Scaling Reads:
Slaves can take reads (remember asynchronous!)
Reporting/Search queries can go to separate slave
Does not take resources on master ‘live-site’-slave.
(‘live site’: lower response time requirements than reports)
Can have other indexes, even other tables (summary tables,...)
Not much benefit when too much writes
Maintenance (schema changes...)
Upgrade slaves first:
Take slave offline
SET SQL_LOG_BIN=0; <-don’t put this session queries in
ALTER TABLE ...; the binary log
SET SQL_LOG_BIN=1;
www.percona.com 47

60. Master-Slave: Reporting Queries
Master
Slave Slave Slave
www.percona.com 48

61. Master-Slave: Reporting Queries
Application
Master
Slave Slave Slave
www.percona.com 48

62. Master-Slave: Reporting Queries
Application
Master
Slave Slave Slave
www.percona.com 48

63. Master-Slave: Reporting Queries
Application Writes/Reads
Master
Slave Slave Slave
www.percona.com 48

64. Master-Slave: Reporting Queries
Application Writes/Reads
Master
Reads
Slave Slave Slave
www.percona.com 48

65. Master-Slave: Reporting Queries
Application Writes/Reads
Master
Reads
Slave Slave Slave
www.percona.com 48

66. Master-Slave: Reporting Queries
Application Writes/Reads
Master
Reads
Slave Slave Slave
www.percona.com 48

67. Master-Slave: Reporting Queries
Application Writes/Reads
Reporting
Master
Reads
Slave Slave Slave
www.percona.com 48

68. Master-Slave: Reporting Queries
Application Writes/Reads
Reporting
Master
Reads
Slave Slave Slave
www.percona.com 48

69. Master-Slave: Write Intensive
Master Slaves Master Slaves
Capacity left for READS Capacity left for READS
Capacity taken by WRITES Capacity taken by WRITES
www.percona.com 49

70. Master-Master
Master Master
www.percona.com 50

71. Master-Master
Used for:
High Availability
Maintenance Tasks
Write to 1 master, failover to the passive master when
necessary
Passive master is a real slave, can be used for reads
Scale Writes? NO!
Every write still has to be performed on both masters
Writing to both masters at the same time? NO*!
Can cause inconsistencies
even with auto_increment_increment/
auto_increment_offset problems can happen
www.percona.com 51

72. Master-Master: Reads & Writes
Application
Writes/Reads
Reads
Master Master
www.percona.com 52

73. Master-Master: Maintenance
Application
Writes/Reads
Master Master
www.percona.com 53

74. Master-Master: Maintenance
Application
Writes/Reads
Master Master
SET SQL_LOG_BIN=0;
ALTER TABLE ...;
...
SET SQL_LOG_BIN=1;
www.percona.com 53

75. Master-Master: Maintenance
Application
Writes/Reads
Master Master
www.percona.com 53

76. Master-Master: Maintenance
Application
Writes/Reads
Master Master
SET SQL_LOG_BIN=0;
ALTER TABLE ...;
...
SET SQL_LOG_BIN=1;
www.percona.com 53

77. Master-Master: Write to Both?
Inconsistencies
Application
Master Master
+--------+----------------+ +--------+----------------+
| userid | email | | userid | email |
+--------+----------------+ +--------+----------------+
| 1 | kg@percona.com | | 1 | kg@percona.com |
+--------+----------------+ +--------+----------------+
CREATE TABLE users (
userid INT AUTO_INCREMENT,
email VARCHAR(128) NOT NULL,
PRIMARY KEY (userid),
UNIQUE KEY idx_email (email)
);
www.percona.com 54

78. Master-Master: Write to Both?
Inconsistencies
Application UPDATE users SET
email=‘kenny@percona.com’
UPDATE users SET where userid=1;
email=‘gryp@percona.com’
where userid=1;
Master Master
+--------+----------------+ +--------+----------------+
| userid | email | | userid | email |
+--------+----------------+ +--------+----------------+
| 1 | kg@percona.com | | 1 | kg@percona.com |
+--------+----------------+ +--------+----------------+
CREATE TABLE users (
userid INT AUTO_INCREMENT,
email VARCHAR(128) NOT NULL,
PRIMARY KEY (userid),
UNIQUE KEY idx_email (email)
);
www.percona.com 54

79. Master-Master: Write to Both?
Inconsistencies
Application UPDATE users SET
email=‘kenny@percona.com’
UPDATE users SET where userid=1;
email=‘gryp@percona.com’
where userid=1;
Master Master
+--------+------------------+ +--------+-------------------+
| userid | email | | userid | email |
+--------+------------------+ +--------+-------------------+
| 1 | gryp@percona.com | | 1 | kenny@percona.com |
+--------+------------------+ +--------+-------------------+
CREATE TABLE users (
userid INT AUTO_INCREMENT,
email VARCHAR(128) NOT NULL,
PRIMARY KEY (userid),
UNIQUE KEY idx_email (email)
);
www.percona.com 54

80. Master-Master: Write to Both?
Inconsistencies
Application
UPDATE users SET email=‘kenny@percona.com’ where userid=1;
Master Master
UPDATE users SET email=‘gryp@percona.com’ where userid=1;
+--------+------------------+ +--------+-------------------+
| userid | email | | userid | email |
+--------+------------------+ +--------+-------------------+
| 1 | gryp@percona.com | | 1 | kenny@percona.com |
+--------+------------------+ +--------+-------------------+
CREATE TABLE users (
userid INT AUTO_INCREMENT,
email VARCHAR(128) NOT NULL,
PRIMARY KEY (userid),
UNIQUE KEY idx_email (email)
);
www.percona.com 54

81. Master-Master: Write to Both?
Inconsistencies
Application
UPDATE users SET email=‘kenny@percona.com’ where userid=1;
Master Master
UPDATE users SET email=‘gryp@percona.com’ where userid=1;
+--------+-------------------+ +--------+------------------+
| userid | email | | userid | email |
+--------+-------------------+ +--------+------------------+
| 1 | kenny@percona.com | | 1 | gryp@percona.com |
+--------+-------------------+ +--------+------------------+
CREATE TABLE users (
userid INT AUTO_INCREMENT,
email VARCHAR(128) NOT NULL,
PRIMARY KEY (userid),
UNIQUE KEY idx_email (email)
);
www.percona.com 54

82. Master-Master: Write to Both?
Replication Breaks
Application
Master Master
+--------+----------------+ +--------+----------------+
| userid | email | | userid | email |
+--------+----------------+ +--------+----------------+
| 1 | kg@percona.com | | 1 | kg@percona.com |
+--------+----------------+ +--------+----------------+
CREATE TABLE users (
userid INT AUTO_INCREMENT,
email VARCHAR(128) NOT NULL,
PRIMARY KEY (userid),
UNIQUE KEY idx_email (email)
);
www.percona.com 55

83. Master-Master: Write to Both?
Replication Breaks
Application UPDATE USERS SET
email=‘legryp@percona.com’
INSERT INTO USERS WHERE userid=1;
(email) VALUES
(‘legryp@percona.com’)
Master Master
+--------+----------------+ +--------+----------------+
| userid | email | | userid | email |
+--------+----------------+ +--------+----------------+
| 1 | kg@percona.com | | 1 | kg@percona.com |
+--------+----------------+ +--------+----------------+
CREATE TABLE users (
userid INT AUTO_INCREMENT,
email VARCHAR(128) NOT NULL,
PRIMARY KEY (userid),
UNIQUE KEY idx_email (email)
);
www.percona.com 55

84. Master-Master: Write to Both?
Replication Breaks
Application UPDATE USERS SET
email=‘legryp@percona.com’
INSERT INTO USERS WHERE userid=1;
(email) VALUES
(‘legryp@percona.com’)
Master Master
+--------+--------------------+ +--------+--------------------+
| userid | email | | userid | email |
+--------+--------------------+ +--------+--------------------+
| 1 | kg@percona.com | | 1 | legryp@percona.com |
| 2 | legryp@percona.com | +--------+--------------------+
+--------+--------------------+
CREATE TABLE users (
userid INT AUTO_INCREMENT,
email VARCHAR(128) NOT NULL,
PRIMARY KEY (userid),
UNIQUE KEY idx_email (email)
);
www.percona.com 55

85. Master-Master: Write to Both?
Replication Breaks
Application
UPDATE USERS SET email=‘legryp@percona.com’ WHERE userid=1;
Master Master
INSERT INTO USERS (email) VALUES (‘legryp@percona.com’)
+--------+--------------------+ +--------+--------------------+
| userid | email | | userid | email |
+--------+--------------------+ +--------+--------------------+
| 1 | kg@percona.com | | 1 | legryp@percona.com |
| 2 | legryp@percona.com | +--------+--------------------+
+--------+--------------------+
CREATE TABLE users (
userid INT AUTO_INCREMENT,
Error 'Duplicate entry 'legryp@percona.com'
email VARCHAR(128) NOT NULL,
PRIMARY KEY (userid), for key 'idx_email'' on query. Default
UNIQUE KEY idx_email (email) database: 'test'. Query: 'insert into
); users(email) values ('legryp@percona.com')'
www.percona.com 55

86. Multi Tiered Replication
Master
Slave Slave Slave
Slave Slave
www.percona.com 56

87. Multi Tiered Replication
2 Second query takes 6 seconds to reach the 3rd level
Top level slaves could be a master for some dbs/tables too
Replication main data (users...) to top level slaves
Top level slaves are sort of functionally partitioned
www.percona.com 57

88. Multi Tiered Replication:
Delay
Master
Slave Slave Slave
Slave Slave
www.percona.com 58

89. Multi Tiered Replication:
Delay
Master 2 Seconds
Slave Slave Slave
Slave Slave
www.percona.com 58

90. Multi Tiered Replication:
Delay
Master 2 Seconds
2 Seconds
Slave Slave Slave
Slave Slave
www.percona.com 58

91. Multi Tiered Replication:
Delay
Master 2 Seconds
2 Seconds
Slave Slave Slave
2 Seconds
Slave Slave
www.percona.com 58

92. Multi Tiered Replication:
Top Level Slaves
MASTER
Master
MASTER SEARCH LOG
Slave Slave Slave
SEARCH SEARCH
Slave Slave
www.percona.com 59

93. Multi Tiered Replication
Master
Slave Slave Slave
Slave Slave
No Updates Anymore
www.percona.com 60

94. Circular Replication
Master
Master Master
Master
www.percona.com 61

95. Circular Replication
Master
Master Master
Master
www.percona.com 62

96. Circular Replication
Master
Master Master
Master
www.percona.com 63

97. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 64

98. Common Issues
Replication Lag
Replication Breaking
Crash Safe Replication
Consistency
www.percona.com 65

99. Replication Lag - Causes
Single threaded execution on slave!!!
Long running queries with SBR: will take ~same time on
slave
Many row changes with RBR
MyISAM & table level locking: Long running SELECT on
slave might block replication
Schema changes:
20min on master,
20min on slave <-- 20 min replication lag
www.percona.com 66

100. Replication Lag - Tips 1
Optimize DML Statements
Avoid DDL Statements in Replication (SQL_LOG_BIN=0)
Hardware specifications usually need to be equal or better
on slave
MySQL 5.6:
http://d2-systems.blogspot.com/2011/04/mysql-56x-feature-preview-multi.html
Ensure BBU&Write Back Cache is used or lower durability:
innodb_flush_log_at_trx_commit=2
Change architecture: functional/horizontal partitioning
www.percona.com 67

101. Replication Lag - Tips 2
Try to avoid SQL Thread to need to do disk I/O:
Replication Booster (Yoshinori Matsunobu)
prefetches relay logs and converts queries to select and runs them,
causing data to be cached before SQL thread reaches
innodb_fake_changes
http://www.percona.com/doc/percona-server/5.5/management/
innodb_fake_changes.html
http://dom.as/2011/12/03/replication-prefetching/
www.percona.com 68

102. Replication Booster
http://yoshinorimatsunobu.blogspot.com/2011/10/making-slave-pre-fetching-work-better.html
www.percona.com 69

103. Replication Breaking - Examples
Last_Errno: 1146
Last_Error: Error 'Table 'db.table' doesn't exist' on query.
Default database: 'userdb'. Query: '...'
Last_Errno: 1062
Last_Error: Error 'Duplicate entry '1748099' for key 1' on query.
Default database: 'db'. Query: 'INSERT INTO table (url)
VALUES(‘http://www.google.com')'
Last_Errno: 1053
Last_Error: Query partially completed on the master (error on
master: 1053) and was aborted. There is a chance that your master
is inconsistent at this point. If you are sure that your master is
ok, run this query manually on the slave and then restart the
slave with SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE; .
Query: ALTER TABLE ‘users’ ADD ‘name’ VARCHAR(128) NULL DEFAULT;
www.percona.com 70

104. Replication Breaking - Causes
Writes happening on Slaves
SBR with bad statements
Crashes: Relay Log Corruption
Different schemas
Temporary tables
Mixing MyISAM with InnoDB
Different behavior between MySQL versions (minor/major)
Killing queries that change MyISAM tables will be partially
executed, and break replication
...
www.percona.com 71

105. Replication Breaking - Fixing
Questions to ask yourself: Why? Where? Impact?
Quick ‘fix’:
SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;
Causes 1 event to be skipped
Causes inconsistencies!!!
Look at:
Check error & Investigate data on master/slave
MySQL errorlogs
SHOW SLAVE STATUSG
mysqlbinlog on relaylogs/binarylogs and investigate
pt-slave-restart
More On: http://www.percona.com/files/presentations/percona-live/london-2011/
PLUK2011-diagnosing-and-fixing-mysql-replication.pdf
www.percona.com 72

106. Crash Safe Replication
On master: sync_binlog=1 to fsync at every binlog write
Performance impact, mainly on:
Ext3 filesystem
http://www.mysqlperformanceblog.com/2009/01/21/beware-ext3-and-sync-binlog-do-not-play-well-together/
High write-response time disksubsystem (no WriteBackCache& BBU)
On slave:
relay-log.info/master.info: not durable/consistent!
When OS crash, old information might be in the file
If InnoDB only: use innodb_overwrite_relay_log_info
in Percona Server:
http://www.percona.com/doc/percona-server/5.5/reliability/
innodb_recovery_update_relay_log.html
www.percona.com 73

107. Consistency
Replication can cause inconsistencies:
Statement Based Replication
Certain functions like UUID()...
UPDATE/DELETE with LIMIT without ORDER BY unique key
Writing to multiple masters
Writing to slaves (by accident? use read_only)
Failover done wrong
MyISAM with killing queries and/ord MySQL Crashes
Wrongly restored slaves
Replication broke and used SQL_SLAVE_SKIP_COUNTER
...
How to know if my data is consistent?
www.percona.com 74

108. Consistency
Checking Consistency: pt-table-checksum
Checksums chunks of data on master and slaves
Monitors replication lag
Low checksum response time to lower impact on production
Uses Replication to ensure consistent view of data
Fixing Consistency: pt-table-sync
Can checksum and use pt-table-checksum results
Fixes inconsistencies
www.percona.com 75

109. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 76

110. Replication Use Cases
Scale Reads
High Availability
Backups
www.percona.com 77

111. High Availability
Failover:
Virtual IPs
Change IP in application servers
Warning: Split-Brain possible!!!
Ensure no writes happen on old master before moving
Make sure new master is in sync
Scripts available to automate failover:
MySQL-MHA http://code.google.com/p/mysql-master-ha/
MMM (not-recommended) http://mysql-mmm.org/
Percona-PRM
http://www.mysqlperformanceblog.com/2011/11/29/percona-replication-manager-a-
solution-for-mysql-high-availability-with-replication-using-pacemaker/
www.percona.com 78

112. High Availability
Due to asynchronous’ness, data might be lost:
Semi-Synchronous Replication:
ensure at least one slave’s IO thread has received the event
before the transaction is committed.
http://dev.mysql.com/doc/refman/5.5/en/replication-semisync.html
www.percona.com 79

113. Backups
Point In Time Recovery, using binary logs
Delayed slave
Backup Slaves
www.percona.com 80

114. Incremental / Point In Time Recovery
Restore Full Backup from your favorite backup tool
Replay binary logs with mysqldump from the last position
of the backup
# mysqlbinlog --start-position=15115 1sbinarylog | mysql
# mysqlbinlog next-binary-logs | mysql
If PITR, stop at the desired position/time:
# mysqlbinlog --stop-position=102151 binarylog | mysql
# mysqlbinlog --stop-datetime=“2012-01-01 23:00:00”
lastbinarylog | mysql
www.percona.com 81

115. Delayed Slave
Deliberately lag a slave for a predefined amount of time
Decrease recovery time in case of bad changes caused by:
Humans (DROP DATABASE)
Application (DELETE FROM table;)
How to configure:
pt-slave-delay --delay 60m
(http://www.percona.com/doc/percona-toolkit/2.0/pt-slave-delay.html)
5.6 Built-in feature:
> CHANGE MASTER TO MASTER_DELAY=3600;
When a problem happens stop replication just before the
malicious statement.
START SLAVE UNTIL MASTER_LOG_FILE='log_name',
MASTER_LOG_POS = pos;
www.percona.com 82

116. Backup using Slaves
Online Backups might not be possible on active master:
Too much impact to make online:
InnoDB backup
(using MySQL enterprise backup or XtraBackup)
LVM Snapshots MyISAM tables
Offsite Backup
www.percona.com 83

117. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 84

118. Other Replication Implementations
Tungsten Replicator
Uses binary log
Parallel replication
Enhanced filtering
Multi-master
Replication between different databases
Galera Cluster
Does not use binary log
Synchronous replication
Active-active master
Parallel replication
Percona XtraDB Cluster! http://www.mysqlperformanceblog.com/2012/01/09/
announcement-of-percona-xtradb-cluster-alpha-release/
www.percona.com 85

119. Tungsten Replicator
http://continuent.com/
http://code.google.com/p/tungsten-replicator/
www.percona.com 86

120. Galera Cluster
http://codership.com/
www.percona.com 87

121. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 88

122. Tools & More Info
Percona Toolkit http://www.percona.com/software/percona-toolkit/
pt-slave-find, pt-table-checksum, pt-table-sync,
pt-slave-delay, pt-slave-restart, pt-heartbeat
Percona Server http://www.percona.com/software/percona-server/
OpenArk kit http://code.openark.org/forge/openark-kit
Replication Booster https://github.com/yoshinorim/replication-booster-for-mysql
High Availability
MMM http://mysql-mmm.org/
Percona-PRM
http://www.mysqlperformanceblog.com/2011/11/29/percona-replication-manager-a-
solution-for-mysql-high-availability-with-replication-using-pacemaker/
MySQL-MHA http://code.google.com/p/mysql-master-ha/
High Performance MySQL, 2nd Edition:
http://shop.oreilly.com/product/9780596101718.do
www.percona.com 89

123. MySQL Replication
Replication Overview
Binary Logs
Setting Up Replication
Commands
Other Common Configuration Settings
Replication Architectures
Common Issues
Replication Use Cases
Other Replication Implementations
Tools
www.percona.com 90

124. Kenny Gryp
<kenny.gryp@percona.com>
@gryp
We're Hiring!
www.percona.com/about-us/careers/