2. about me - Giuseppe Maxia
• a.k.a. The Data Charmer
• QA Director at Continuent, Inc
• Long time hacking with MySQL features
• Formerly, community manager,db consultant, designer,
coder.
• A passion for QA and open source
• Blogger
• http://datacharmer.blogspot.com
Tuesday, April 10, 12 2
3. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
3
Tuesday, April 10, 12 3
4. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
4
Tuesday, April 10, 12 4
5. www
$$$
The world today is
dominated by the
web economy
5
Tuesday, April 10, 12 5
6. www
$$$
Databases are
the backbone
of the web
economy
6
Tuesday, April 10, 12 6
14. www
$$$
More precisely, MySQL
REPLICATION is the
backbone of the web
economy
10
Tuesday, April 10, 12 10
15. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
11
Tuesday, April 10, 12 11
16. database server
a simple web
application
scheme
r/w requests
web server
clients
12
Tuesday, April 10, 12 12
17. database server
scaling web
requests
r/w requests
web servers
load balancer
clients
13
Tuesday, April 10, 12 13
18. database load
on a simple
web
application
r
e
write a
d
85% 15%
14
Tuesday, April 10, 12 14
19. write read
20% 80%
database load on a
successful web
application
15
Tuesday, April 10, 12 15
20. database server
scaling up
means buying
a bigger
✘ r/w requests
database
server
web servers
load balancer
clients
16
Tuesday, April 10, 12 16
21. write read
20% 80%
the bigger database
server will eventually
have the same problem
17
Tuesday, April 10, 12 17
22. read/write
master a web
application
read/only
slaves
scheme with
replication
load balancer
R/W
R/O
web servers
load balancer
clients
18
Tuesday, April 10, 12 18
23. r
e
write a read
d
85% 15% 100%
read/write
master
read/only
slaves
database load
with
replication
19
Tuesday, April 10, 12 19
24. r
e
write a read
d
85% 15% 100%
read/write
master
read/only
slaves
scaling
database load
with
replication
20
Tuesday, April 10, 12 20
25. Replication assessment
without replication with replication
database handling easy harder
performance high lower (binary logs)
Point in Time recovery none easy
failover none possible
write scaling none minimal
backup with downtime without downtime
read scaling none easy
21
Tuesday, April 10, 12 21
26. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
22
Tuesday, April 10, 12 22
28. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
24
Tuesday, April 10, 12 24
29. 1 SHUT DOWN THE DATABASE SERVER
Master
25
Tuesday, April 10, 12 25
30. 2 MAKE A BACKUP COPY
Master
26
Tuesday, April 10, 12 26
31. 3 ENABLE THE MASTER
Master
Configuration file
[mysqld]
log-bin=mysql-bin
server-id=1
27
Tuesday, April 10, 12 27
32. 4 RESTART THE MASTER
Master
28
Tuesday, April 10, 12 28
33. 5 CREATE REPLICATION USER
Master
SQL command
GRANT REPLICATION SLAVE ON *.*
to 'slave_user@'10.10.100.%'
IDENTIFIED BY 'slave_pass';
29
Tuesday, April 10, 12 29
34. 6 INSTALL MySQL on the slave
Slave 1
Make sure that:
• You're using the same version of MySQL
• You have the same directory structure
• The server is not started yet
30
Tuesday, April 10, 12 30
35. 7 COPY THE MASTER DATA to the slave
Slave 1
31
Tuesday, April 10, 12 31
37. 9 START THE SLAVE SERVER
Slave 1
33
Tuesday, April 10, 12 33
38. 10 INITIALIZE THE SLAVE
Slave 1
SQL command
SET MASTER TO
MASTER_HOST=master_IP,
MASTER_PORT=3306,
MASTER_USER=slave_user,
MASTER_PASSWORD='slave_pwd';
34
Tuesday, April 10, 12 34
39. 11 START THE SLAVE SERVICE
Slave 1
SQL command
START SLAVE;
35
Tuesday, April 10, 12 35
40. 12 CHECK THE SLAVE
Slave 1
SQL command
SHOW SLAVE STATUS G
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
36
Tuesday, April 10, 12 36
41. Troubleshooting
• SHOW SLAVE STATUS says
SLAVE_IO_RUNNING=No
• Make sure that the slave host can connect
to the master
• Make sure that master and slave have
different Server-id
• Check the error log of both master and
slave
37
Tuesday, April 10, 12 37
42. Testing the slave
• Create a table in the master.
• Make sure that the slave has replicated the
table.
• Insert data in the master
• read that data in the slave
38
Tuesday, April 10, 12 38
43. What if the master was already logging?
• You have two options:
• Physical copy
• Logical copy
39
Tuesday, April 10, 12 39
44. Physical copy
• stop master
• make copy
• remove binary log files and index
• start master
• (alternative: use xtrabackup)
40
Tuesday, April 10, 12 40
46. Common replication commands
• CHANGE MASTER TO
• SHOW MASTER STATUS
• SHOW SLAVE STATUS
• START SLAVE
• STOP SLAVE
• RESET MASTER (caution!)
• RESET SLAVE (caution!)
42
Tuesday, April 10, 12 42
47. CHANGE MASTER TO basic syntax
STOP SLAVE;
CHANGE MASTER TO
MASTER_HOST='hostname', # or IP
MASTER_PORT=3306,
MASTER_USER='slaveuser',
MASTER_PASSWORD='slavepassword',
MASTER_LOG_FILE='filename', # default: first binlog
MASTER_LOG_POS=123456; # default: from the beginning
START SLAVE;
Tuesday, April 10, 12 43
48. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
44
Tuesday, April 10, 12 44
49. 1 NO NEED TO STOP THE MASTER!
Master
45
Tuesday, April 10, 12 45
50. 2 STOP THE SLAVE
Slave 1
SQL command
STOP SLAVE IO_THREAD;
# wait until the SQL_THREAD
# has done everything
STOP SLAVE SQL_THREAD;
# STOP THE SERVER
46
Tuesday, April 10, 12 46
51. 3 MAKE A COPY OF THE DATA DIRECTORY
Slave 1
47
Tuesday, April 10, 12 47
52. 4 RESTART THE SLAVE
Slave 1
48
Tuesday, April 10, 12 48
53. 5 INSTALL MySQL on the new slave
Slave 2
Make sure that:
• You're using the same version of MySQL
• You have the same directory structure
• The server is not started yet
49
Tuesday, April 10, 12 49
54. 6 COPY THE old slave DATA on the slave
Slave 2
50
Tuesday, April 10, 12 50
55. 7 ENABLE THE NEW SLAVE
Slave 2
Configuration file
uni que!
[mysqld]
mus t be
server-id=3
relay-log=mysql-relay
read-only
# optional:
log-bin=mysql-bin 51
Tuesday, April 10, 12 51
56. 8 START THE NEW SLAVE
Slave 2
52
Tuesday, April 10, 12 52
57. 9 CHECK THE SLAVE
Slave 2
SQL command
SHOW SLAVE STATUS G
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
53
Tuesday, April 10, 12 53
58. Why it works
• No need to issue a CHANGE MASTER TO
command.
• Because we cloned the old slave
• The new slave gets its parameters from
the .info files in the data directory
54
Tuesday, April 10, 12 54
67. DA
EN Hijacking the agenda
AG
• MySQL Sandbox
56
Tuesday, April 10, 12 56
68. MySQL Sandbox
http://mysqlsandbox.net
• Free software (Perl under GPL)
• One (unix) host
• Many database servers
• Single or multiple sandboxes
• Customized scripts to use the servers
• Standard or circular replication
• Installs IN SECONDS
Tuesday, April 10, 12 57
69. overview
MySQL MySQL
server server
Data DB1 Data DB1
DB2 DB3
DATA DIRECTORY DB2 DB3
PORT
SOCKET
Tuesday, April 10, 12 58
70. overview
MySQL MySQL
server server
Data DB1
SAME Data DB1
DB2 DB3
DATA DB2 DB3
DIRECTORY?
/var/lib/mysql /var/lib/mysql
DATA CORRUPTION
Tuesday, April 10, 12 59
71. overview
MySQL MySQL
server server
SAME
PORT or
SOCKET?
3306 3306
/tmp/mysql.sock /tmp/mysql.sock
DOES NOT START
Tuesday, April 10, 12 60
73. The hard way
Read the manual
Tuesday, April 10, 12 61
74. The hard way
try to figure out
Read the manual
what to change
Tuesday, April 10, 12 61
75. The hard way
try to figure out
Read the manual Install
what to change
Tuesday, April 10, 12 61
76. The easy way
MySQL Sandbox
$ make_sandbox
/path/to/mysql-5.1.54_linux.tar.gz
# it should work always
Tuesday, April 10, 12 62
77. The easier way
Prepare once Install many times
# some $ make_sandbox 5.1.54
# preliminary
# work
Tuesday, April 10, 12 63
78. The easiest way
Prepare once Install many times
# some
# preliminary $ sb 5.1.54
# work
Tuesday, April 10, 12 64
79. MySQL Sandbox
VERSION
MySQL
server
Data DB1
DB2 DB3
$SANDBOX_HOME/msb_VERSION/data
VERSION
/tmp/mysql_VERSION.sock
Tuesday, April 10, 12 65
80. MySQL Sandbox
5.1.54
MySQL
server
Data DB1
DB2 DB3
$SANDBOX_HOME/msb_5_1_54/data
5154
/tmp/mysql_5154.sock
Tuesday, April 10, 12 66
81. MySQL Sandbox
5.5.9
MySQL
server
Data DB1
DB2 DB3
$SANDBOX_HOME/msb_5_5_09/data
5509
/tmp/mysql_5509.sock
Tuesday, April 10, 12 67
82. Single Sandbox
MySQL customized scripts
server
start
stop
restart
status
clear
send_kill
use
Tuesday, April 10, 12 68
83. Multiple Sandbox
MySQL customized scripts
server
start_all
stop_all
restart_all m n1
status_all s1 n2
clear_all s2 n3
send_kill_a
ll
use_all
Tuesday, April 10, 12 69
84. Where do you get it
•from CPAN
sudo su -
cpan MySQL::Sandbox
•from launchpad
http://launchpad.net/mysql-sandbox
Tuesday, April 10, 12 70
85. The easy replication way
MySQL Sandbox
$ make_replication_sandbox
/path/to/mysql-5.1.54_linux.tar.gz
Prepare once Install many times
# some $ make_replication_sandbox
# preparation 5.1.54
Tuesday, April 10, 12 71
86. default architecture
$HOME
/sandboxes opt
expanded
tarballs
$SANDBOX_HOME
mysql
$SANDBOX_BINARY
installed
sandboxes
Tuesday, April 10, 12 72
89. creating a single sanbox
make_sandbox
/path/to/mysql-X.X.XX-OS.tar.gz
Tuesday, April 10, 12 75
90. using a single sanbox
# after
# make_sandbox
# /path/to/mysql-X.X.XX-OS.tar.gz
$ cd $SANDBOX_HOME/msb_X_X_XX
$ ./use
Tuesday, April 10, 12 76
91. creating a single sanbox
with a specific options file
make_sandbox
/path/to/mysql-X.X.XX-OS.tar.gz
--my_file=/path/to/my.cnf
Tuesday, April 10, 12 77
92. easily create a sandbox after the first
one
The long way
$ cd $HOME/opt/mysql # $SANDBOX_BINARY
$ gunzip -c
/path/to/mysql-5.1.34-osx10.5-x86.tar.gz
| tar -xf -
$ mv mysql-5.1.34-osx10.5-x86 5.1.34
$ make sandbox 5.1.34
Tuesday, April 10, 12 78
93. easily create a sandbox after the first
one
The short way
$ make_sandbox
path/to/mysql-5.1.34-osx10.5-x86.tar.gz
--export_binaries
Tuesday, April 10, 12 79
94. starting a single sanbox
$ cd $SANDBOX_HOME/msb_X_X_XX
$ ./start
Tuesday, April 10, 12 80
95. starting a single sanbox
with temporary options
$ cd $SANDBOX_HOME/msb_X_X_XX
$ ./start --option=value
$ ./restart --option=value
$ ./start --key-buffer=20000000
Tuesday, April 10, 12 81
96. creating a sandbox with custom port
and directory
$ make_sandbox 5.1.34
--sandbox_port=7800
--sandbox_directory=mickeymouse
Tuesday, April 10, 12 82
97. creating a sandbox with automatic
port checking
$ make_sandbox 5.1.34 --check_port
# if 5.1.34 is free
# port=5134
# directory=msb_5_1_34
# else
# port=5135 (or the first free)
# directory=msb_5_1_34_a
Tuesday, April 10, 12 83
98. create a replication sandbox
$ make_replication_sandbox
path/to/mysql-5.1.34-osx10.5-x86.tar.gz
Tuesday, April 10, 12 84
99. create a circular replication sandbox
$ make_replication_sandbox
--circular=4
path/to/mysql-5.1.34-osx10.5-x86.tar.gz
Tuesday, April 10, 12 85
100. changing port to an existing sandbox
$ sbtool -o port
-s /path/to/source/sandbox
--new_port=XXXX
Tuesday, April 10, 12 86
101. installing the innodb plugin
$ sbtool -o plugin
--plugin=innodb
-s /path/to/source/sandbox
Tuesday, April 10, 12 87
102. creating a replication sandbox with
new base port
$ make_replication_sandbox
--replication_directory=newwdir
--check_base_port 5.0.79
# Creates a replication directory under
# $SANDBOX_HOME/newdir
# The previous one is preserved.
# No conflicts happen
Tuesday, April 10, 12 88
103. DA
EN How to set up replication
AG
DEMO
89
Tuesday, April 10, 12 89
104. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
90
Tuesday, April 10, 12 90
105. Binary log formats
• Statement based replication
• default
• available since 3.23
• Row based replication
• introduced in 5.1
• ROW or MIXED
91
Tuesday, April 10, 12 91
106. Viewing the binary logs
# statement based replication
$ mysqlbinlog binary-log-name
# or, as a SQL command
SHOW BINLOG EVENTS IN 'binary-log-name';
# row based replication
$ mysqlbinlog --verbose
--base64-output=decode-rows binary-log-name
Tuesday, April 10, 12 92
107. binary log examples
# statement based replication
mysqlbinlog binary-log-name
# row based replication
mysqlbinlog --verbose
--base64-output=decode-rows binary-log-name
Tuesday, April 10, 12 93
108. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
94
Tuesday, April 10, 12 94
109. what gets replicated
What Statement-based row-based
insert/update/delete statement affected records
schema/table/view
statement statement
creation/drop
stored routine/trigger
statement statement
creation/drop
statements executed
stored procedure call affected records
inside SP
stored function call function call affected records
none: the slave runs the affected records
trigger execution
trigger moved. No trigger runs
slaveside disabled slaveside disabled
event creation
event event
event execution statement affected records
95
Tuesday, April 10, 12 95
110. what gets replicated
What Statement-based row-based
insert/update/delete statement affected records
schema/table/view
statement statement
creation/drop
stored routine/trigger
statement statement
creation/drop
statements executed
stored procedure call affected records
inside SP
stored function call function call affected records
none: the slave runs the affected records
trigger execution
trigger moved. No trigger runs
slaveside disabled slaveside disabled
event creation
event event
event execution statement affected records
95
Tuesday, April 10, 12 95
111. what gets replicated
What Statement-based row-based
insert/update/delete statement affected records
schema/table/view
statement statement
creation/drop
stored routine/trigger
statement statement
creation/drop
statements executed
stored procedure call affected records
inside SP
stored function call function call affected records
none: the slave runs the affected records
trigger execution
trigger moved. No trigger runs
slaveside disabled slaveside disabled
event creation
event event
event execution statement affected records
95
Tuesday, April 10, 12 95
112. what gets replicated
What Statement-based row-based
insert/update/delete statement affected records
schema/table/view
statement statement
creation/drop
stored routine/trigger
statement statement
creation/drop
statements executed
stored procedure call affected records
inside SP
stored function call function call affected records
none: the slave runs the affected records
trigger execution
trigger moved. No trigger runs
slaveside disabled slaveside disabled
event creation
event event
event execution statement affected records
95
Tuesday, April 10, 12 95
113. what gets replicated
What Statement-based row-based
insert/update/delete statement affected records
schema/table/view
statement statement
creation/drop
stored routine/trigger
statement statement
creation/drop
statements executed
stored procedure call affected records
inside SP
stored function call function call affected records
none: the slave runs the affected records
trigger execution
trigger moved. No trigger runs
slaveside disabled slaveside disabled
event creation
event event
event execution statement affected records
95
Tuesday, April 10, 12 95
114. what gets replicated
What Statement-based row-based
insert/update/delete statement affected records
schema/table/view
statement statement
creation/drop
stored routine/trigger
statement statement
creation/drop
statements executed
stored procedure call affected records
inside SP
stored function call function call affected records
none: the slave runs the affected records
trigger execution
trigger moved. No trigger runs
slaveside disabled slaveside disabled
event creation
event event
event execution statement affected records
95
Tuesday, April 10, 12 95
115. what gets replicated
What Statement-based row-based
insert/update/delete statement affected records
schema/table/view
statement statement
creation/drop
stored routine/trigger
statement statement
creation/drop
statements executed
stored procedure call affected records
inside SP
stored function call function call affected records
none: the slave runs the affected records
trigger execution
trigger moved. No trigger runs
slaveside disabled slaveside disabled
event creation
event event
event execution statement affected records
95
Tuesday, April 10, 12 95
116. what gets replicated
What Statement-based row-based
insert/update/delete statement affected records
schema/table/view
statement statement
creation/drop
stored routine/trigger
statement statement
creation/drop
statements executed
stored procedure call affected records
inside SP
stored function call function call affected records
none: the slave runs the affected records
trigger execution
trigger moved. No trigger runs
slaveside disabled slaveside disabled
event creation
event event
event execution statement affected records
95
Tuesday, April 10, 12 95
117. what gets replicated
What Statement-based row-based
insert/update/delete statement affected records
schema/table/view
statement statement
creation/drop
stored routine/trigger
statement statement
creation/drop
statements executed
stored procedure call affected records
inside SP
stored function call function call affected records
none: the slave runs the affected records
trigger execution
trigger moved. No trigger runs
slaveside disabled slaveside disabled
event creation
event event
event execution statement affected records
95
Tuesday, April 10, 12 95
118. DA
EN Binary logs in practice
AG
DEMO
96
Tuesday, April 10, 12 96
119. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
97
Tuesday, April 10, 12 97
120. From single server application
r/w requests
98
Tuesday, April 10, 12 98
122. Single server application
$link = mysql_connect(
$server_IP,
'mysql_user',
'mysql_password'
);
$result = mysql_query(
'INSERT INTO table_name (x) VALUES (1)',
$link
);
$result = mysql_query(
'SELECT * FROM table_name WHERE x=1',
$link
);
100
Tuesday, April 10, 12 100
123. Making an application aware of
replication
<<database handling>>
db
<<R/W database
handling>> IP
user
db password
IP connect
user
password
connect <<read-only database <<R/W database
handling>> handling>>
read db db
write
IP IP
user user
password password
read connect
read
write
101
Tuesday, April 10, 12 101
124. Using replication:
the WRONG way
No
Write Yes
statement?
Connect to the Connect to
next available the master
slave
Read from Write to the
slave master
R/W split
by Stop
statement 102
Tuesday, April 10, 12 102
125. Why statement split is wrong
• Breaks transactions
• High risk of inconsistency
• Loses or corrupts data
103
Tuesday, April 10, 12 103
126. Using replication:
the RIGHT way
No
Write Yes
function?
Connect to the Connect to
next available the master
slave
Read from Read and write
slave from master
more Yes more
queries? queries?
Yes
R/W split No No
by function Stop
104
Tuesday, April 10, 12 104
127. Replication is single threaded
• Let’s assume you have two queries on the
master
• one query takes 3 minutes to complete
• the other takes 5 minutes
• Both start at 12:00 noon
105
Tuesday, April 10, 12 105
133. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
108
Tuesday, April 10, 12 108
134. Sample get slave
status
monitoring
slave
Running? No
Yes
master
Get master
binlog and
position
Same or later
No
Yes
binlog/position?
check table
contents
alert 109
Tuesday, April 10, 12 109
135. commands for checking replication
master> SHOW MASTER STATUS
master> SHOW PROCESSLIST
slave> SHOW SLAVE STATUS
slave> SHOW PROCESSLIST
Tuesday, April 10, 12 110
136. replication user seen in the master
master > show processlistG
*************************** 1. row ***************
Id: 2
User: rsandbox
Host: localhost:57011
db: NULL
Command: Binlog Dump
Time: 81625
State: Master has sent all binlog to slave; waiting
for binlog to be updated
Info: NULL
[...]
Tuesday, April 10, 12 111
137. IO-thread in the slave
slave > show processlistG
*************************** 1. row ***************
Id: 2
User: system user
Host:
db: NULL
Command: Connect
Time: 124801
State: Waiting for master to send event
Info: NULL
[...]
Tuesday, April 10, 12 112
138. SQL thread in the slave
slave > show processlistG
[...]
*************************** 2. row ***************
Id: 3
User: system user
Host:
db: NULL
Command: Connect
Time: 124712
State: Slave has read all relay log; waiting for the
slave I/O thread to update it
Info: NULL
[...]
Tuesday, April 10, 12 113
139. monitoring replication
master> SHOW MASTER STATUS
slave> SHOW SLAVE STATUS
FULL SCRIPTS:
http://datacharmer.blogspot.com/2011/04/refactored-again-poor-mans-
mysql.html
http://forge.mysql.com/tools/tool.php?id=6
Tuesday, April 10, 12 114
140. show master status
File: mysql-bin.000002
Position: 78045744
Binlog_Do_DB:
Binlog_Ignore_DB:
Tuesday, April 10, 12 115
141. show slave status
Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: rsandbox
Master_Port: 27371
Connect_Retry: 60
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 78045744
Relay_Log_File: mysql_sandbox27372-relay-bin.000055
Relay_Log_Pos: 78045889
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Tuesday, April 10, 12 116
142. show slave status
...
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
...
Tuesday, April 10, 12 117
143. show slave status
...
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 78045744
Relay_Log_Space: 78046100
...
Seconds_Behind_Master: 0
...
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Tuesday, April 10, 12 118
144. DA
EN Breaking (and fixing) replication
AG
DEMO
119
Tuesday, April 10, 12 119
145. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
120
Tuesday, April 10, 12 120
146. Logs rotation
# server variables
max-binlog-size
expire-log-days
# logs commands
SHOW MASTER LOGS;
PURGE MASTER LOGS TO 'filename';
FLUSH [BINARY] LOGS;
Tuesday, April 10, 12 121
147. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
122
Tuesday, April 10, 12 122
148. Replacing Slave
a slave crashe
s
No
are there Yes
more slaves?
STOP the
master STOP one
slave
add the first add another
slave slave
Stop
123
Tuesday, April 10, 12 123
149. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
124
Tuesday, April 10, 12 124
150. Replacing the Master
master crashe
s
Let all slaves
catch up with
execution STOP
replication in
all slaves
FIND the most make it the
up to date slave master
FIND which run missing
connect all
transactions are transactions
slaves to the
missing from to other
new master
other slaves slaves
Stop
125
Tuesday, April 10, 12 125
151. Planned master switch
• Stop accepting writes
• Wait until all slaves have caught up
• Stop replication in all slaves
• Promote a slave to master
• Point all slaves to the new master
Tuesday, April 10, 12 126
152. Changing a failed master
• Pre-requisite:
• log_bin and log_slave_updates must be
enabled in all the slaves
• If not, there is more manual labor
Tuesday, April 10, 12 127
153. Changing a failed master (1)
• Wait until all slaves have caught up
• Identify the most advanced slave
• Make that slave the new master
• ... so far, so good
Tuesday, April 10, 12 128
154. Changing a failed master (2)
• For each remaining slave:
• Find the missing statements
• Find the LAST statement replicated by the slave
• Find the same statement in the new master
binlog (*)
• get the position of the NEXT statement
(*) if log_slave_updates was not enabled, you
need to convert the relay log statements to SQL
and do the next step manually
Tuesday, April 10, 12 129
155. Changing a failed master (3)
• For each remaining slave:
• Apply the missing statements
• CHANGE MASTER TO
master_host=”new_master_hostname”,
master_port=new_master_port,
master_log_file=”mysql-bin.xxxxxx”,
master_log_pos=YYYY
Tuesday, April 10, 12 130
157. What is a global transaction ID
• A unique identifier of a transaction
• Unique for the whole cluster, not for each node
• Generated by the ultimate source (the master)
• Does not change when the transaction goes
through an intermediate master
Tuesday, April 10, 12 132
158. Why should you care
• Failure recovery
• MySQL DOES NOT have it
• What can you do? Either wait for MySQL 5.6
or use Tungsten Replicator.
Tuesday, April 10, 12 133
159. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
134
Tuesday, April 10, 12 134
160. Slave = master replacement
• if:
• there are no filters;
• you are monitoring replication
• you make sure data is consistent
135
Tuesday, April 10, 12 135
161. Slave != backup
• There is no replacement for a good backup.
• Data loss due to a mistake in the master will
propagate to the slaves.
• Disasters are always smarter than
replication.
• Do your backups!
• Session on Wednesday at 12pm: "Be a Data
Management Hero with Good Backups!"
136
Tuesday, April 10, 12 136
162. DA Why replication
• The web economy
EN • Scaling out
AG
• From single server to replication
How to set replication
• Adding a slave
• Binary log formats
Using Replication • What gets replicated and how
• Replication awareness
• Monitoring
• Log management
Managing replication
• Replacing a slave
• Replacing a master
• master replacement
What Replication is for • backup
• better reads
Gotchas, tips, and tricks
• What to read
More info
• More replication sessions
137
Tuesday, April 10, 12 137
163. Replication
=
let the slave do the dirty work
Tuesday, April 10, 12 138