Easy MySQL Replication Setup and Troubleshooting

Easy MySQL Replication
Setup and Troubleshooting
Bob Burgess
Salesforce ExactTarget Marketing Cloud (Radian6)

• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing

Slides
• I will post these slides on the conference site
and slideshare
• No need to copy them down

Who Am I?
• MySQL user since 2007
• Radian6: Social Media Monitoring
• Now part of
Salesforce ExactTarget Marketing Cloud
• Large Data Set
• Dozens of Databases
• Replication is CRUCIAL for us

Why Am I Doing This?
• Replication isn’t hard
• Show simplest replication case
• Share how we deal with common problems

Our First Replication
• Set up a separate database for running
reports, to separate back-end from user-facing
activity
• Alexander Rubin of MySQL Support in 2008

Why Replication?
• Database Failure
• Share Read Traffic (Scale-Out)
• Reporting/Analytics Databases
• Network Load
• Backups

Replication Compared
• DRBD
– Block-level
– Replica is unusable until switched to
• Galera / Percona Cluster
– More complicated
– Not WAN-Friendly for High Transaction Rate
• Tungsten Replicator
• Others..?

Replication Described
Master Replica

Master
table1
table2
mysql
Update table1 set...
Insert into table2...
Binlogs:
• Events in commit order
• Contains timestamps and auto-increment IDs
client
Replica
table1
table2
IO Thread SQL Thread
mysql

log_bin=/data/log/mysql-bin
Config items
Master
table1
table2
mysql
client
Replica
table1
table2
mysql

server_id=1 server_id=2
Config items
(1) Master
table1
table2
mysql
client
Replica (2)
table1
table2
mysql

Replication Commands
• change master to
• start slave
• stop slave
• reset slave
• show slave statusG
• show master status
• mysqlbinlog
• grant replication slave

change master to...
Which server? Port? Credentials? Log file & position?
(1) Master
table1
table2
mysql
client
Replica (2)
table1
table2
mysql

• change master to...
Which server? Port? Credentials? Log file & position?
• change master to
master_host=‘Master’, master_port=3306,
master_user=‘repl’, master_password=‘pass’,
master_log_file=‘mysql_bin.000001’,
master_log_pos=4563744;

(1) Master
table1
table2
mysql
client
Replica (2)
table1
table2
mysql

change master to
master_host=‘Master’, master_port=3306,
master_user=‘repl’, master_password=‘pass’,
master_log_file=‘mysql_bin.000001’,
master_log_pos=4563744;
(1) Master
table1
table2
mysql
client
Replica (2)
table1
table2
IO Thread
mysql

start slave;
(1) Master
table1
table2
mysql
client
Replica (2)
table1
table2
mysql

stop slave;
(1) Master
table1
table2
mysql
client
Replica (2)
table1
table2
mysql

reset slave;
(1) Master
table1
table2
mysql
client
Replica (2)
table1
table2
mysql

Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1

master to
send event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Relay_Log_Pos: 548
Replicate_Do_DB:
Replicate_Do_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Relay_Log_Space: 52
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1
What it’s doing

master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Relay_Log_Pos: 548
Replicate_Do_DB:
Replicate_Do_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Relay_Log_Space: 52
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1
Where it’s connected

master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Relay_Log_Pos: 548
Replicate_Do_DB:
Replicate_Do_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Relay_Log_Space: 52
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1
Where the IO Thread has read up to

Master
table1
table2
mysql
client
Replica
table1
table2
mysql

master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Relay_Log_Pos: 548
Replicate_Do_DB:
Replicate_Do_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Relay_Log_Space: 52
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1
Where the SQL Thread has executed up to
(in relaylog)

Master
table1
table2
mysql
client
Replica
table1
table2
mysql
Relay_Log_Pos: 548

master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Relay_Log_Pos: 548
Relay_Master_Log_File:
mysql-bin.000001
Replicate_Do_DB:
Replicate_Do_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos:
23456234
Relay_Log_Space: 52
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1
Where the SQL Thread has executed up to
(corresponding position in Master’s binlog)

Master
table1
table2
mysql
client
Replica
table1
table2
mysql

master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Relay_Log_Pos: 548
Replicate_Do_DB:
Replicate_Do_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Relay_Log_Space: 52
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1
If the IO and SQL Threads are running

master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Relay_Log_Pos: 548
Replicate_Do_DB:
Replicate_Do_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Relay_Log_Space: 52
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1
Replication Filters in effect

master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Relay_Log_Pos: 548
Replicate_Do_DB:
Replicate_Do_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Relay_Log_Space: 52
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1
Replication Lag (how long ago was
timestamp of latest-processed query?)
Can be “NULL”

master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Relay_Log_Pos: 548
Replicate_Do_DB:
Replicate_Do_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Relay_Log_Space: 52
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1
Replication Error (SQL Thread)

master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Relay_Log_Pos: 548
Replicate_Do_DB:
Replicate_Do_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Relay_Log_Space: 52
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1
Replication Error (IO Thread)

Basic Setup Walk-Through
• Unique server_id values
my.cnf on master
[mysqld]
server_id=1
my.cnf on replica
[mysqld]
server_id=2

• Replication User
On Master:
grant replication slave on *.* to ‘repl’
identified by ‘pass’;

• Point to the correct binlog position
• On master:
show master status;
...once both databases are equal
• On Replica:
change master to ...

• Start it up
• On replica:
start slave;
• Check it
• On replica:
show slave statusG

Limiting Tables or Schemas
• If you don’t need all the tables
• In my.cnf:
replicate-do-table=schema.tableName
replicate-skip-table=schema.otherTable
• If you don’t need all the schemas
• In my.cnf
replication-do-db=schemaOne
replication_skip_db=schemaTwo

Limiting Tables
replicate-do-table=schema.tableName
(1) Master
table1
table2
mysql
client
Replica (2)
table1
table2
with filter
mysql

Limiting Columns or Rows
• If you don’t need all columns
• Subset of table for reporting or analytics

• Original table
create table x (a int primary key,
b int, c int) engine=InnoDB;
• Blackhole table on replica like master table
create table x (a int, b int, c int)
engine=blackhole;
• Target table on replica
create table y (a int primary key,
b int) engine=InnoDB;

• Before-insert trigger on replica to put data into
second “real” table
• Only columns a and b, and only for b=1
create trigger x_bi before insert on x
for each row
insert into y (a, b)
select new.a, new.b
from dual
where new.b=1;

(1) Master
table1
table2
mysql
client
Replica (2)
x y
mysql
TRG

Multi-Level
• Local and Remote DR Site
• Spreading load over many servers
M R M R
M R
R
R
R
R
R
R
R
...

Multi-Level
• Config
log_slave_updates=1

Multi-Level
Master Replica
table1
table2
table1
table2
mysql IO Thread SQL Thread
Replica
table1
table2
mysql

Types of Binlogs
• Statement-Based
– Contains the actual statements run
– Data drift is more likely (code modified by
triggers, etc)
• Row-based
– Replicates changes for each row
– Mostly unreadable by humans
– Unaffected by how the change is done (data drift less
likely)
– mysqlbinlog -v

Fault: Lag
• Long-running transactions
• High concurrency on master becomes single
session on replica
• Easy to alarm on

Fault: Stopping from Error
• Any error stops SQL Thread!
• Duplicate Key
• Unknown Function
• slave_exec_mode=idempotent
...suppresses duplicate-key and no-key-found
errors

Faults: Can’t Connect to Master
• Fault shows in slave status as soon as you
start slave.
• Did you create replication slave user?
• Correct user/password?
• Correct hostname/port?
• Master is running and reachable?

Fault: Can’t Find Binlog
• Clearly stated in the slave status
• Can happen if replica has been stopped (or is
lagged) and the binlogs are removed from the
master
• Execute from archive
OR
• Rebuild replica and restart from a known good
binlog position

Faults: Configuration
• replication-do-table
• replication-do-schema
• log_slave_updates

Faults: server_id
• Must be unique!
• If equal: “Seconds behind” jumps everywhere.

Gotcha: functions
• Functions will be run on the replica even if
only appearing in a select on the master
• Make sure all functions exist on replica
DB, even if only a dummy version
create function myFtn (a int, b int)
returns int deterministic return 0;

Gotcha: Reconnection
• When a replica is stopped and reset, until the
database is restarted, replication may
reconnect to a master after “reset slave”
change master to master_port=1234;

Gotcha: Master Crash
• Replication becomes unstable if master crashes
• Error: unknown instruction at position xxx
• Often need to use mysqlbinlog to find a
suitable position in the binlog to restart
replication from
stop slave; reset slave;
change master to ...;
start slave;

Gotcha: MySQL Versions
• Replication not completely compatible
between versions (especially later=>earlier)
• 5.5 master to 5.1 replica: character set error
“sometimes”
• Stops replication

Impossible
• One replica can have only one master
(until 5.7)
• Can’t change table or schema filters
dynamically
• Can’t put binlogs back on master server, for
recovery purposes

Alternatives
• Tungsten Replicator from Continuent
• Others?

Thank you!
Bob Burgess
rburgess@salesforce.com
Don’t forget your session evaluation!

Easy MySQL Replication Setup and Troubleshooting

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Easy MySQL Replication Setup and Troubleshooting

Ähnlich wie Easy MySQL Replication Setup and Troubleshooting (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Easy MySQL Replication Setup and Troubleshooting