This new webinar on Managing MySQL Replication for High Availability led by Krzysztof Książek, Senior Support Engineer at Severalnines is part of our ongoing ‘Become a ClusterControl MySQL DBA’ series. Deploying a MySQL Replication topology is only the beginning of your journey. Maintaining it also involves topology changes, managing slave lag, promoting slaves, repairing replication issues, fixing broken nodes, managing schema changes and scheduling backups. Multi-datacenter replication also adds another dimension of complexity. It is always good to be prepared up front and know how to deal with these cases. In this webinar we will cover deployment and management of MySQL replication topologies using ClusterControl, show how to schedule backups, promote slaves and what are the most important metrics to keep a close eye on. We will also cover how you can deal with schema and topology changes and, if time permits, solve the most common replication issues.

1. Copyright 2015 Severalnines AB
Managing MySQL Replication for High Availability
February 2, 2016
Krzysztof Książek
Severalnines
krzysztof@severalnines.com
1

2. 2
Your host & some logistics
I'm Jean-Jérôme from the Severalnines Team
and I'm your host for today's webinar!
Feel free to ask any questions in the
Questions section of this application or via
the Chat box.
You can also contact me directly via the chat
box or via email: jj@severalnines.com
during or after the webinar.

3. Copyright 2015 Severalnines AB
! MySQL replication and ClusterControl (Demo)
! Deployment
! Topology changes
! Important metrics
! How to deal with
! Schema changes
! Topology changes
! Potential issues in MySQL replication
3
Agenda

4. Copyright 2015 Severalnines AB
! Available since 3.23
! Proven, robust, based on binary logs
! GTID introduced in MySQL 5.6
! Multi-threaded replication introduced in 5.6, vastly
improved in 5.7
! Asynchronous or semisynchronous
! Backbone of majority of large MySQL deployments
4
MySQL Replication

5. Copyright 2015 Severalnines AB
MySQL replication and ClusterControl
5

6. Copyright 2015 Severalnines AB
! Deployment process
! Add slave
! Promote slave
! Recover slave
! Monitoring of the replication environment
6
MySQL replication and ClusterControl

7. Copyright 2015 Severalnines AB
MySQL replication and ClusterControl
7
DEMO

8. Copyright 2015 Severalnines AB
Deployment scenarios for replication
8

9. Copyright 2015 Severalnines AB
! Single DC
! Master - slave
! Intermediate master
! Master - master
! Multi-DC
! Master - slave, over WAN
! Active - passive
! WAN replication with Galera or MySQL Cluster
9
Deployment scenarios for replication

10. Copyright 2015 Severalnines AB
Deployment scenarios for replication
10
Master - slave
Intermediate master

11. Copyright 2015 Severalnines AB
Deployment scenarios for replication
11
Master - master, active - standby
Two synchronous clusters
connected over WAN

12. Copyright 2015 Severalnines AB
How to deal with schema changes?
12

13. Copyright 2015 Severalnines AB
! Schema changes in MySQL
! Online DDL’s
! not-so-online DDL’s
! Rolling schema upgrades
! external tools like pt-online-schema-change
13
How to deal with schema changes?

14. Copyright 2015 Severalnines AB
! With MySQL replication lag is a serious limiting factor in
DDL’s
! Online DDL will be online only on the master
! Replication will be locked for the duration of the DDL
! Even when you use Multi-Threaded Slave
! Direct alters are suitable only for small tables
! unless you can accept replication lag, of course
14
How to deal with schema changes?

15. Copyright 2015 Severalnines AB
! Rolling Schema upgrades - run DDL from the bottom to the top
of replication chain
! SET SESSION sql_log_bin=0 to avoid errant transactions
! Once you reach the master, failover and then alter the old
master
! Schema changes have to be compatible, you’ll be running
DML’s on a mix of altered and not altered tables
! Schema changes have to be compatible, you’ll be reading
from a mix of altered and not altered tables
15
How to deal with schema changes?

16. Copyright 2015 Severalnines AB
! pt-online-schema-change - part of the Percona Toolkit
! Create a new table with desired schema
! Setup triggers to copy data from old to the new table
! Copy the data from old to new table in batches
! Rename table once the data has been copied
16
How to deal with schema changes?

17. Copyright 2015 Severalnines AB
How to deal with schema changes?
17

18. Copyright 2015 Severalnines AB
! Point-in-time change, you can use for any DDL
! Replication-aware, it can monitor lag and throttle itself
! Introduces additional load, slow down your system
! It has issues with foreign keys
! Can’t work with tables which have triggers
! Won’t work with tables without PK and unique key
18
How to deal with schema changes?

19. Copyright 2015 Severalnines AB
! Otherwise pt-online-schema-change is a very reliable tool
! Schema change may take a long time but it’s throttling
itself and can be paused at any time
! One of the most important tools in the DBA arsenal
! Sometimes it’s the only way to run a schema change
without downtime
19
How to deal with schema changes?

20. Copyright 2015 Severalnines AB
How to deal with topology changes?
20

21. Copyright 2015 Severalnines AB
! If you use GTID - it’s easy
! GTID allows for a flexible topology change
! CHANGE MASTER TO … MASTER_AUTO_POSITION=1;
! You can slave off every node in the replication chain
! Be aware of errant transactions, though
! http://www.severalnines.com/blog/mysql-replication-and-
gtid-based-failover-deep-dive-errant-transactions
21
How to deal with topology changes?

22. Copyright 2015 Severalnines AB
! If you do not use GITD, things are more complex
! Slave knows only a binlog position of itself and it’s master
! You can use binary and relay logs to identify position of a
given transaction in the replication chain but it’s a
complex and error prone process
! To avoid it, you have to stop writes on a involved subset of
hosts
22
How to deal with topology changes?

23. Copyright 2015 Severalnines AB
How to deal with topology changes?
23

24. Copyright 2015 Severalnines AB
! First step requires to slave
DB3 and DB4 off DB2
! You need to enable log-
slave-updates on DB2
! Stop DB2, DB3 and DB4 at
the same position using
START SLAVE UNTIL …
! Start it until a beginning of
binary log two files later
24
How to deal with topology changes?

25. Copyright 2015 Severalnines AB
! Locate your first position in binlog using mysqlbinlog tool
! START SLAVE UNTIL master_log_file='mysql-bin.000121',
master_log_pos=4;
! Run FLUSH LOGS on the master two times to rotate logs -
slaves should end up stopped at the same position
! Check the position on DB2 (SHOW MASTER STATUS)
! Use it to slave DB3 and DB4 off DB2
25
How to deal with topology changes?

26. Copyright 2015 Severalnines AB
! We should finally reach the
desired topology
! Next step - setup master -
master replication between
DB1 and DB2
! Confirm that DB2 is not taking
writes other than the replication
! Use server_id in binary logs for
that - you should see only DB1
! If it does writes, you’re in
trouble - investigate and
eliminate the culprit
26
How to deal with topology changes?

27. Copyright 2015 Severalnines AB
! Once confirmed writes are
not hitting DB2, execute
CHANGE MASTER TO … on
DB1 pointing it to the DB2 and
using any recent coordinates
! Monitor Exec_Master_Log_Pos
in the SHOW SLAVE STATUS on
DB1, it should be stable. If it
does increase, something is
still writing to DB2.
! Once all is set up, you are
ready for a failover
27
How to deal with topology changes?

28. Copyright 2015 Severalnines AB
Potential issues in MySQL replication
28

29. Copyright 2015 Severalnines AB
! Lag is inevitable part of MySQL replication, async or
semisync, it doesn’t matter
! Lag can be caused by large number of writes and DDL’s
! Multi-threaded slave + GTID is a good choice to minimize
lag caused by writes (for DDL’s - use pt-online-schema-
change)
! Hardware has to follow too, especially I/O subsystem
29
Potential issues in MySQL replication

30. Copyright 2015 Severalnines AB
! Data drift may happen even if you take all necessary
precautions
! Use ROW binary log format for the best data consistency
! Use read_only and super_read_only on slaves to minimize
possibilities of errant writes
! Use pt-table-checksum and pt-table-sync to keep an eye on
the data consistency
! When in doubt, rebuild a slave from the source of truth (master)
30
Potential issues in MySQL replication

31. Copyright 2015 Severalnines AB
! Errant transactions can ruin your day
! Those are transactions executed on a slave but not on a
master
! read_only and super_read_only can help to avoid them
! GTID_SUBSET(), GTID_SUBTRACT() will help you to detect them
! We’ve covered the topic here:
http://www.severalnines.com/blog/mysql-replication-and-
gtid-based-failover-deep-dive-errant-transactions
31
Potential issues in MySQL replication

32. Copyright 2015 Severalnines AB
! More blogs featuring MySQL replication:
! http://severalnines.com/blog/become-mysql-dba-blog-series-
common-operations-schema-changes
! http://severalnines.com/blog/become-mysql-dba-blog-series-
common-operations-replication-topology-changes
! http://severalnines.com/blog/become-mysql-dba-blog-series-live-
migration-using-mysql-replication
! http://severalnines.com/tutorials/mysql-replication-high-availability-
tutorial
! Contact: krzysztof@severalnines.com
32
Thank You!