This document discusses replication topology changes for MySQL and MariaDB databases. It covers making changes using GTID or regular replication, the failover process, and tools like MaxScale and ProxySQL that can help automate query rerouting during a failover. Specific topics covered include reslaving nodes, setting up master-master replication, and performing both offline and online failovers.
Secure your environment with UiPath and CyberArk technologies - Session 1
Webinar slides: Replication Topology Changes for MySQL and MariaDB
1. 1
Your host & some logistics
I'm Jean-Jérôme from the Severalnines Team
and I'm your host for today's webinar!
Feel free to ask any questions in the
Questions section of this application or via
the Chat box.
You can also contact me directly via the chat
box or via email: jj@severalnines.com
during or after the webinar.
2. Copyright 2015 Severalnines AB
Replication Topology Changes for MySQL and MariaDB
September 29, 2015
Krzysztof Książek
Severalnines
krzysztof@severalnines.com
2
3. Copyright 2015 Severalnines AB
! We want to help all non-DBA people who look after
MySQL infrastructures
! Share tips and good practices
! Watch the replays of the previous webinars on our
Slideshare page
! http://www.slideshare.net/Severalnines/videos
3
“Become a MySQL DBA” series
4. Copyright 2015 Severalnines AB
! MySQL Replication topology changes
! using GTID
! using regular replication
! Failover process
! Using MaxScale for automatic re-routing of queries
! Other external tools useful when dealing with failover
4
Agenda
6. Copyright 2015 Severalnines AB
! Maintenances
! Adding an index on a large table
! Delete data and then rebuild a large table to reclaim
disk space
! Reslaving nodes for a planned failover
! Reslaving nodes after master crash
6
Why do we make changes in a replication chain?
7. Copyright 2015 Severalnines AB
! GTID allows for a flexible topology changes
! CHANGE MASTER TO … MASTER_AUTO_POSITION=1;
! You can slave off every node in the replication chain
! Be aware of errant transactions, though
! http://www.severalnines.com/blog/mysql-replication-and-
gtid-based-failover-deep-dive-errant-transactions
7
Replication topology changes - GTID
8. Copyright 2015 Severalnines AB
! Far less flexible than GTID
! Slave knows only a binlog position of itself and it’s master
! You can use binary and relay logs to identify position of a
given transaction in the replication chain but it’s a
complex and error prone process
! To avoid it, you have to stop writes on a involved subset of
hosts
8
Replication topology changes - no GTID
10. Copyright 2015 Severalnines AB
! First step requires to slave
DB3 and DB4 off DB2
! You need to enable log-
slave-updates on DB2
! Stop DB2, DB3 and DB4 at
the same position using
START SLAVE UNTIL …
! Start it until a beginning of
binary log two files later
10
Replication topology changes - no GTID
11. Copyright 2015 Severalnines AB
! Locate your first position in binlog using mysqlbinlog tool
! START SLAVE UNTIL master_log_file='mysql-bin.000121',
master_log_pos=4;
! Run FLUSH LOGS on the master two times to rotate logs -
slaves should end up stopped at the same position
! Check the position on DB2 (SHOW MASTER STATUS)
! Use it to slave DB3 and DB4 off DB2
11
Replication topology changes - no GTID
12. Copyright 2015 Severalnines AB
! We should finally reach the
desired topology
! Next step - setup master -
master replication between
DB1 and DB2
! Confirm that DB2 is not taking
writes other than the replication
! Use server_id in binary logs for
that - you should see only DB1
! If it does writes, you’re in
troubles - investigate and
eliminate the culprit
12
Replication topology changes - no GTID
13. Copyright 2015 Severalnines AB
! Once confirmed writes are
not hitting DB2, execute
CHANGE MASTER TO … on
DB1 pointing it to the DB2 and
using any recent coordinates
! Monitor Exec_Master_Log_Pos
in the SHOW SLAVE STATUS on
DB1, it should be stable. If it
does increase, something is
still writing to DB2.
! Once all is set up, you are
ready for a failover
13
Replication topology changes - no GTID
15. Copyright 2015 Severalnines AB
! Exact failover process may depend strongly on the
infrastructure that you use
! In short - it’s a process of moving writes from one
database node to another
! It may or may not be graceful, depending on your
application and tools that you use
15
Failover Process
16. Copyright 2015 Severalnines AB
! Requires downtime period
for a master switch
! Not ideal solution but
easiest to implement and it
will work with every
application that can be
stopped
! We are assuming the
topology we reached in our
previous slides, with failover
about to happen from DB1
to DB2
16
Offline failover
17. Copyright 2015 Severalnines AB
! As a first step, you need to stop all activity in your
application - you don’t want to have any writes hitting db
! You can monitor it using SHOW MASTER STATUS on the
master
! You can monitor binary logs to locate and identify writes
that hit the database
! Com_* counters can also be monitored
17
Offline failover
18. Copyright 2015 Severalnines AB
! Next step - you need to repoint your application to the
new master, DB2 in our example
! Depending on your setup you can:
! Change DNS entries
! Set read_only values accordingly on MySQL hosts
! Use your proxy’s settings to redirect connections from
one host to the other
18
Offline failover
19. Copyright 2015 Severalnines AB
! Once you pointed writes to a new master, you may want
to perform some tests before going back live
! All the time, ensure that any writes are happening only on
the correct host - new master, DB2 in our case
! Do not use any ‘hardcoded’ IP’s in your scripts - this may
become a serious problem when you will be performing a
failover
19
Offline failover
20. Copyright 2015 Severalnines AB
! Moving writes without downtime period - planned failover
! Stop the writes on the master, failover, move the writes to
the new master - sounds familiar?
! Main problem is - how to do it gracefuly?
! FTWRL + read_only = 1? May take too long
! read_only = 1? Will kill current connections
! Drop the user + wait few seconds + read_only = 1? Still not
ideal but may be acceptable
20
Online failover
21. Copyright 2015 Severalnines AB
! Proxies may help significantly in this regard - single point of
config change, immediate result
! Caching reads and writes can also be useful
! Application will have to be able to handle errors and
rollbacks as transition may but also may not be graceful
! Some of the tools can help you to make it graceful,
without application seeing any error at all
21
Online failover
23. Copyright 2015 Severalnines AB
! MaxScale can implement read-write split
! MaxScale can be used with MySQL replication backend
! It monitors health of the backend nodes
! It detects replication topology changes
! Makes easier to handle any changes like failover
23
Using MaxScale to reroute queries
24. Copyright 2015 Severalnines AB
! For reads, most of the failures are being handled
gracefully
! For writes, not yet - errors will be thrown and transactions
rolled back, application has to handle them
! But it should be handling them anyway, right?
! Experimental support for handling failovers (along with
reslaving) in the latest MaxScale
! No easy way of performing a planned failover, though
24
Using MaxScale to reroute queries
26. Copyright 2015 Severalnines AB
! Handles the failover process for you
! Both automatic failover and manual one
! Can be used as a standalone solution or a part of a set of
scripts or tools
! Integrates nicely with other tools in a manual failover
process
! Can work with GTID and regular replication
26
MHA
27. Copyright 2015 Severalnines AB
! Part of MySQL Utilities from Oracle
! Allows user to perform a manual failover
! Works only with GTID replication
! Checks for errant transactions
! Can be easily combined with other scripts through --exec-
before and --exec-after
27
mysqlrpladmin
28. Copyright 2015 Severalnines AB
! New addition to the ‘database-aware’ proxy family
! Works as a gateway for all connections to databases
! Performs read/write split based on user-defined set of rules
! Gives you ability to perform a sub second graceful
failover in case of a master-master replication.
! Works great with external scripts for handling topology
changes
28
ProxySQL
29. Copyright 2015 Severalnines AB
! Idea is simple - all queries go through proxy to backend nodes
! You can use failover script to switch a backend node into a
‘failover’ mode
! Proxy will not forward more connections to it, it will “buffer”
new connections
! Your failover script can wait till the current transactions finish or
timeout will be reached and transactions can be killed
! In a next step you can reconfigure the replication topology as
needed
29
ProxySQL
30. Copyright 2015 Severalnines AB
! Once there are no transactions running and new
topology has been set up, proxy can be reconfigured to
move writes to another backend
! Depends on how long are your transactions and how long
timeout is set, failover can take even a fraction of a
second
! Application will see temporary slowdown but should not
see any errors
! Additional scripting is required - ProxySQL doesn’t monitor
replication topology on it’s own
30
ProxySQL
31. Copyright 2015 Severalnines AB
! More blogs in “Become a MySQL DBA” series:
! http://www.severalnines.com/blog/become-mysql-
dba-blog-series-query-tuning-process
! http://www.severalnines.com/blog/become-mysql-
dba-blog-series-configuration-tuning-performance
! Contact: krzysztof@severalnines.com
31
Thank You!