In this webinar, we will introduce the Severalnines Blueprint for MySQL Replication - this includes all aspects of a MySQL Replication topology with the ins and outs of deployment, setting up replication, monitoring, upgrades, performing backups and managing high availability using proxies as ProxySQL, MaxScale and HAProxy. Agenda This webinar will cover the Full Scale MySQL replication stack: Deployment Setting up replication Backups High Availability Speaker Art van Scheppingen is a Senior Support Engineer at Severalnines. He’s a pragmatic MySQL and Database expert with over 15 years experience in web development. He previously worked at Spil Games as Head of Database Engineering, where he kept a broad vision upon the whole database environment: from MySQL to Couchbase, Vertica to Hadoop and from Sphinx Search to SOLR. He regularly presents his work and projects at various conferences (Percona Live, FOSDEM) and related meetups.

1. Confidential
MySQL Replication Blueprint
April 5, 2015
Art van Scheppingen
Senior Support Engineer, Severalnines
art@severalnines.com

2. Confidential
Your host & some logistics
I'm Jean-Jérôme from the Severalnines Team and I'm your host for
today's webinar!
Feel free to ask any questions in the Questions section of this
application or via the Chat box.
You can also contact me directly via the chat box or via email:
jj@severalnines.com during or after the webinar.

3. Confidential
Agenda
☐ Why a Blueprint for Replication?
☐ MySQL Replication Blueprint - Components & Topology
☐ Monitoring & Trending
☐ Management
☐ Load balancing
☐ Q&A

4. Confidential
Why the need for a MySQL Replication blueprint?

5. Confidential
Replication in the pre-MySQL 5.6 era
☐ Pre-MySQL 5.6 replication
☐ Easy to set up
☐ Easy to break
☐ Difficult and sometimes impossible to repair
☐ Tools become necessary
☐ Failover tools (MHA, PRM, Mysqlfailover)
☐ Percona-Toolkit (pt-heartbeat, pt-table-sync, pt-slave-restart)

6. Confidential
Is this production ready?
☐ Enable master binlog
☐ Have slave(s) read from master
☐ Are we done?

7. Confidential
Issues to be addressed in production (1 / 2)
☐ Master as a SPOF - failover (automatic vs manual)
☐ Data consistency
☐ Read write split
☐ Dynamic topology changes
☐ Load balancing

8. Confidential
Issues to be addressed in production (2 / 2)
☐ Config changes
☐ Version upgrades
☐ Schema changes
☐ Backup strategies
☐ Scaling
☐ Monitoring

9. Confidential
Making MySQL Replication Production Ready
☐ Holistic approach needed
☐ Deployment automation
☐ Failover tools
☐ Virtual ip addresses
☐ Load Balancers
☐ Backups
☐ Topology logic implemented many times (in each component)

10. Confidential
Only one version of the truth
☐ Who is the current master?
☐ In case of failover, who is the new master?
☐ Which are the slaves?
☐ Application requests always directed to the right server
(depending on reads vs writes)

11. Confidential
GTID - A Stronger Foundation for Replication
☐ Global Transaction Identifier (GTID) introduced in 5.6
☐ Used to identify and associate each transaction to its origin
☐ Makes the transaction unique in the topology
☐ Different GTID implementation in MySQL and MariaDB

12. Confidential
Management: Repair A Broken Topology
☐ Slave promotion without GTID
☐ Find the most advanced slave
☐ Elect this node as the new master
☐ Find difference between new master and slaves
☐ Find binlog position per slave
☐ Re-attach slaves to new master
☐ Why do we need to find the binlog position?
☐ Slaves have their own binary logs (slave updates)
☐ Position in slave logs do not correspond to the master

13. Confidential
Management: Repair A Broken Topology
☐ Slave promotion with GTID
☐ Find the most advanced slave
☐ Elect this node as the new master
☐ Re-attach slaves to new master with auto-position
☐ Auto-position
☐ Master finds the binlog file and position for the slave
☐ Slaves will automatically receive remaining transactions
☐ Transactions are in the correct order

14. Confidential
Introducing the MySQL Replication Blueprint
Components and Topology

15. Confidential
Components and topology

16. Confidential
Components and Topology: Master/Slave
☐ Most simplistic MySQL replication topology

17. Confidential
Components and Topology: Master/Slave

18. Confidential
Components and Topology: Multi Master
☐ Similar to Master/Slave
☐ All nodes are master and replica
☐ Circular replication
☐ Active/Passive
☐ One node is the master
☐ Other node(s) are hot standby.
☐ Active/Active
☐ All nodes are master

19. Confidential
Components and Topology: Multi Master

20. Confidential
Components and Topology: Multi Master
☐ Prevent infinite loops
☐ log-slave-updates=1 + replicate-same-server-id=0
☐ Even with GTID
☐ Prevent conflicts
☐ Writing to the same data/schemas
☐ Replication bottleneck
☐ Active/Active: two replication streams
☐ After a master fails you only have one replication stream

21. Confidential
Components and Topology: Parallel Replication
☐ MySQL replication is single threaded
☐ Replication blocked
☐ Long running queries
☐ DDL changes
☐ Parallel Replication
☐ Multiple replication streams

22. Confidential
Components and Topology: Parallel Replication

23. Confidential
Components and Topology: Parallel Replication
☐ MySQL 5.6
☐ Replication thread per schema
☐ MySQL 5.7
☐ Intra schema
☐ Group commit large write sets

24. Confidential
Components and Topology: Multi Source Replication
☐ Slave replicates from multiple masters
☐ No conflicting schemas
☐ Replication channels
☐ Little support with replication tools
☐ Unmanaged slave
☐ Useful for
☐ Data Warehousing
☐ Delayed slaves
☐ Data locality

25. Confidential
Components and Topology: Multi Source Replication

26. Confidential
Components and Topology: Monitoring
☐ Replication Manager / Failover tool
☐ Monitors replication stream
☐ Monitor slave lag
☐ Flap protection
☐ Global state of the topology
☐ Performance monitoring (Trending)
☐ Replication stream
☐ Slave lag

27. Confidential
Components and Topology: Management
☐ Replication Manager / Failover tool
☐ Repair broken replication
☐ Scaling
☐ Backup Management
☐ Updates
☐ Schema Changes
☐ Configuration Management

28. Confidential
Components and Topology: Load Balancers
☐ Various types of load balancers
☐ Layer 4 (TCP/IP)
☐ Layer 7 (Application)
☐ Needs to monitor the topology

29. Confidential
Monitoring & Trending

30. Confidential

31. Confidential
Monitoring vs Trending
☐ Monitoring
☐ Keeps an eye on your systems
☐ Will alert if a threshold is met
☐ Trending
☐ Insight into the system internals
☐ Trends can warn you before anything has happened yet
☐ Keep historical state/data

32. Confidential
Monitoring: Availability
☐ Do more than SELECT 1;
☐ Measure true status of nodes and cluster
☐ Test read/write
☐ Open essential schemas
☐ Check the full topology

33. Confidential
Trending: Why do we need trends?
☐ Trending
☐ Plot trends of key (performance) metrics
☐ Find problems before they arise
☐ Pre-emptive problem management
☐ Trending tools
☐ Granularity of sampling
☐ More datapoints = better

34. Confidential
Trending: What metrics to store?
☐ Resource monitoring
☐ CPU
☐ Memory
☐ IO capacity
☐ Diskspace
☐ Replication monitoring
☐ Slave lag
☐ Transactions per second

35. Confidential

36. Confidential
Trending: Correlate
☐ Correlate your data
☐ Include other metrics into your graphs
☐ Deployments
☐ Failovers
☐ Application metrics

37. Confidential

38. Confidential
Trending: Solutions
☐ Storage
☐ Grafana
☐ Prometheus
☐ InfluxDB
☐ Trending tools
☐ Grafana
☐ Cacti
☐ Hosted
☐ VividCortex
☐ Integrated Framework
☐ ClusterControl

39. Confidential
Alerting
☐ Get notified when it goes wrong
☐ Integrate alerting with monitoring tools
☐ Ready made integrations
☐ APIs
☐ Alerting services
☐ Pagerduty
☐ OpsGenie
☐ VictorOps

40. Confidential
Management

41. Confidential
Management: Scaling Out
☐ Add new slaves
☐ Prime new slave with a copy of the data
☐ Add to existing topology
☐ Monitoring
☐ Failover tools
☐ Load balancers

42. Confidential

43. Confidential

44. Confidential
Management: Scaling Out
☐ Performance issues:
☐ Too many replication threads on master
☐ Add intermediate slaves
☐ Adds more complexity
☐ Not all failover tools support intermediate slaves

45. Confidential
Management: Delayed Slave
☐ Why slow down the replication?
☐ Quick access to historical state
☐ Accidentally wiped data
☐ Hacked databases
☐ MySQL 5.7
☐ CHANGE MASTER TO … MASTER DELAY=3600;
☐ Other versions
☐ pt-slave-delay

46. Confidential
Management: Repair A Broken Topology
☐ MySQL replication is fragile
☐ Master failure
☐ Network problems
☐ Data inconsistency (data drift)

47. Confidential
Management: Repair A Broken Topology
☐ Failover automation
☐ Promote slaves automatically
☐ Controlled failover
☐ Move virtual ip addresses
☐ Perform other pre- and post-failover tasks
☐ Failover tools
☐ MySQL Master HA (MHA)
☐ Percona Replication Manager (PRM)
☐ Orchestrator
☐ Mysqlfailover

48. Confidential
Management: Backups
☐ Logical backups
☐ Dump of all tables and rows
☐ Mysqldump
☐ Physical backups
☐ File copy of the MySQL data directory
☐ Xtrabackup
☐ MySQL Hotcopy
☐ Filesystem snapshots (LVM, ZFS)

49. Confidential
Management: Backups
☐ Why would you make a backup?
☐ Disaster recovery
☐ Testing/staging/development
☐ Priming new slaves
☐ Replication is not a backup
☐ DDL, Truncate and delete statements are replicated
☐ Delayed slave will not help if your DC is down

50. Confidential
Management: Backups
☐ Full or incremental backups?
☐ Xtrabackup allows incremental backups
☐ Only copy altered data
☐ Smaller backups
☐ Full backup necessary as starting point
☐ Slower recovery

51. Confidential
Management: Backups
☐ Scheduling
☐ Off-peak hours scheduling
☐ Don’t overlap schedules
☐ Backup suites
☐ MySQL Enterprise backup
☐ Bacula
☐ Zmanda Recovery Manager (ZRM)
☐ Holland Backup

52. Confidential
Management: Backups
☐ Backup validation
☐ Unpack backups
☐ Start MySQL from recovered backup
☐ Rebuild slave from backup
☐ Rebuild topology
☐ Scheduled verification
☐ Offsite backups
☐ Copy to another DC
☐ Copy to cloud (Amazon Galcier / S3)

53. Confidential
Management: Updating or Upgrading
☐ Minor version updates
☐ New features
☐ Bugfixes
☐ Security fixes
☐ Easy to update, rolling restart
☐ Major version upgrades
☐ Need preparation
☐ Cluster downtime

54. Confidential
Management: Schema Changes
☐ DDL Changes propagated through replication
☐ DDL statements are blocking the replication stream
☐ Can cause slave lag
☐ Apply DDL Changes locally
☐ Node by node change
☐ Master failover necessary
☐ DDL change has to be compatible
☐ Apply DDL Changes through Percona Online Schema
Change

55. Confidential
Management: Schema Changes
☐ Percona Online Schema Change
☐ Creates the new table structure
☐ Creates triggers on the old table
☐ Backfills data in the background
☐ Swaps the two tables once done
☐ Not 100% compatible with all DDL changes
☐ Always test your changes first!

56. Confidential
Management: Configuration Changes
☐ Manage your configurations
☐ Deployment systems (Puppet, Chef, etc.)
☐ Keep a copy inside Git/SVN
☐ Sync your configuration
☐ Settings may differ per host
☐ Runtime changes not propagated to configuration files
☐ Visual differences (MonYog, ClusterControl)

57. Confidential
Load Balancing

58. Confidential
Load Balancing
☐ Ensure any component can/may fail
☐ Master node is a single point of failure
☐ Master failover must be handled application side as well
☐ Configuration managers (Zookeeper, Consul)
☐ Virtual ip addresses (not always available / allowed)
☐ Load Balancers

59. Confidential

60. Confidential

61. Confidential
Load Balancers: Benefits
☐ Transparent
☐ Sits between application and database
☐ Application does not have to be topology aware
☐ Split database connections (reads and writes)
☐ Runtime read/write splitting
☐ Instant failover
☐ Failover happens for all hosts at the same time

62. Confidential
Load Balancing: Proxies
☐ Which proxies are available today?
☐ Layer 3 (network)
☐ HAProxy
☐ LVS
☐ Layer 7 (application)
☐ MySQL Proxy
☐ MaxScale
☐ ProxySQL

63. Confidential
Load Balancing: Proxies
☐ MaxScale
☐ Understands SQL
☐ Understands MySQL topologies
☐ Runtime read/write splitting
☐ Query rewriting
☐ Can be resource intensive
☐ Topology discovery may be different

64. Confidential
Load Balancing: Proxies
☐ ProxySQL
☐ Understands SQL
☐ Understands MySQL topologies
☐ Runtime read/write splitting
☐ Query rewriting
☐ Query caching
☐ Relatively new (GA since November 2015)

65. Confidential
Load Balancing: Proxies
☐ HAProxy
☐ Fast
☐ Proven technology
☐ MySQL scripts readily available

66. Confidential
Load Balancing: Query Caching
☐ MySQL Query Cache flawed
☐ Shared memory space
☐ Locks on every read/write operation ( seconds)
☐ Evictions take longer
☐ High concurrency
☐ ProxySQL
☐ No locks
☐ Regular expression match
☐ Caching for milliseconds
☐ Performs similar as Redis and Memcache

67. Confidential
Load Balancing: Query Rewriting
☐ Why would you rewrite queries?
☐ Bad performing queries
☐ Broken queries (bad code drop)
☐ Partitioning/sharding
☐ Negate SQL injections / hacks
☐ Which proxies?
☐ ProxySQL
☐ MaxScale

68. Confidential
Our implementation

69. Confidential
ClusterControl: An Integrated Framework

70. Confidential
ClusterControl: Demo

71. Confidential
Q & A

72. Confidential
Additional Resources
☐ Ebook: MySQL Replication for High Availability
☐ www.severalnines.com/whitepapers
☐ Become a MySQL DBA blog series
☐ severalnines.com/blog-categories/db-ops
☐ Blog: MySQL Replication failover - MaxScale vs MHA
☐ severalnines.com/blog/mysql-replication-failover-maxscale-vs-mha-part-1
☐ Load Balancing Tutorial
☐ severalnines.com/tutorials/mysql-load-balancing-haproxy-tutorial
☐ We will soon have a whitepaper on the MySQL Replication Blueprint