1. Failover, or not Failover,
that is the question
Percona Live MySQL Conference and Expo 2013
Massimo Brignoli, SkySQL
Henrik Ingo, Nokia
Please share and reuse this presentation licensed under the Creative Commonse Attribution License

2. Agenda
● Why HA is more difficult for databases
● Steps to failover
● Monitoring
● Automating failover
● Sounds great!
What could possibly go wrong?
● Amazon Dynamo
● Galera and NDB

3. Fault tolerance = redundancy
● RAID
● 2 power units per server
● Cluster of servers
● 2 kidneys per person
● Redudancy at all levels:
Software, Hardware, Network, Electricity...
A chain is as strong as the weakest link.

4. Durability
"Durability is an interesting concept.
If I flush a transaction to disk,
it is said to be durable.
But if I then take a backup,
it is even more durable."
Heikki Tuuri

5. Why High Availability is
More Difficult for Databases
Redundancy of server
AND
Redundancy of data
WHILE
Performing thousands of write operations
per second onto the dataset

6. What failover?
1. Primary server
2. Secondary / Standby server
for redundancy
3. In case Primary fails,
Secondary server must
become the new Primary

7. Steps to failover (theory)
1. Notice failure
2. Move VIP
3. Continue

8. Automating failover
Generic Clustering Solutions
● Pacemaker/Corosync
● Linux Heartbeat
● Red Hat Cluster Suite
● Solaris Cluster
● Windows Server Failover
Clustering
● etc...
MySQL Specific Solutions
● MMM
● PRM
● MHA
● JDBC connector

9. Steps to failover (DRBD)
VIP
VIP
1. Have DRBD
2. Notice failure
3. Shutdown MySQL on primary
4. Unmount disk on primary
5. Mount disk on secondary
6. Start MySQL on secondary
7. Wait for InnoDB recovery
8. Wait for InnoDB recovery
9. Wait for InnoDB recovery
10. Unset VIP on primary
11. Set VIP on secondary
12. Continue
13. Should you add a new secondary?

10. Steps to failover (MySQL replication)
VIP
VIP
1. Have replication
2. Notice failure
3. Make slave writable
4. Make master read-only
5. Unset VIP on master
6. Set VIP on slave
7. Continue
8. Should you add a new slave?

11. What if you have more than 2 servers?
(MySQL replication)
VIP
VIP
?
● MySQL replication failover with more than 2
servers can be a hassle.
● Which slave should become the new master?
● All slaves must be pointed to the new master.
● They must figure out where to continue
replication (binlog position)
● MySQL 5.6 GTID helps.

12. MHA and SkySQL...
● Combination of
resource manager
+ scripts
● Automating failover
process:
○ New Master
selection
○ Slaves
reconfiguration
○ VIP management
○ Missing binlogs
retrieval

13. Sounds great,
what could possibly go wrong?

14. Sounds great,
what could possibly go wrong?
VIP
1. Have replication
○ Ok, is it working? What if it's not working?
○ Is it replicating in the right direction?
○ Does your bash script handle binlog positions correctly?
○ Asynchronous?
2. Notice failure
○ Polling interval
○ Who is polling?
○ ...and from where?
○ How is he handling failure himself?
○ False positives
○ Is failover the right response to every failure?
3. STONITH
○ Shutdown MySQL on Primary? How? It's not responding...
○ Unmount disk on Primary? How? It's not responding...
○ "You need a STONITH device"! Hehe, nice try...
4. Move VIP
○ Unset VIP on Master/Primary? How? It's not responding...
○ Set VIP on Secondary/Slave. This will work fine. Unfortunately.
5. Continue
6. Add back new/same Secondary
○ Automatically of course. Even if it just failed 15 seconds ago.
VIP

15. Case Github
https://github.com/blog/1261-github-availability-this-week
● MySQL replication, Pacemaker, Corosync, Percona Replication Manager
● PRM health check fails due to high load during schema migration.
● Failover!
● New node has cold caches, so even worse performance.
● Failover! (back)
● Disable PRM
● A slave is found outdated as replication is not happening
● Enable PRM and hope it will fix it
● Pacemaker segfaults, causing cluster partition
● PRM selects the outdated node as master, shuts down others
● All kinds of data inconsistencies
● Restart PRM on all nodes
● ...

16. Case Github
Lesson learned:
Automated failover is dangerous
Cold cache is dangerous

17. But...
Not automating is also dangerous
Baron Schwartz:
75% of replication failures are human errors
http://www.percona.com/about-us/mysql-white-paper/causes-of-downtime-in-production-mysql-servers
80% of Aviation accidents
are caused by human errors
http://asasi.org/papers/2004/Shappell%20et%20al_HFACS_ISASI04.pdf
80% Events caused by human errors, 70% of
them due to organization weaknesses
http://www.hss.doe.gov/sesa/corporatesafety/hpc/fundamentals.html

18. Are we solving the right problem?

19. Instead of automating the problem...
Eliminate the problem!

21. Amazon Dynamo
R + W > N
Voldemort, Cassandra, RIAK, DynamoDB, S3
http://openlife.cc/blogs/2012/september/failover-evil

22. N=3, R=W=2
R + W > N

23. Eventual consistency is internal only
R + W > N

24. Failover?
Single node failure is a non-event!

25. For relational databases?
Synchronous replication is
special case of Dynamo:
W=N & R=1

26. Or is there a failover after all?
Due to W=N, writers actually notice node
failures! Cluster reconfiguration needed.
(Readers are ok.)
?

27. Example: Galera
OKTimeout

28. Example: MySQL NDB Cluster

29. What have we learned?
● Failover with DRBD is painful because it is
slow.
● Failover with MySQL replication is painful
because it's a mess.
● Amazon Dynamo has no failover
● Galera Cluster has no failover but needs
cluster reconfiguration. Same thing...
● MySQL NDB Cluster has failover but you
can't see it.