Replication enhancements in MySQL 5.6, including GTIDs, HA / Self-healing, multi-threaded slaves and more. Slides over design rationale, implementation and how to get started using these new capabilities
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
My sql 5.6_replwebinar_may12
1. MySQL 5.6 Replication:
Supporting High Scale Web & Cloud Services
Mat Keep Andrew Morgan
MySQL Product Management MySQL Product Management
mat.keep@oracle.com andrew.morgan@oracle.com
2. Session Agenda
<Insert Picture Here>
• Development Priorities
• MySQL 5.6 Replication Enhancements
• Resources to Get Started
The image cannot be
Copyright 2012 Oracle Corporation 2
3. The presentation is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle s
products remains at the sole discretion of Oracle.
The image cannot be
Copyright 2012 Oracle Corporation 3
4. Industry Leaders Rely on MySQL
Replication is Key to Scale & HA
Web & Enterprise OEM & ISVs
Cloud
The image cannot be
Copyright 2012 Oracle Corporation 4
5. MySQL Database 5.6: A Better MySQL.
• MySQL 5.6 builds on MySQL 5.5 by improving:
• Performance and Scalability
• Optimizer for better query execution times, diagnostics
• Performance Schema for better instrumentation
• InnoDB for better transactional throughput
• NoSQL API for better flexibility
• Replication largest set of enhancements ever released!
• Available now under GPL
• Development Milestone Release model enabling early
evaluation
The image cannot be
6. MySQL Replication:
Development Priorities
HIGH PERFORMANCE AUTOMATED FAILOVER & RECOVERY
DATA INTEGRITY DEV/OPS AGILITY
The image cannot be
Copyright 2012 Oracle Corporation 6
7. Translating to User Needs
PERFORMANCE
• Improves read consistency from slaves
• Reduces risk of data loss if master fails
FAILOVER & RECOVERY
• Minimize service interruption
• Self-Healing
DATA INTEGRITY
• Correct, consistent & accessible
DEV/OPS AGILITY
• Reducing TCO
• Quickly responding to business demand
The image cannot be
Copyright 2012 Oracle Corporation 7
10. 5x Higher Performance with MySQL 5.6
Multi-Threaded Slave Performance
300
250
Queries per Second
200
150
100
50
0
0 5 10
QPS 58.11 144.4 282.53
Worker Threads
The image cannot be
11. Multi-Threaded Slaves
Relay • Increases slave throughput,
Log
reducing lag
T4 T3 T2 T1 I/O
• All transactions received into
slave’s relay log
Coordinator • Implements multiple SQL
threads, based on database
• Applies events to different
SQL-A
SQL-B
SQL-C
T4
databases in parallel
T1 T2 T3 • Great for systems which
isolate application data using
DB-A DB-B DB-C
databases – e.g. multi-tenant
The image cannot be
Copyright 2012 Oracle Corporation 11
12. Multi-Threaded Slaves
• Throughput of slave increased by allowing multiple
slave threads:
• 0 – functionality disabled
• Up to 1024
• Exec_Master_Log_Posn in SHOW SLAVE STATUS
now shows the low-water-mark
• Configure using:
slave-parallel-workers=4
• Only ensures sequencing within a database
• Unsuitable if ordering of updates between databases is not
required
The image cannot be
Copyright 2012 Oracle Corporation 12
13. Multi-Threaded Slaves
Master Slave thread1 Slave thread2
INSERT INTO dba.tab1 ...
dba.tab1 dbb.tab1
INSERT INTO dba.tab2 ...
dba.tab2 dbc.tab2
INSERT INTO dbb.tab1 ...
dba.tab1 dbb.tab3
INSERT INTO dbc.tab2 ...
dba.tab3 dbc.tab2
INSERT INTO dba.tab1 ...
dba.tab4
INSERT INTO dbb.tab3 ...
• Transactions applied in parallel on
INSERT INTO dba.tab3 ...
the slave
INSERT INTO dbc.tab2 ... • Sequencing of transactions within a
INSERT INTO dba.tab4 ...
given database will be the same
The image cannot be
Copyright 2012 Oracle Corporation 13
14. Optimized Row Based Replication
Primary Key Changed Columns
• Increases replication throughput for master and slave
• Reduces Binlog size, memory requirements & network
bandwidth
• Only replicates those elements of the Row image that
have changed
The image cannot be
Copyright 2012 Oracle Corporation 14
15. Optimized Row Based Replication
• Default is to include full before & after image for every
changed row
• New option: binlog-row-image=minimal
• Reduces space:
• Inserts: only after image
• Deletes: only before image
• Updates: after image + primary key or indexed columns of
before image
The image cannot be
Copyright 2012 Oracle Corporation 15
17. Global Transaction Identifiers
Master
GTID=123456
GTID=123456
GTID=123456 GTID=123456
• Foundation for reliable, automatic failover & recovery
• Unique identifier for each replication event written to the Binlog
• Simple to track & compare replication across the cluster
• Automatically identify the most up-to-date slave for failover
• Deploy complex replication topologies
• Eliminates Dev/Ops overhead
The image cannot be
Copyright 2012 Oracle Corporation 17
18. Before Global Transaction IDs
Master
{my-bin.000101,
873}
{my-bin.000132,
782}
{my-bin.000099, {my-bin.00088,
123} 1027}
The image cannot be
Copyright 2012 Oracle Corporation 18
19. Before Global Transaction IDs
Failover & Recovery
Must convert
{file,pos} from
Master old master to one
from the new master
{my-bin.000101,
873}
CHANGE MASTER TO??
{my-bin.000132,
782}
{my-bin.000099, {my-bin.00088,
123} 1027}
The image cannot be
Copyright 2012 Oracle Corporation 19
20. With Global Transaction IDs
Master
GTID=123456
GTID=123456
GTID=123456 GTID=123456
The image cannot be
Copyright 2012 Oracle Corporation 20
21. With Global Transaction IDs
Failover & Recovery
Slave(s) can start
from same GTID
Master
GTID=123456
CHANGE MASTER TO??
GTID=123456
GTID=123456 GTID=123456
The image cannot be
Copyright 2012 Oracle Corporation 21
22. Global Transaction ID Utilities
• Self-Healing, Low Administration replication cluster
• Automated failover & recovery
• mysqlfailover Utility
• Switchover & administration
• mysqlrpladmin Utility
• Delivers HA within the core MySQL distribution
• Eliminates the need to integrate 3rd party HA frameworks
• Allows extensibility to support variety of HA mechanisms
The image cannot be
23. Fail-
Replication Failover Utility Over
• Automatic failover & slave promotion
• Default is to promote most up-to-date
slave, based on GTID
Failed • Slave promotion policies are fully
Master configurable; select slave(s) that are
candidates for promotion
• Implemented as a command-line utility
Promoted Monitoring Failover
Master Auto-Failover & Utility
Slave Promotion
• Option to bind in your own pre/post
failover scripts
• For example, initiate VIP failover
• Never lose replicated events
Slaves • Nominated slave automatically retrieves any
missing updates from other slaves
The image cannot be
Copyright 2012 Oracle Corporation 23
24. Fail-
Use of mysqlfailover Over
$ mysqlfailover --master=root@rod:3306
--discover-slaves-login=root
MySQL Replication Monitor and Failover Utility
Failover Mode = auto Next Interval = Mon Mar 19 15:56:03 2012
Master Information
------------------
Binary Log File Position Binlog_Do_DB Binlog_Ignore_DB
mysql-bin.000001 571
Replication Health Status
+------------+-------+---------+--------+------------+---------+
| host | port | role | state | gtid_mode | health |
+------------+-------+---------+--------+------------+---------+
| rod | 3306 | MASTER | UP | ON | OK |
| jane | 3306 | SLAVE | UP | ON | OK |
| jane | 3307 | SLAVE | UP | ON | OK |
| freddy | 3306 | SLAVE | UP | ON | OK |
+------------+-------+---------+--------+------------+---------+
Q-quit R-refresh H-health G-GTID Lists U-UUIDs L-log entries
The image cannot be
Copyright 2012 Oracle Corporation 24
25. Fail-
mysqlfailover – master failed Over
Failover starting...
# Candidate slave jane:3306 will become the new master.
# Preparing candidate for failover.
# Creating replication user if it does not exist.
# Stopping slaves.
# Performing STOP on all slaves.
# Switching slaves to new master.
# Starting slaves.
# Performing START on all slaves.
# Checking slaves for errors.
# Failover complete.
# Discovering slaves for master at jane:3306
Failover console will restart in 5 seconds.
The image cannot be
Copyright 2012 Oracle Corporation 25
26. Fail-
mysqlfailover – monitoring Over
resumes
MySQL Replication Monitor and Failover Utility
Failover Mode = auto Next Interval = Mon Mar 19 16:05:12 2012
Master Information
------------------
Binary Log File Position Binlog_Do_DB Binlog_Ignore_DB
mysql-bin.000001 1117
UUIDs
+------------+-------+---------+--------+------------+---------+
| host | port | role | state | gtid_mode | health |
+------------+-------+---------+--------+------------+---------+
| jane | 3306 | MASTER | UP | ON | OK |
| jane | 3307 | SLAVE | UP | ON | OK |
| freddy | 3306 | SLAVE | UP | ON | OK |
+------------+-------+---------+--------+------------+---------+
The image cannot be
Copyright 2012 Oracle Corporation 26
27. Replication Administration Utility Admin
• Perform switchover to eliminate
downtime during planned
Master maintenance
• Start and stop slaves
Status &
Switchover Administration
Utility
• Slave discovery & monitoring
• Slave status, thread status
• Replication processing, including any lag
• Configure slave promotion policies
• Non GTID-functionality can be used
Slaves with pre-MySQL 5.6 releases
The image cannot be
Copyright 2012 Oracle Corporation 27
28. Switch-
Planned switchover to new master Over
with mysqlrpladmin
$ mysqlrpladmin --master=root@rod:3306
--slaves=root@jane:3306,root@jane:3307,root@freddy:3306
--new-master=root@jane:3306 --demote-master switchover
# Performing switchover from master at rod:3306 to slave at jane:3306.
# Checking candidate slave prerequisites.
# Waiting for slaves to catch up to old master.
# Stopping slaves.
# Performing STOP on all slaves.
# Demoting old master to be a slave to the new master.
# Switching slaves to new master.
# Starting all slaves.
# Performing START on all slaves.
# Checking slaves for errors.
# Switchover complete.
# ...done.
The image cannot be
Copyright 2012 Oracle Corporation 28
29. Crash-Safe Slaves & Binlog
Atomic Data • Automatic recovery of a slave and
CRASH! Binlog after a failure
Position Info Time • Binlog and table data are transactionally
consistent
Before:
– Transaction Data: in tables • Resumes replication without Dev/Op
– Replication Info: in files intervention
• Automatically rolling back replication to
Atomic Data last committed event
• Eliminates risk of data loss or
Position Info
Time corruption
MySQL 5.6
– Transaction Data: in tables
– Replication Info: in tables
The image cannot be
Copyright 2012 Oracle Corporation 29
30. Crash-Safe Slaves
• Writes and reads back only complete events or
transactions from the binary log
• master.info & relay-log.info files can be
replaced with tables
• ACID transaction prevent them getting out of sync with InnoDB
tables
--master-info-repository=TABLE
--relay-log-info-repository=TABLE
mysql.slave_master_info
mysql.slave_relay_log_info
The image cannot be
Copyright 2012 Oracle Corporation 30
32. Replication Event Checksums
• Ensures replicated data is
Master Slave correct, consistent and accessible
• Detects corrupt replication events
before they’re applied
• Returns an error
• Protects entire replication path
• Memory
# # • Disk
• Network
• Bugs
The image cannot be
Copyright 2012 Oracle Corporation 32
33. Replication Event Checksums
• Implemented in the binary and relay logs
• New mysqld options:
• binlog-checksum=CRC32
Checksums written to the binary log
• master-verify-checksum=1
Master validates checksum read from the binary log
• slave-sql-verify-checksum=1
Slave validates checksum when reading from the relay log
The image cannot be
Copyright 2012 Oracle Corporation 33
35. MySQL Utilities
• Automate common Dev/Ops tasks
• Replication: provisioning, testing, monitoring and failover
• Database comparisons: consistency checking
• Database administration: users, connections, tables, etc
• New utilities in development, ie log analysis
• Implemented as Python scripts, plug-in for MySQL
Workbench
• Also available from LaunchPad
• Extensible to include custom scripting
• Resources: Documentation & Community Forum
• On-Demand Webinar: http://t.co/7bFAV083
• http://dev.mysql.com/doc/workbench/en/mysql-utils-man.html
• http://forums.mysql.com/list.php?155
The image cannot be
Copyright 2012 Oracle Corporation 35
36. Utility Workflow for Replication
Fail-Over
Check Repl Show & Admin
• Check: Verifies pre-requisites for Replication
• Repl: Initiates Replication to the new slave
• Show: Display Replication topology
• Fail-Over & Admin: Detects and failovers (or switches)
from master to slave. Status monitoring
The image cannot be
37. Time Delayed Replication
Slave
1
Relay
Log
:3307
• Configure time period before
replication events applied to slave
Master
• Per-slave, via execution of SQL
:3306
Binary
Logs
Thread
• Second-level granularity, up to 68
Slave2
years!
:3308
Relay
Log
• Protects against operational error
• Dropping a table, etc.
• Allows database to be inspected
10 Minute Delay without loading a back-up
The image cannot be
Copyright 2012 Oracle Corporation 37
39. Time Delayed Replication – Doh!
slave2> STOP SLAVE;
Breath slowly!
master> SHOW BINLOG EVENTSG
*************************** 12. row ***************************
Log_name: ws2-bin.000001 Pos: 984 Event_type: Query Server_id: 1
End_log_pos: 1096 Info: use `clusterdb`; DROP TABLE `towns` /*
generated by server */
slave2> START SLAVE UNTIL
-> MASTER_LOG_FILE='ws2-bin.000001',
-> MASTER_LOG_POS=984;
The image cannot be
Copyright 2012 Oracle Corporation 39
40. Informational Log Events
• Simplifies debugging and auditing when using Row
Based Replication
• Original query written to the Binlog
• Replicated with row event to the slave
• Activate with
binlog-rows-query-log-events=TRUE
master> INSERT INTO simples VALUES (20),
(21),(22);
The image cannot be
Copyright 2012 Oracle Corporation 40
41. Informational Log Events
BEFORE: mysqlbinlog (-v):
### INSERT INTO clusterdb.simples
### SET
### @1=20 /* INT meta=0 nullable=0 is_null=0 */
### INSERT INTO clusterdb.simples
### SET
### @1=21 /* INT meta=0 nullable=0 is_null=0 */
### INSERT INTO clusterdb.simples
### SET
### @1=22 /* INT meta=0 nullable=0 is_null=0 */
The image cannot be
Copyright 2012 Oracle Corporation 41
42. Informational Log Events
AFTER: mysqlbinlog (-vv):
# at 443
#111128 16:04:24 server id 1 end_log_pos 504 Rows_query
# INSERT INTO simples VALUES (20),(21),(22)
# at 504
The image cannot be
Copyright 2012 Oracle Corporation 42
43. Other Useful Enhancements
• Remote BinLog Backup
• Creates real-time backup of binary log on remote server
• Eliminates requirement for:
• MySQL instance to parse binary log into SQL statements
• User needing privileges to each master for binary log
backup
• Universally Unique Server IDs
• Enables auto-discovery and analysis of replication topology
• Simplifies management of large and highly dynamic MySQL
replication environments
The image cannot be
44. How to Evaluate MySQL 5.6
http://dev.mysql.com/downloads/mysql/#downloads
Select Development Release tab
The image cannot be
Copyright 2012 Oracle Corporation 44
45. Next Steps
• Evaluate the new features
• DevZone Article:
http://dev.mysql.com/tech-resources/articles/mysql-5.6-replication.html
• Questions & Feedback: forums.mysql.com/list.php?26
• Bugs: bugs.mysql.com/
• Replication documentation
• dev.mysql.com/doc/refman/5.6/en/replication.html
The image cannot be
Copyright 2012 Oracle Corporation 45
47. Remote Binary Log Back-up
MySQL • Enhances operational efficiency
Slave • Creates real-time backup of binary
MySQL
Master log on remote server
• Adds a “raw” flag to the
Relay Log
mysqlbinlog command
• Sent via the replication channel
Backup • No need for…..
• MySQL instance to parse binary log
into SQL statements
• User needing privileges to each
Binary Log Backup
master for binary log backup
The image cannot be
Copyright 2012 Oracle Corporation 47
48. Remote Binary Log Backup
• Writes to a local file with the same name as the
original
$> mysqlbinlog --read-from-remote-
server --raw –h secret_server -P 3306
-u root mysql-bin.000001
The image cannot be
Copyright 2012 Oracle Corporation 48
49. Universally Unique Server IDs
!
• Enables auto-discovery and analysis of replication
topology
• UUID automatically generated for each new server that
becomes part of the replication cluster
• Can be viewed with systems monitoring tools, i.e.
MySQL Enterprise Monitor
• Simplifies management of large and highly dynamic
MySQL replication environments
The image cannot be
Copyright 2012 Oracle Corporation 49
50. Universally Unique Server IDs
• A UUID generated when the MySQL server first starts
• Stored in auto.cnf file
• Available as:
• server_uuid system variable for local host
• SHOW SLAVE HOSTS for UUIDs of slaves
• SHOW SLAVE STATUS for UUIDs of master
slave> SHOW SLAVE STATUSG
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
....
Master_UUID: e43ac311-19cc-11e1-a06e-002318c418fd
....
The image cannot be
Copyright 2012 Oracle Corporation 50
51. How to get Utilities
• Available on Launchpad
• https://launchpad.net/mysql-utilities
• bzr branch lp:mysql-utilities
• Requires Connector/Python
• https://launchpad.net/myconnpy
• bzr branch lp:myconnpy
• Available as a plugin in MySQL Workbench
• http://www.mysql.com/downloads/workbench/
• Documentation is here:
• http://dev.mysql.com/doc/workbench/en/mysql-
utilities.html
The image cannot be