Arquiteturas avançadas e usos curiosos do recurso nativo de replicação do MySQL. Também são detalhadas algumas novidades da versão 5.6 que elevam a Replicação MySQL a outro nível.

1. 1 Copyright © 2011, Oracle and/or its affiliates. All rights Insert Information Protection Policy Classification from Slide 8
reserved.

2. ADVANCED MYSQL REPLICATION ARCHITECTURES
Luís Soares Lars Thalmann, Development Director,
Sr. Software Developer Replication, Connectors and Backup
MySQL Replication Team Lead Mats Kindahl, Lead Software Dev, Replication
2 Copyright © 2011, Oracle and/or its affiliates. All rights Insert Information Protection Policy Classification from Slide 8
Oracle Open World Latin America 2011
reserved.

3. 3 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

4. THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL
PRODUCT DIRECTION. IT IS INTENDED FOR INFORMATION
PURPOSES ONLY, AND MAY NOT BE INCORPORATED INTO
ANY CONTRACT. IT IS NOT A COMMITMENT TO DELIVER
ANY MATERIAL, CODE, OR FUNCTIONALITY, AND SHOULD
NOT BE RELIED UPON IN MAKING PURCHASING DECISION.
THE DEVELOPMENT, RELEASE, AND TIMING OF ANY
FEATURES OR FUNCTIONALITY DESCRIBED FOR
ORACLE'S PRODUCTS REMAINS AT THE SOLE DISCRETION
OF ORACLE.
4 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

5. Agenda
● MySQL Replication Basics
● Traditional Architectures
● Load Balancing
● Data Aggregation and Multi-source Replication
● Hierarchical Replication
● Data Integration
● Advanced Replication Architectures Enablers
5 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

6. MySQL Replication Basics
6 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

7. Components
● Master
● Changes data
● Logs changes (events) into a file (the binary log)
● Slave
● Retrieves events from the master
● Replays the events on the slaves databases
● Binary Log
● Records every change on the master (in either format: row or statement)
● Split into transactional groups
7 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

8. Big Picture
Session
Session Dump I/O SQL
Session
Binary log Relay log
Master Slave
8 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

9. Propagation of Changes
● Asynchronous Replication
● Transactions committed immediately
● Events are propagated after the commit operation is acknowledged
● Faster but vulnerable to lost updates on server crashes and
inconsistency
● Built into the server
● Semi-synchronous Replication
● Master commits transaction but waits for N slaves to acknowledge
having received and stored the correspondent event before replying to
the client
9 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

10. Traditional Architectures
10 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

11. Single Master, Single Slave
● Backups
Master ● Reports
● Disaster Recovery (geographically distant
servers)
● Recovering from Human errors (Time-delayed
replication)
Slave ● Add more slaves seamlessly
11 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

12. Scaling-out read operations
● Off-loading the master
● Different type of queries routed to
Master different servers
Writes
● Different hardware profiles for master
and slaves
Client
● SSD for enhancing master
performance
Slave ● Large RAM / caches to enhance
Slave
Slave Slaves performance
Reads
● Load Balancing
● Query routing policies
12 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

13. Relay Server
● Relay Slave.
Master ● Blackhole storage engine.
● No data stored on the relay slave.
● Reduce load at the master.
Relay
● Improved master side filtering.
● Sensitive data can be kept only at one
S1
S1
Slave N ●
physical server.
Relay has binary log active.
13 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

14. High Availability
Virtula IP Manager
Master Master Master Master
Shared/replicated disk
Dual Masters Binlog Binlog
Slave
Master Master
Active/Passive
Shared disk
Circular
Replication Master Master
14 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

15. High Availability
● High Availability: fail over
● Servers can crash (hardware, software or even power failure).
● Services should not.
● Dual masters, circular replication (conflict free partition workload on each).
● Seamless fail-over of affected partitions
● Scaling out with slaves.
● Ready to step up and replace a failed master (on dual masters watch out
whether the slave is already ahead of second master – slave promotion
instead)
● Active / Passive – binlog is shared by the two master's, on fail-over binlog positions
match
15 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

16. Load Balancing
16 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

17. Load Balancing Writes to a
Master master.
Client
Reads from a
pool of slaves.
Which one?
Slave Slave Slave
17 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

18. Load Balancing: Application Level Writes to a
Master master
Client
Which server should
I use for this query?
LB
Slave Slave Slave
18 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

19. Load Balancing: Intermediate Proxy Writes to a
Master master
LB
Client
Reads from a
slave in a pool
of slaves.
Read/Write Split
Slave Slave Slave
19 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

20. Load Balancing
● Requires monitoring tools (status and performance)
● Servers crash every now and then!
● Requires intelligent query routing (reads vs writes)
● Which queries go where?
● Session consistency is something to keep in mind!
● Enter MySQL Proxy, PHP mysqlnd, ...
● http://dev.mysql.com/doc/refman/5.5/en/mysql-proxy.html
● http://dev.mysql.com/downloads/connector/php-mysqlnd
● http://blog.ulf-wendel.de/?p=320
20 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

21. Hierarchical Replication
21 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

22. Hierarchical Replication
Master
Writes
Client
Master becomes
overloaded!
Reads
Hundreds of slaves connected to the master.
...
22 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

23. Hierarchical Replication
Master
Writes
Client
Relay Relay
Slave Slave
Reads
Leaf or Relay
Servers
Hundreds of slaves more down the hierarchy ...
23 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

24. Hierarchical Replication
● Scale-out hundreds of servers
● Blackhole storage engine on relays.
● Relay servers have binary logs ON, leaf slaves do not need it.
● Group data domains/partitions
● Sensitive information routed through parts of the hierarchy.
● Event filtering.
24 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

25. Hierarchical Replication
● Hierarchy-wide consistency!
● Asynchronous propagation => writes take time to propagate from master
to slaves: reads on a slave may show stale data.
● Event positions are filename and offset => different on every intermediate
slave.
● Wait for data to propagate.
● global transaction identifiers.
● poll each relay server down the chain until data has propagated.
25 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

26. Data Aggregation: Multi-source Replication
26 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

27. Multi-source Replication
● Why?
● Aggregate data from different masters, different data-centers, different clusters
● How?
● Time-share Replication
● Typically round-robin policy
● Aggregating data from clusters in different timezones
● Slave controlled by specific client
● You can do it all at SQL level
● Inter-cluster
● Requirements: Conflict free load
27 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

28. Multi-source Replication
Master 1 Master 2 Master 3
Conflicts?
Replication
Stream is
Single
Threaded
Binlog
Slave Position
Bookkeeping
28 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

29. Multi-source Replication: Time share
Master 1 Master 2 Master 3
No conflict resolution, so let
masters update different data
Control
Client Slave
29 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

30. Multi-source Replication: Control Client
import itertools
position = {}
def round_robin_multi_master(slave, masters):
current = masters[0]
for master in itertools.cycle(masters):
slave.sql("STOP SLAVE IO_THREAD");
mysqlrep.wait_for_empty_relay_log(slave)
slave.sql("STOP SLAVE SQL_THREAD");
position[current.name] = mysqlrep.fetch_slave_position(slave)
slave.change_master(position[current.name])
master.sql("START SLAVE")
current = master
sleep(60) # Sleep 1 minute
30 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

31. Inter-cluster Multi-source Replication
MySQL MySQL
Cluster Cluster
Server Server
Again, no conflict resolution,
so let masters update different
data
Server Server
MySQL
Cluster
31 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

32. Data Integration
32 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

33. Integrate Data Into Other Systems
HBASE Data Mining
Master
Client Dump
Dump
Client Dump ?
Client
Binlog
MySQL
SOLR
Slave
Full Text
Search
33 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

34. Integrate Data Into Other Systems
via mysql binary log API
Handles binlog events,
translates them and
Dump Server pushes data to SOLR
A
P Transformer SOLR
I
File
Binlog Connects to and reads
from the binlog source
(either server or file)
34 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

35. Integrate Data Into Other Systems
via mysql binary log API
Event flow in the
binary log API
library when reading
from a dump thread.
35 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

36. Integrate Data Into Other Systems
● Library to process replication events
● API ready to use
● Simple, extensible, efficient
● Just hook a client thread in the wait_for_next_event handler
● It's out there:
● https://launchpad.net/mysql-replication-listener
● http://labs.mysql.com
36 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

37. Advanced Replication Architectures Enablers
37 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

38. Replication Metadata on System Tables 5.6
● Crash-safe (replication metadata stored in DMR
transactional tables)
● Robust, highly available setups
● SQL interface to metadata
● Access data through regular user session
● Self-contained: time-shared multi-source VS
easily implemented in SQL and in the server
● Make use of special features like MySQL
events, stored procedures, triggers, etc...
38 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

39. Global Transaction Identifiers
● Event positions in the binary log are a tuple: <filename, file offset>
● Not very intuitive in a large hierarchical topology
● Makes it hard to get most recent transaction - fail-over
● Global transaction identifiers: <server_id, tx_seqno>
● Makes hierarchical consistency checks and fail-over close to trivial
● Early feature preview available on MySQL labs:
● http://labs.mysql.com
39 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

40. En
MySQL Utilities: What is it? t
Re erp
ad ris
y! e
● A collection of Python utilities for managing MySQL databases
● Part of MySQL Workbench 5.2.31
● Available under the GPLv2 license
● Written in Python
● Easily enhanced using a growing code library
● Python library to grow solutions for common administrative problems
40 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

41. En
MySQL Utilities: What is it? t
Re erp
ad ris
y! e
Scripts mysqlprocgrep
Command Common
Module Module
mysql.utilities.command mysql.utilities.common
MySQL Utilities Library
41 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

42. En
MySQL Utilities: What can I do with it? t
Re erp
ad ris
y! e
● Easily administer MySQL servers from the command line
●
mysqldbcompare – compare databases
●
mysqldbcopy – copy databases between servers
●
mysqlprocgrep – search process information
●
mysqlrplshow – show a graph of your topology
●
mysqlreplicate – setup replication
●
mysqlrplcheck – check replication configuration
●
...
● Build your own tools on top of the core of the library, e.g., automate timeshare multi-
source replication setup or failing-over to a slave
42 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

43. En
MySQL Utilities: Where to go from here? t
Re erp
ad ris
y! e
Download MySQL Workbench from:
http://www.mysql.com/downloads/workbench/
You can also download the latest development source code tree for the
MySQL Workbench Utilities from:
http://launchpad/net/mysql-utilities
43 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

44. MySQL Proxy
● Simple program in-between client and mysql server
● Monitor, analyze, transform, load balance
● Use cases and possibilities are enormous
● Available at:
● http://dev.mysql.com/doc/refman/5.5/en/mysql-proxy.html
44 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

45. The Binary Log API
● A C++ library used for connecting to a MySQL server and process the
replication stream as a slave.
● Enables complex setups and data streaming to heterogeneous slaves.
● Available at:
● https://launchpad.net/mysql-replication-listener
● http://labs.mysql.com (binaries)
45 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

46. Summary
46 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

47. Summary
● MySQL replication concepts and components: replication is simple
● Popular MySQL replication use case scenarios: replication is flexible and versatile
● Scenarios that take MySQL replication to a whole new more advanced level:
● Load Balancing
● Hierarchical replication
● Data replication and aggregation
● Data integration
● MySQL features that are enablers for more advanced replication scenarios
47 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

48. Q&A
48 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

49. 49 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

50. 50 Copyright © 2011, Oracle and/or its affiliates. All rights Insert Information Protection Policy Classification from Slide 8
reserved.