Full tutorial of Tungsten Replicator installation and management

1. ©Continuent 2013
Using Tungsten Replicator
to solve replication
problems
Neil Armitage, Cluster implementation Engineer, Continuent
Giuseppe Maxia, QA Director, Continuent
1
1Monday, April 22, 13

2. ©Continuent 2013
ABOUT US
• Neil Armitage
• Continuent Tungsten Deployment and Support
Engineer, Continuent, Inc
• 20 years development and DB experience
• Giuseppe Maxia, a.k.a. "The Data Charmer"
• QA Director, Continuent, Inc
• 25 years development and DB experience
• long timer MySQL community member. Oracle
ACE Director
2
2Monday, April 22, 13

3. ©Continuent 2013
Tungsten replicator
• Global transaction ID
• Multiple masters
• Multiple sources
• Flexible topologies
• Parallel replication
• Heterogeneous replication
• ... and more
3
3Monday, April 22, 13

4. ©Continuent 2013
What Tungsten Replicator is NOT
• Automated management
• Automatic failover
• Transparent connections
• All the above (and more) are available with a
commercial solution named Continuent
Tungsten (a.k.a. Tungsten Enterprise)
4
4Monday, April 22, 13

5. ©Continuent 2013
What are we talking about?
• Requirements
• Components
• Installation
• Topologies
• Administration
• Troubleshooting
5
5Monday, April 22, 13

6. ©Continuent 2013
Tungsten Replicator Concepts
6
Role
service
Replicator
Master, slave, direct slave
A.k.a. "pipeline"
The replication engine
stage extract,queue,apply
6Monday, April 22, 13

7. ©Continuent 2013
Tungsten Replicator Components
7
THL
service schema
properties file
Transaction History Log
Makes the node crash proof
service definition
tools
Ruling from a centralized
location
7Monday, April 22, 13

8. ©Continuent 2013
Tungsten Replicator in a nutshell
binlog THL
slave
master
host1
host2
THL
trep_commit_seqnotrep_commit_seqnotrep_commit_seqno
origin seqno eventid
trep_commit_seqnotrep_commit_seqnotrep_commit_seqno
origin seqno eventid
global
transaction ID
8
8Monday, April 22, 13

9. ©Continuent 2013
Planning
9
• Hosts
• Topology
• Stand-alone or taking over
9Monday, April 22, 13

10. ©Continuent 2013
star
master-slave Heterogeneous
fan-in slave all-masters
MySQL
Oracle
Oracle
MySQL Oracle
Oracle
MySQL MySQL
10Monday, April 22, 13

11. ©Continuent 2013
Installation
11
11Monday, April 22, 13

12. ©Continuent 2013
Installation
• System Requirements
• Validate !rst
• Deploying from a single location
12
12Monday, April 22, 13

13. ©Continuent 2013
Installation - tools
• tools/ tungsten-installer
• tools/ con!gure-service
• tools/update
• (Using the cookbook recipes, you hardly see
them)
13
13Monday, April 22, 13

14. ©Continuent 2013
Tungsten in practice
Installation
14
14Monday, April 22, 13

15. ©Continuent 2013
Installation
• Check the requirements
• Get the binaries
• Expand the tarball
• Run cookbook
15
15Monday, April 22, 13

16. ©Continuent 2013
REQUIREMENTS
• Java JRE or JDK (Sun/Oracle or Open-jdk)
• Ruby 1.8 (only during installation)
• ssh access to the same user in all nodes
• MySQL user with all privileges
16
16Monday, April 22, 13

17. ©Continuent 2013
Installation - Choices
• --master-slave
• --direct
17
17Monday, April 22, 13

18. ©Continuent 2013
binlog
THL
THL
slave
slave
master
host1
host2
host3
THL
18
master-slave
18Monday, April 22, 13

19. ©Continuent 2013
binlog
THL
slave
master
relay log
host1
host2
host3
THL
slave
relay log
19
direct
19Monday, April 22, 13

20. ©Continuent 2013
Overview of Virtual Machines
• Copy zip !les from USB Key
• Expand on local disk
• Start all 4 Machines in VirtualBox
20
20Monday, April 22, 13

21. ©Continuent 2013
Virtual Machines
• 4 Nodes host1->host4
• Running centos 6.3 and Percona 5.5
• Root and tungsten password = ‘password’
• localhost port 2222 redirects to 22 on hosts
21
ssh
-­‐p
2222
tungsten@localhost
21Monday, April 22, 13

22. ©Continuent 2013
VERY important de!nitions
• Staging directory:
• Where you unpack the software and run the
installer.
• There is generally only one, in one host;
• Can be discarded after installation
• Installation directory:
• Where your installed software will go;
• There is one for every host;
22
22Monday, April 22, 13

23. ©Continuent 2013
Example
23
host1
host3
Staging directory:
$HOME/tungsten-replicator-2.0.8-167
host2
Installation directory:
/opt/replication
Installation directory:
/opt/replication
Installation directory:
/opt/replication
23Monday, April 22, 13

24. ©Continuent 2013
Requirements : how to
• step by step: how it happened
24
24Monday, April 22, 13

25. ©Continuent 2013
installing VMs
• Step-by-step demo
25
25Monday, April 22, 13

26. ©Continuent 2013
Overview of Tungsten
cookbook
26
26Monday, April 22, 13

27. ©Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cluster-home
+--/cookbook
+--/tools
+--/tungsten-replicator
27
27Monday, April 22, 13

28. ©Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_MASTER_SLAVE.sh
+--install_master_slave
+--show_cluster
+--test_cluster
...
28
28Monday, April 22, 13

29. ©Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_ALL_MASTERS.sh
+--install_all_masters
+--show_cluster
+--test_cluster
...
29
29Monday, April 22, 13

30. ©Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_STAR.sh
+--install_star
+--show_cluster
+--test_cluster
...
30
30Monday, April 22, 13

31. ©Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_FAN_IN.sh
+--install_fan_in
+--show_cluster
+--test_cluster
...
31
31Monday, April 22, 13

32. ©Continuent 2013
tungsten cookbook
$ cat COMMON_NODES.sh
export NODE1=host1
export NODE2=host2
export NODE3=host3
export NODE4=host4
32
32Monday, April 22, 13

33. ©Continuent 2013
tungsten cookbook
$ cat USER_VALUES.sh
# User defined values for the cluster to be
installed.
export TUNGSTEN_BASE=$HOME/installs/cookbook
export DATABASE_USER=tungsten
export BINLOG_DIRECTORY=/var/lib/mysql
export MY_CNF=/etc/my.cnf
export DATABASE_PASSWORD=secret
export DATABASE_PORT=3306
export TUNGSTEN_SERVICE=cookbook
export RMI_PORT=10000
export THL_PORT=2112
export START_OPTION=start
33
33Monday, April 22, 13

34. ©Continuent 2013
Getting started: VALIDATE FIRST
export VERBOSE=1
./cookbook/check_cookbook
./cookbook/validate_cluster
34
34Monday, April 22, 13

35. ©Continuent 2013
sample master-slave installation
• edit cookbook/COMMON_NODES.sh
• edit cookbook/USER_VALUES.sh
• run cookbook/install_master_slave
• and then:
• run cookbook/show_cluster
• run cookbook/test_cluster
35
35Monday, April 22, 13

36. ©Continuent 2013
What does the installation do
host4
1: Validate all servers
host1 host2 host3
Report all errors
36
36Monday, April 22, 13

37. ©Continuent 2013
What does the installation do
host4
1: (again) Validate all servers
host1 host2 host3
37
37Monday, April 22, 13

38. ©Continuent 2013
What does the installation do
2: install Tungsten in all servers
host3
$HOME/
tinstall/
config/
releases/
relay/
thl/
tungsten/
backups/
host4
host1
host2
38
38Monday, April 22, 13

39. ©Continuent 2013
example (from manual installation)
ssh r2 chmod 444 $HOME/tinstall
./tools/tungsten-installer
--master-slave --master-host=r1
--datasource-user=tungsten
--datasource-password=secret
--service-name=dragon
--home-directory=$HOME/tinstall
--thl-directory=$HOME/tinstall/logs
--relay-directory=$HOME/tinstall/relay
--cluster-hosts=r1,r2,r3,r4 --start
ERROR >> qa.r2.continuent.com >> /home/tungsten/
tinstall is not writeable
39
39Monday, April 22, 13

40. ©Continuent 2013
example
ssh r2 chmod 755 $HOME/tinstall
./tools/tungsten-installer
--master-slave --master-host=r1
--datasource-user=tungsten
--datasource-password=secret
--service-name=dragon
--home-directory=$HOME/tinstall
--thl-directory=$HOME/tinstall/logs
--relay-directory=$HOME/tinstall/relay
--cluster-hosts=r1,r2,r3,r4 --start
# no errors
40
40Monday, April 22, 13

41. ©Continuent 2013
After installation.
A tour of the cookbook
utilities
41
41Monday, April 22, 13

42. ©Continuent 2013
General principles (1)
42
• Scripts without extension are designed to be
launched by users
• e.g. ./cookbook/help
• ./cookbook/install_master_slave
• Scripts with extension ".sh" are either for
internal use only or deprecated.
• ./cookbook/install_* scripts can be used
before installing. Most everything else
require an installed topology
42Monday, April 22, 13

43. ©Continuent 2013
General principles (2)
43
• After installation there is a !le
CURRENT_TOPOLOGY in the staging
directory
• cookbook scripts can be used either from the
staging directory or from the installation
directory.
43Monday, April 22, 13

44. ©Continuent 2013
Cookbook tour: help and checks
44
./cookbook/check_cookbook
./cookbook/help
./cookbook/readme
44Monday, April 22, 13

45. ©Continuent 2013
Cookbook tour: Getting information
45
./cookbook/show_cluster
./cookbook/paths
./cookbook/backups
./cookbook/services
./cookbook/query_node {node} {query}
./cookbook/query_all_nodes {query}
45Monday, April 22, 13

46. ©Continuent 2013
Cookbook tour: Inspecting replication
46
./cookbook/replicator
./cookbook/trepctl
./cookbook/thl
./cookbook/show_conf
./cookbook/edit_conf
./cookbook/show_log
./cookbook/vimlog
./cookbook/emacslog
46Monday, April 22, 13

47. ©Continuent 2013
Cookbook tour: testing tools
47
./cookbook/test_cluster
./cookbook/start_load [start|stop]
./cookbook/test_all_topologies
47Monday, April 22, 13

48. ©Continuent 2013
Cookbook tour: powerful admin tools
48
./cookbook/heartbeat
./cookbook/switch
./cookbook/add_node_master_slave
./cookbook/add_node_star
./cookbook/copy_backup
./cookbook/clear_cluster # <--- CAUTION!
48Monday, April 22, 13

49. ©Continuent 2013
More installation
49
49Monday, April 22, 13

50. ©Continuent 2013
DRY-RUN
50
• Method to simulate installation;
• Does NOT perform installation;
• Does NOT even do validation;
• It only shows the commands used to install;
• Allows you to get the commands and do an
installation manually (e.g. when you can't ssh
between nodes)
50Monday, April 22, 13

51. ©Continuent 2013
DRY-RUN
51
export DRYRUN=1
./cookbook/install_master_slave
51Monday, April 22, 13

52. ©Continuent 2013
Intro to multi-master installation
52
52Monday, April 22, 13

53. ©Continuent 2013
How tungsten-installer Works for
Basic Master/Slave Deployment
53
db1
db2
db3
Staging copy
of files
check prereqs
copy code
configure
53Monday, April 22, 13

54. ©Continuent 2013
From Master/Slave Replication ...
54
db1
Replicator
db3
Service alpha
db2
Replicator
Service alpha
Replicator
Service alpha
Install master and slaves on the whole cluster
tungsten-
installer
tungsten-
installer
tungsten-
installer
54Monday, April 22, 13

55. ©Continuent 2013
To Multi-Master
55
db1 Replicator
Service alpha
Service bravo
db2
Replicator
Service bravo
Service alpha
Install master on
db1
tungsten-
installer
install master
on db2
tungsten-
installer
install slave
service on db1
con!gure-
service
install slave
service on db2
con!gure-
service
55Monday, April 22, 13

56. ©Continuent 2013
tungsten-installer master 1
56
TUNGSTEN_HOME=/home/tungsten/installs/cookbook
./tools/tungsten-installer
--master-slave
--master-host=$MASTER1
--datasource-port=3306
--datasource-user=tungsten
--datasource-password=secret
--datasource-log-directory=/var/lib/mysql
--service-name=alpha
--home-directory=$TUNGSTEN_HOME
--cluster-hosts=$MASTER1
--start
creating service 'alpha'
Notice: --cluster-hosts has only one host
56Monday, April 22, 13

57. ©Continuent 2013
tungsten-installer master 2
57
TUNGSTEN_HOME=/home/tungsten/installs/cookbook
./tools/tungsten-installer
--master-slave
--master-host=$MASTER2
--datasource-port=3306
--datasource-user=tungsten
--datasource-password=secret
--datasource-log-directory=/var/lib/mysql
--service-name=bravo
--home-directory=$TUNGSTEN_HOME
--cluster-hosts=$MASTER2
--start
creating service 'bravo'
Notice: --cluster-hosts has only one host
57Monday, April 22, 13

58. ©Continuent 2013
Con!gure Service master 1
58
TUNGSTEN_HOME=/home/tungsten/installs/cookbook
$TUNGSTEN_HOME/tungsten/tools/configure-service -C
--quiet
--host=$MASTER1
--datasource=$MASTER1
--local-service-name=alpha
--role=slave
--service-type=remote
--release-directory=$TUNGSTEN_HOME/tungsten
--skip-validation-check=THLStorageCheck
--master-thl-host=$MASTER2
--master-thl-port=2112
--svc-start bravo
Notice: bravo is the master service in host 2
58Monday, April 22, 13

59. ©Continuent 2013
Con!gure Service master 2
59
TUNGSTEN_HOME=/home/tungsten/installs/cookbook
$TUNGSTEN_HOME/tungsten/tools/configure-service -C
--quiet
--host=$MASTER2
--datasource=$MASTER2
--local-service-name=bravo
--role=slave
--service-type=remote
--release-directory=$TUNGSTEN_HOME/tungsten
--skip-validation-check=THLStorageCheck
--master-thl-host=$MASTER1
--master-thl-port=2112
--svc-start alpha
Notice: alpha is the master service in host 1
59Monday, April 22, 13

60. ©Continuent 2013
From Master/Slave Replication ...
60
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
./cooobook/install_master_slave
60Monday, April 22, 13

61. ©Continuent 2013
How Do I Install Fan-In Replication?
61
db1
Replicator
db3
Service db1
db2
Replicator
Service db2
Replicator
Service db1
Service db2
./cooobook/install_fan_in
61Monday, April 22, 13

62. ©Continuent 2013
How Do I Install Multi-Master?
62
db1 Replicator
Service db1
Service db2
db2
Replicator
Service db1
Service db2
./cooobook/install_all_masters
62Monday, April 22, 13

63. ©Continuent 2013
How Do I Extend Multi-Master?
63
db1 Replicator
Service db1
Service db2
Service db3
db3
Service db1
Service db2
Service db3
db2
Replicator
Service db1
Service db2
Service db3
Replicator
63Monday, April 22, 13

64. ©Continuent 2013
How Do I Extend Multi-Master?
64
db1
db3
Service db1
Service db2
Service db3
db2
Replicator
db4
Service db1
Service db2
Service db4
Replicator
Service db3
Service db4
Service db1
Service db2
Service db3
Replicator
Service db4
Service db1
Service db2
Service db3
Replicator
Service db4
64Monday, April 22, 13

65. ©Continuent 2013
How Do I Install a Star Topology?
65
db1
Replicator
Service db1
Service db3
db3
Service db1
Service db2
Service db3
db2
Replicator
Service db2
Service db3
HUB
Replicator
./cooobook/install_star
65Monday, April 22, 13

66. ©Continuent 2013
How Do I Extend a Star Topology?
66
db1
Replicator
Service db1
Service db3
db3
Service db1
Service db2
Service db3
db2
Replicator
Service db2
Service db3
db4
Replicator
Service db3
Service db4
HUB
Service db4
66Monday, April 22, 13

67. ©Continuent 2013
How Do I Extend a Star Topology?
67
db1
Replicator
Service db1
Service db3
db3
Service db1
Service db2
Service db3
db2
Replicator
Service db2
Service db3
db4
Replicator
Service db3
Service db4
HUB
Service db4
db5
Replicator
Service db5
Service db3
Service db5
67Monday, April 22, 13

68. ©Continuent 2013
BI-DIR: the painless way
• edit cookbook/COMMON_NODES.sh
• edit cookbook/USER_VALUES.sh
• remove two nodes
• edit the variables in cookbook/
NODES_ALL_MASTERS.sh
• cookbook/install_all_masters
68
68Monday, April 22, 13

69. ©Continuent 2013
Multiple masters
• fan-in
• Steps:
• install a master service in each node
• install a slave service for each master in the fan-
in node
• or :
• cookbook/install_fan_in
69
69Monday, April 22, 13

70. ©Continuent 2013
Multiple masters
• star topology
• Steps:
• install a master service in each server
• in the hub, install a slave service for each spoke
• in each spoke, install a slave service for the hub,
using bypass option
• cookbook/install_star
70
70Monday, April 22, 13

71. ©Continuent 2013
Taking Over from Standard
Replication
• cookbook/install_standard_replicaton
• cookbook/takeover
71
71Monday, April 22, 13

72. ©Continuent 2013
Replication Management
72
72Monday, April 22, 13

73. ©Continuent 2013
Common Commands
• replicator
• trepctl
• thl
• the Tungsten service schema
73
73Monday, April 22, 13

74. ©Continuent 2013
replicator
• It’s the service provider
• You launch it once when you start
• You may restart it when you change con!g
74
74Monday, April 22, 13

75. ©Continuent 2013
trepctl
• Tungsten Replicator ConTroLler
• It’s the driving seat for your replication
• You can start, update, and stop services
• You can get speci!c info
75
75Monday, April 22, 13

76. ©Continuent 2013
trepctl
• Tungsten Replicator Controller
• put services online or o"ine
• check status
• skip events
• inspect internals
• change roles
• heartbeat
• backup/restore
• ... and a lot more
76
76Monday, April 22, 13

77. ©Continuent 2013
thl
• Transaction History List
• Gives you access to the Tungsten transaction
history logs
77
77Monday, April 22, 13

78. ©Continuent 2013
thl
• Transaction History Log
• info
• index
• list (total or a speci!c event, or by range)
• purge
78
78Monday, April 22, 13

79. ©Continuent 2013
Tungsten service schema
• one for each service
• named "tungsten_SERVICE_NAME"
• e.g. tungsten_alpha, tungsten_dragon
• Most important table: trep_commit_seqno
79
79Monday, April 22, 13

80. ©Continuent 2013
Looking at the tungsten service db
select * from tungsten_dragon.trep_commit_seqnoG
******************* 1. row *******************
task_id: 0
seqno: 102
fragno: 0
last_frag: 1
source_id: qa.r1.continuent.com
epoch_number: 0
eventid: mysql-bin.
000002:0000000000018903;0
applied_latency: 0
update_timestamp: 2012-02-06 05:56:12
shard_id: tungsten_dragon
extract_timestamp: 2012-02-06 05:56:09
80
80Monday, April 22, 13

81. ©Continuent 2013
Where are the tools
in the tungsten directory:
$TUNGSTEN_BASE/tungsten/tungsten-replicator/bin
replicator # the daemon
trepctl # replicator controller
thl # transaction history log tool
81
81Monday, April 22, 13

82. ©Continuent 2013
Starting and stopping the replicator
cd $TUNGSTEN_BASE/tungsten/tungsten-replicator/bin
./replicator status
Tungsten Replicator Service is running (PID:32400).
./replicator stop
Stopping Tungsten Replicator Service...
Stopped Tungsten Replicator Service.
./replicator start
Starting Tungsten Replicator Service...
.... or ./cookbook/replicator ...
82
82Monday, April 22, 13

83. ©Continuent 2013
checking replicator vitals
trepctl services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: -1 # bad sign?
appliedLatency : -1.0
role : slave
serviceName : dragon
serviceType : local
started : true
state : ONLINE
Finished services command...
83
83Monday, April 22, 13

84. ©Continuent 2013
sending a heartbeat
trepctl -host $MASTER_HOST heartbeat
trepctl services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 102
appliedLatency : 3.139
role : slave
serviceName : dragon
serviceType : local
started : true
state : ONLINE
Finished services command...
84
84Monday, April 22, 13

85. ©Continuent 2013
replicator status (1)
trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysql-bin.000002:0000000000018903;0
appliedLastSeqno : 102
appliedLatency : 3.139
clusterName : default
currentEventId : NONE
currentTimeMillis : 1328504342058
dataServerHost : qa.r4.continuent.com
extensions :
latestEpochNumber : 0
masterConnectUri : thl://qa.r1.continuent.com:2112/
masterListenUri : thl://qa.r4.continuent.com:2112/
maximumStoredSeqNo : 102
minimumStoredSeqNo : 0
[...]
85
85Monday, April 22, 13

86. ©Continuent 2013
replicator status (2)
[...]
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
resourcePrecedence : 99
rmiPort : 10000
role : slave
seqnoType : java.lang.Long
serviceName : dragon
serviceType : local
simpleServiceName : dragon
siteName : default
sourceId : qa.r4.continuent.com
state : ONLINE
timeInStateSeconds : 245.215
uptimeSeconds : 245.539
Finished status command...
86
86Monday, April 22, 13

87. ©Continuent 2013
A failover scenario
1: MySQL native replication
87
87Monday, April 22, 13

88. ©Continuent 2013
1. one Master, two slaves
• Loading the “employees” test database
88
88Monday, April 22, 13

89. ©Continuent 2013
2. Master goes away
* Stop replication
* Slaves are updated at di"erent levels
# 2
select count(*) from titles
333,145
# 3
select count(*) from titles
443,308
89
89Monday, April 22, 13

90. ©Continuent 2013
3. Look into Slave #2 binary logs
• !nd the last transaction
90
90Monday, April 22, 13

91. ©Continuent 2013
4. Look into Slave #3 binary logs
1. !nd the transaction that was last in slave #2
2. Recognize that last transaction in the log of
slave #3 (This can actually take you a
LOOOONG TIME)
3. Get the position immediately after this
transaction
4. (e.g. 134000 in !le mysql-bin.000018)
91
91Monday, April 22, 13

92. ©Continuent 2013
5. promote Slave #3 to master
* in slave #2
CHANGE MASTER TO
master_host=‘slave_3_IP’,
master_user=‘slavename’,
master_password=‘slavepassword’,
master_log_file=‘mysql-bin.000018’,
master_log_pos=134000;
92
92Monday, April 22, 13

93. ©Continuent 2013
A failover scenario
1I:Tungsten Replicator
93
93Monday, April 22, 13

94. ©Continuent 2013
1. one master, two slaves
• loading the ‘employees’ test database
94
94Monday, April 22, 13

95. ©Continuent 2013
2. Master goes away
* Stop replication
* Slaves are updated at di"erent levels
# 2
select count(*) from titles
333,145
# 3
select count(*) from titles
443,308
95
95Monday, April 22, 13

96. ©Continuent 2013
3. no need to !nd the last transaction
# simply change roles
trepctl -host slave3 setrole -role master
trepctl -host slave2 setrole
-role slave -uri thl://slave3
trepctl -host slave3 online
State: ONLINE
trepctl -host slave2 online
State: GOING-ONLINE:SYNCHRONIZING
96
96Monday, April 22, 13

97. ©Continuent 2013
4. Check that the slave has
synchronized
# new master
select seqno from tungsten.trep_commit_seqno;
78
# new slave
select seqno from tungsten.trep_commit_seqno;
64
97
97Monday, April 22, 13

98. ©Continuent 2013
4. Tell the replicator to hurry up
# new master
trepctl -node slave3 flush
Master log is synchronized with database at log
sequence number: 78
# new slave
trepctl host slave2 wait -applied 78
ONLINE
select seqno from tungsten.trep_commit_seqno;
78
98
98Monday, April 22, 13

99. ©Continuent 2013
4. ... and we’re done
# new master
select count(*) from employees.titles
count(*)
443308
# new slave:
count(*)
443308
99
99Monday, April 22, 13

100. ©Continuent 2013
planned role switch
cookbook/install_master_slave
cookbook/switch
100
100Monday, April 22, 13

101. ©Continuent 2013
Switching roles in master/slave
replication (1)
101
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
✔online
✔online
✔online
101Monday, April 22, 13

102. ©Continuent 2013
Switching roles in master/slave
replication (2)
102
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine✗
✔online
✔online
102Monday, April 22, 13

103. ©Continuent 2013
Switching roles in master/slave
replication (3)
103
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine✗
✔online
✔online
Wait for transactions to be applied
103Monday, April 22, 13

104. ©Continuent 2013
Switching roles in master/slave
replication (4)
104
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine✗
o"ine✗
o"ine✗
Slaves go offline
104Monday, April 22, 13

105. ©Continuent 2013
Switching roles in master/slave
replication (5)
105
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine✗
o"ine✗
o"ine✗
Slave is promoted.
Notice: 2 masters, but
o"ine
105Monday, April 22, 13

106. ©Continuent 2013
Switching roles in master/slave
replication (6)
106
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine✗
o"ine✗
o"ine✗
old master becomes slave
106Monday, April 22, 13

107. ©Continuent 2013
Switching roles in master/slave
replication (7)
107
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine✗
o"ine✗
o"ine✗
slaves are directed to new master
107Monday, April 22, 13

108. ©Continuent 2013
Switching roles in master/slave
replication (8)
108
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
✔online
✔online
✔online
all nodes go online, using new master
108Monday, April 22, 13

109. ©Continuent 2013
Tungsten GTID vs MySQL 5.6 GTID
• What is GTID
• How it works in Tungsten
• How it works (or not) in MySQL 5.6
109
109Monday, April 22, 13

110. ©Continuent 2013
without global transaction ID
110
slave
master
slave
A
B C
commitcommitcommitcommit
binlog
position
binlog
position
position
binlog
110Monday, April 22, 13

111. ©Continuent 2013
with global transaction ID
111
slave
master
slave
A
B C
commitcommitcommitcommit
id#200
id#200id#200
111Monday, April 22, 13

112. ©Continuent 2013
Tungsten and global transaction ID:
activation
(none)
active by default
112
112Monday, April 22, 13

113. ©Continuent 2013
Tungsten and global transaction ID:
status
trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysql-bin.000002:0000000000001442;0
appliedLastSeqno : 6
appliedLatency : 0.862
clusterName : default
currentEventId : NONE
currentTimeMillis : 1354304680923
dataServerHost : qa.r4.continuent.com
113
113Monday, April 22, 13

114. ©Continuent 2013
Tungsten and global transaction ID:
seeing transactions
thl list -seqno 6
SEQ# = 6 / FRAG# = 0 (last frag)
- TIME = 2012-11-30 20:44:35.0
- EPOCH# = 0
- EVENTID = mysql-bin.000002:0000000000001442;0
- SOURCEID = qa.r1.continuent.com
- SQL(0) = insert into test.v1 values (1, 'inserted
by node #1') /* ___SERVICE___ = [cookbook] */
114
114Monday, April 22, 13

115. ©Continuent 2013
Tungsten and global transaction ID:
changing master connection
trepctl offline
trepctl online -seqno 105
115
115Monday, April 22, 13

116. ©Continuent 2013
Tungsten and Global transaction ID:
crash-safe slave tables
mysql -e 'select * from tungsten_cookbook.trep_commit_seqnoG'
*************************** 1. row ***************************
task_id: 0
seqno: 6
fragno: 0
last_frag: 1
source_id: qa.r1.continuent.com
epoch_number: 0
eventid: mysql-bin.000002:0000000000001442;0
applied_latency: 0
update_timestamp: 2012-11-30 20:44:35
shard_id: test
extract_timestamp: 2012-11-30 20:44:35
116
116Monday, April 22, 13

117. ©Continuent 2013
Tungsten and Global transaction ID:
crash-safe tables and parallel replication
mysql -e 'select seqno, source_id, shard_id,update_timestamp from
tungsten_cookbook.trep_commit_seqno'
+-------+----------------------+----------+---------------------+
| seqno | source_id | shard_id | update_timestamp |
+-------+----------------------+----------+---------------------+
| 7 | qa.r1.continuent.com | db1 | 2012-11-30 20:54:14 |
| 8 | qa.r1.continuent.com | db2 | 2012-11-30 20:54:14 |
| 9 | qa.r1.continuent.com | db3 | 2012-11-30 20:54:14 |
| 10 | qa.r1.continuent.com | db4 | 2012-11-30 20:54:14 |
| 11 | qa.r1.continuent.com | db5 | 2012-11-30 20:54:14 |
| 12 | qa.r1.continuent.com | db6 | 2012-11-30 20:54:14 |
| 13 | qa.r1.continuent.com | db7 | 2012-11-30 20:54:14 |
| 14 | qa.r1.continuent.com | db8 | 2012-11-30 20:54:14 |
| 15 | qa.r1.continuent.com | db9 | 2012-11-30 20:54:14 |
| 16 | qa.r1.continuent.com | db10 | 2012-11-30 20:54:14 |
+-------+----------------------+----------+---------------------+
117
117Monday, April 22, 13

118. ©Continuent 2013
MySQL 5.6 and global transaction ID
activation
mysqld --log-slave-updates
--gtid-mode=on
--enforce-gtid-consistency
WARNING: before MySQL 5.6.10, it was
--disable-gtid-unsafe-statements
118
118Monday, April 22, 13

119. ©Continuent 2013
MySQL 5.6 and global transaction ID
seeing transactions
#121203 11:15:49 server id 1 end_log_pos 344 CRC32 0x45b25c8f
GTID [commit=yes]
SET @@SESSION.GTID_NEXT= '7A77A490-3D3A-11E2-8CC9-7DCF9991097B:
2'/*!*/;
# at 344
#121203 11:15:49 server id 1 end_log_pos 423 CRC32 0x873c8fac
Query thread_id=3 exec_time=0 error_code=0
SET TIMESTAMP=1354533349/*!*/;
BEGIN
/*!*/;
# at 423
#121203 11:15:49 server id 1 end_log_pos 522 CRC32 0xb4bf4372
Query thread_id=3 exec_time=0 error_code=0
SET TIMESTAMP=1354533349/*!*/;
insert into t1 values (1)
119
119Monday, April 22, 13

120. ©Continuent 2013
MySQL 5.6 and global transaction ID
status
show slave statusG
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: rsandbox
Master_Port: 13233
Connect_Retry: 60
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 1837
Relay_Log_File: mysql_sandbox13234-relay-bin.000005
Relay_Log_Pos: 2047
Relay_Master_Log_File: mysql-bin.000002
...
Retrieved_Gtid_Set: 46E13434-3B28-11E2-BF47-6C626DA07446:1-7
Executed_Gtid_Set: 46E13434-3B28-11E2-BF47-6C626DA07446:1-7
120
120Monday, April 22, 13

121. ©Continuent 2013
MySQL 5.6 and global transaction ID
changing master connection
CHANGE MASTER TO master_log_file='mysql-bin-000003',
master_log_pos='1234'
# No global transaction ID is used
121
121Monday, April 22, 13

122. ©Continuent 2013
MySQL 5.6 and global transaction ID
crash-safe slave table
select * from slave_relay_log_infoG
********************* 1. row ********************
Number_of_lines: 7
Relay_log_name: ./mysql_sandbox13234-relay-bin.000005
Relay_log_pos: 2047
Master_log_name: mysql-bin.000002
Master_log_pos: 1837
Sql_delay: 0
Number_of_workers: 5
Id: 1
# NO Global transaction ID is used!
122
122Monday, April 22, 13

123. ©Continuent 2013
MySQL 5.6 and global transaction ID
crash-safe slave table + parallel
select * from mysql.slave_worker_infoG
Id: 12
Relay_log_name: ./mysql_sandbox13234-relay-bin.000007
Relay_log_pos: 4299
Master_log_name: mysql-bin.000002
Master_log_pos: 7155
Checkpoint_relay_log_name: ./mysql_sandbox13234-relay-bin.000007
Checkpoint_relay_log_pos: 1786
Checkpoint_master_log_name: mysql-bin.000002
Checkpoint_master_log_pos: 4642
Checkpoint_seqno: 9
Checkpoint_group_size: 64
Checkpoint_group_bitmap: ?
# NO Global transaction ID is used!
123
123Monday, April 22, 13

124. ©Continuent 2013
Filters
124
124Monday, April 22, 13

125. ©Continuent 2013
Tungsten Replication Service
125
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Pipeline
Master
DBMS
Transaction
History Log
In-Memory
Queue
Slave
DBMS
125Monday, April 22, 13

126. ©Continuent 2013
Restrict replication to some schemas
and tables
126
./tools/tungsten-installer
--master-slave -a
...
--svc-extractor-filters=replicate
"--property=replicator.filter.replicate.do=test,*.foo"
...
--start-and-report
# test="test.*" -> same drawback as binlog-do-db in MySQL
# *.foo = table 'foo' in any database
# employees.dept_codes,employees.salaries => safest way
126Monday, April 22, 13

127. ©Continuent 2013
Exclude some schemas and tables
from replication
127
./tools/tungsten-installer
--master-slave -a
...
--svc-extractor-filters=replicate
"--property=replicator.filter.replicate.ignore=test,*.foo"
...
--start-and-report
# test="test.*" -> same drawback as binlog-ignore-db in MySQL
# *.foo = table 'foo' in any database
# employees.dept_codes,employees.salaries => safest way
# DO NOT MIX .do and .ignore!
# (you can do it, but it may not do what you mean)
127Monday, April 22, 13

128. ©Continuent 2013
Change name of replicated schema
128
-a --svc-applier-filters=dbtransform
--property=replicator.filter.dbtransform.from_regex1=stores
--property=replicator.filter.dbtransform.to_regex1=playground
# from_regex1=stores -> name of the schema in the master
# to_regex1=playground -> name of the schema in the slave
# WARNING: requires "USE schema_name" to work properly.
128Monday, April 22, 13

129. ©Continuent 2013
Multi-master:
Con#ict prevention
129
129Monday, April 22, 13

130. ©Continuent 2012
CONFLICTS
130
130Monday, April 22, 13

131. ©Continuent 2013
What's a con#ict
• Data modi!ed by several sources (masters)
• Creates one or more :
• data loss (unwanted delete)
• data inconsistency (unwanted update)
• duplicated data (unwanted insert)
• replication break
131
131Monday, April 22, 13

132. ©Continuent 2013
Data duplication
132
id name amount
1 Joe 100
2 Frank 110
3 Sue 100
alpha
bravo
charlie
4 Matt 130
4 Matt 140
BREAKS
REPLICATION
132Monday, April 22, 13

133. ©Continuent 2013
auto_increment o$sets
are not a remedy
• A popular recipe
• auto_increment_increment +
auto_increment_offset
• They don't prevent con#icts
• They hide duplicates
133
133Monday, April 22, 13

134. ©Continuent 2013
Hidden data duplication
134
id name amount
1 Joe 100
2 Frank 110
3 Sue 100
alpha
o$set 1
bravo
o$set 2
charlie
o$set 3
13 Matt 130
11 Matt 140
INSERT
INSERT
134Monday, April 22, 13

135. ©Continuent 2013
Data inconsistency
135
id name amount
1 Joe 100
2 Frank 110
3 Sue 100
alpha
bravo
charlie
3 Sue 105
3 Sue 108
UPDATE
UPDATE
135Monday, April 22, 13

136. ©Continuent 2013
Data loss
136
id name amount
1 Joe 100
2 Frank 110
3 Sue 100
alpha
bravo
charlie
record #3
3 Sue 108
MAY BREAK
REPLICATION
UPDATE
DELETE
136Monday, April 22, 13

137. ©Continuent 2012
con#ict handling strategies
• resolving
• after the fact
• Needs information that is missing in async replication
• avoiding
• requires synchronous replication with 2pc
• preventing
• setting and enforcing a split sources policy
• Transforming and resolving
• all records are converted to INSERTs
• con"icts are resolved within a given time window
137
used by Tungsten
planned for
future use
planned for
future use
137Monday, April 22, 13

138. ©Continuent 2013
Multi-master:
Con!ict prevention
138
138Monday, April 22, 13

139. ©Continuent 2013
Tungsten con#ict prevention
in a nutshell
1. de!ne the rules
(which master can update which database)
2. tell Tungsten the rules
3. de!ne the policy
(error, drop, warn, or accept)
4. Let Tungsten enforce your rules
139
139Monday, April 22, 13

140. ©Continuent 2013
Tungsten Con#ict prevention facts
• Sharded by database
• De!ned dynamically
• Applied on the slave services
• methods:
• error: make replication fail
• drop: drop silently
• warn: drop with warning
140
140Monday, April 22, 13

141. ©Continuent 2013
Tungsten con#ict prevention
applicability
• unknown shards
• The schema being updated is not planned
• actions: accept, drop, warn, error
• unwanted shards
• the schema is updated from the wrong master
• actions: accept, drop, warn, error
• whitelisted shards
• can be updated by any master
141
141Monday, April 22, 13

142. ©Continuent 2013
Con#ict prevention directives
--svc-extractor-filters=shardfilter
replicator.filter.shardfilter.unknownShardPolicy=error
replicator.filter.shardfilter.unwantedShardPolicy=error
replicator.filter.shardfilter.enforceHomes=false
replicator.filter.shardfilter.allowWhitelisted=false
142
142Monday, April 22, 13

143. ©Continuent 2013
con#ict prevention in a star topology
143
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie (hub)
database: vehicles
A
B
CB
A
C
C
alpha updates
employees
✔
✔
143Monday, April 22, 13

144. ©Continuent 2013
con#ict prevention in a star topology
144
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie (hub)
database: vehicles
A
B
CB
A
C
C
alpha updates
vehicles
✗
144Monday, April 22, 13

145. ©Continuent 2013
con#ict prevention in a all-masters
topology
145
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie
database: vehicles
A
B
CB
A
C
C
A
B
alpha updates
employees
✔
✔
145Monday, April 22, 13

146. ©Continuent 2013
con#ict prevention in a all-masters
topology
146
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie
database: vehicles
A
B
CB
A
C
C
A
B
charlie updates
vehicles
✔
✔
146Monday, April 22, 13

147. ©Continuent 2013
con#ict prevention in a all-masters
topology
147
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie
database: vehicles
A
B
CB
A
C
C
A
B
bravo updates
employees
✗
✗
147Monday, April 22, 13

148. ©Continuent 2013
con#ict prevention in a all-masters
topology
148
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie
database: vehicles
A
B
CB
A
C
C
A
B
charlie updates
employees
✗
✗
148Monday, April 22, 13

149. ©Continuent 2013
setting con#ict prevention rules
trepctl -host host1 -service charlie
shard -insert < shards.map
cat shards.map
shard_id master critical
personnel alpha false
buildings bravo false
vehicles charlie false
test whitelisted false
# charlie is slave service in host 1
149
149Monday, April 22, 13

150. ©Continuent 2013
setting con#ict prevention rules
trepctl -host host2 -service charlie
shard -insert < shards.map
cat shards.map
shard_id master critical
personnel alpha false
buildings bravo false
vehicles charlie false
test whitelisted false
# charlie is slave service in host 2
150
150Monday, April 22, 13

151. ©Continuent 2013
setting con#ict prevention rules
trepctl -host host3 -service alpha
shard -insert < shards.map
trepctl -host host3 -service bravo
shard -insert < shards.map
cat shards.map
shard_id master critical
personnel alpha false
buildings bravo false
vehicles charlie false
test whitelisted false
# alpha and bravo are slave services in host 3
151
151Monday, April 22, 13

152. ©Continuent 2013
Con#ict prevention demo
152
• reminder
• Server #1 can update "employees"
• Server #2 can update "buildings"
• Server #3 can update "vehicles"
152Monday, April 22, 13

153. ©Continuent 2013
Sample correct operation (1)
mysql #1> create table employees.names( ... )
# all servers receive the table
# all servers keep working well
153
153Monday, April 22, 13

154. ©Continuent 2013
Sample correct operation (2)
mysql #2> create table buildings.homes( ... )
# all servers receive the table
# all servers keep working well
154
154Monday, April 22, 13

155. ©Continuent 2013
Sample incorrect operation (1)
mysql #2> create table employees.nicknames( ... )
# Only server #2 receives the table
# slave service in hub gets an error
# slave service in #1 does not receive anything
155
155Monday, April 22, 13

156. ©Continuent 2013
sample incorrect operation (2)
#3 $ trepct services | simple_services
alpha [slave]
seqno: 7 - latency: 0.136 - ONLINE
bravo [slave]
seqno: -1 - latency: -1.000 - OFFLINE:ERROR
charlie [master]
seqno: 66 - latency: 0.440 - ONLINE
156
156Monday, April 22, 13

157. ©Continuent 2013
sample incorrect operation (3)
#3 $ trepct -service bravo status
NAME VALUE
---- -----
appliedLastEventId : NONE
appliedLastSeqno : -1
appliedLatency : -1.0
(...)
offlineRequests : NONE
pendingError : Stage task failed: q-to-dbms
pendingErrorCode : NONE
pendingErrorEventId : mysql-bin.000002:0000000000001241;0
pendingErrorSeqno : 7
pendingExceptionMessage: Rejected event from wrong shard:
seqno=7 shard ID=employees shard master=alpha service=bravo
(...)
157
157Monday, April 22, 13

158. ©Continuent 2013
Fixing the issue
mysql #1> drop table if exists employees.nicknames;
mysql #1> create table if exists employees.nicknames ( ... ) ;
#3 $ trepct -service bravo online -skip-seqno 7
# all servers receive the new table
158
158Monday, April 22, 13

159. ©Continuent 2013
Sample whitelisted operation
mysql #2> create table test.hope4best( ... )
mysql #1> insert into test.hope4best values ( ... )
# REMEMBER: 'test' was explicitly whitelisted
# All servers get the new table and records
# But there is no protection against conflicts
159
159Monday, April 22, 13

160. ©Continuent 2013
administration
160
160Monday, April 22, 13

161. ©Continuent 2013
Viewing THL Events
thl info
log directory = /home/tungsten/installs/master_slave/thl/dragon/
min seq# = 0
max seq# = 101
events = 101
161
161Monday, April 22, 13

162. ©Continuent 2013
viewing THL events
thl index
LogIndexEntry thl.data.0000000001(0:102)
162
162Monday, April 22, 13

163. ©Continuent 2013
viewing THL events
thl index
[...]
LogIndexEntry thl.data.0000000001(0:18)
LogIndexEntry thl.data.0000000002(19:33)
LogIndexEntry thl.data.0000000003(34:35)
LogIndexEntry thl.data.0000000004(36:3641)
LogIndexEntry thl.data.0000000005(3642:3712)
LogIndexEntry thl.data.0000000006(3713:3838)
LogIndexEntry thl.data.0000000007(3839:3949)
LogIndexEntry thl.data.0000000008(3950:4011)
LogIndexEntry thl.data.0000000009(4012:4039)
LogIndexEntry thl.data.0000000010(4040:4057)
LogIndexEntry thl.data.0000000011(4058:4067)
LogIndexEntry thl.data.0000000012(4068:4073)
LogIndexEntry thl.data.0000000013(4074:4085)
LogIndexEntry thl.data.0000000014(4086:4095)
LogIndexEntry thl.data.0000000015(4096:4101)
LogIndexEntry thl.data.0000000016(4102:4111)
163
163Monday, April 22, 13

164. ©Continuent 2013
viewing THL events
thl list -seqno 102
[...]
SEQ# = 102 / FRAG# = 0 (last frag)
- TIME = 2012-02-06 05:56:09.0
- EPOCH# = 0
- EVENTID = mysql-bin.000002:0000000000018903;0
- SOURCEID = qa.r1.continuent.com
- METADATA =
[mysql_server_id=10;is_metadata=true;service=dragon;shard=tung
sten_dragon;heartbeat=NONE]
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent
- OPTIONS = [##charset = ISO8859_1, autocommit = 1, sql_auto_is_null = 1, foreign_key_checks = 1,
unique_checks = 1, sql_mode = 'IGNORE_SPACE', character_set_client = 8, collation_connection = 8,
collation_server = 8]
- SCHEMA = tungsten_dragon
- SQL(0) = UPDATE tungsten_dragon.heartbeat SET source_tstamp=
"2012-02-06 05:56:09", salt= 2, name= "NONE" WHERE id= 1 /*
___SERVICE___ = [dragon] */
164
164Monday, April 22, 13

165. ©Continuent 2013
Skipping a THL Event
trepctl online -skip-seqno 1092
trepctl online -skip-seqno 1092,1093,1094
# see example
165
165Monday, April 22, 13

166. ©Continuent 2013
Adding a Member
• Let's see the cookbook, and use it
166
166Monday, April 22, 13

167. ©Continuent 2013
parallel replication
167
167Monday, April 22, 13

168. Replicator Pipeline Architecture
THL Slave
DBMS
Transaction
History Log
MySQL
Binlog
shard.list
file
ApplyExtract Extract
Pipeline
Tungsten Replicator Process
Stage
ApplyExtract
ApplyExtract
ApplyExtract
Parallel
Queue
Assign
Shard
ID
Apply
StageStage
“channels”
168Monday, April 22, 13

169. ©Continuent 2013
Parallel replication facts
✓Sharded by database
✓Good choice for slave lag problems
❖Bad choice for single database projects
169
169Monday, April 22, 13

170. Parallel Replication test
binary logs
MySQL slave
Tungsten slave
OFFLINE
STOPPED
replicator alpha
direct:
alpha
(slave)
Concurrent sysbench
on 30 databases
running for 1 hour
TOTAL DATA: 130 GB
RAM per server: 20GB
Slaves will have 1 hour lag
170Monday, April 22, 13

171. measuring results
binary logs
MySQL slave
Tungsten slave
ONLINE
START
replicator alpha
direct:
alpha
(slave)
Recording
catch-up time
171Monday, April 22, 13

172. MySQL native
replication
slave catch up in 04:29:30
172Monday, April 22, 13

173. Tungsten parallel
replication
slave catch up in 00:55:40
173Monday, April 22, 13

174. Parallel replication made simpler
FROM HERE ....
174Monday, April 22, 13

175. Parallel replication made simpler
TO HERE
175Monday, April 22, 13

176. Parallel replication made simpler
176Monday, April 22, 13

177. ©Continuent 2013
parallel replication
direct slave facts
✓No need to install Tungsten on the master
✓Tungsten runs only on the slave
✓Replication can revert to native slave with two
commands (trepctl offline; start slave)
✓Native replication can continue on other slaves
❖Failover (either native or Tungsten) becomes a manual
task
177
177Monday, April 22, 13

178. ©Continuent 2013
installing parallel replication
• MORE_OPTIONS='--channels=10'
• ./cookbook/install_master_slave
178
178Monday, April 22, 13

179. ©Continuent 2013
Checking parallel replication
trepctl status
trepctl status -name tasks
trepctl status -name shards
trepctl status -name stores
179
179Monday, April 22, 13

180. ©Continuent 2013
Parallel replication demo
180
180Monday, April 22, 13

181. ©Continuent 2013
Troubleshooting
181
181Monday, April 22, 13

182. ©Continuent 2013
Identify the Failed Component
• Steps
1. trepctl services
2. trepctl -service SVC_NAME status
3. look at the logs
4. Take action
182
182Monday, April 22, 13

183. ©Continuent 2013
reading the logs
ls $TUNGSTEN_BASE/tungsten/tungsten-replicator/logs/
trepsvc.log user.log
...
or
./cookbook/show_log
# let's see it in practice
183
183Monday, April 22, 13

184. ©Continuent 2013
Parting thoughts
184
184Monday, April 22, 13

185. ©Continuent 2013
Open source Tungsten
Replicator now includes
Oracle-to-MySQL and
Oracle-to-Oracle extractors
and appliers!
185
185Monday, April 22, 13

186. ©Continuent 2012 186
Continuent Website:
http://www.continuent.com
Tungsten Replicator 2.0:
http://code.google.com/p/tungsten-replicator
Our Blogs:
http://scale-out-blog.blogspot.com
http://datacharmer.blogspot.com
http://flyingclusters.blogspot.com
560 S.Winchester Blvd., Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: sales@continuent.com
186Monday, April 22, 13