SlideShare a Scribd company logo
1 of 45
Download to read offline
Preventing con!icts in
                          Multi-master replication
                               with Tungsten

                             Giuseppe Maxia, QA Director, Continuent



   ©Continuent 2012.                  1


Sunday, February 03, 13
Introducing Continuent

                    •     The leading provider of clustering and
                          replication for open source DBMS

                    •     Our Product: Continuent Tungsten
                          • Clustering - Commercial-grade HA, performance
                             scaling and data management for MySQL
                          • Replication - Open source, Flexible, high-
                             performance data movement




   ©Continuent 2012                              2


Sunday, February 03, 13
What is Tungsten Replicator?
       •      Open source drop-in replacement for MySQL replication,
              providing:
             • Global transaction ID
             • Multiple masters
             • Multiple sources
             • Flexible topologies
             • Heterogeneous replication
             • Parallel replication
             • ... and more

   ©Continuent 2012.                   3


Sunday, February 03, 13
Tungsten Replicator Overview
                                                 Master
                                                 Replicator           THL
                           Download
                          transactions
                          via network                         (Transactions + Metadata)


               DBMS
               Logs
                                         Slave   Replicator           THL

                                Apply using JDBC              (Transactions + Metadata)




   ©Continuent 2012                                   4


Sunday, February 03, 13
Tungsten Replication Service

                                              Pipeline
                      Stage                    Stage                    Stage
        Extract Filter        Apply    Extract Filter   Apply    Extract Filter   Apply




         Master                 Transaction                In-Memory              Slave
         DBMS                   History Log                  Queue                DBMS




   ©Continuent 2012                                5


Sunday, February 03, 13
Multiple Services Per Replicator

               NYC
                          Replicator
                            Service nyc
                                              Replicator
                                                Service nyc

                                                Service fra
         Frankfurt
                          Replicator
                            Service fra                       London




   ©Continuent 2012                       6


Sunday, February 03, 13
Multi-Master Replication

                    •     Updates on 2+ sites (active-active mode)

                    •     Enables geographic distribution of data

                    •     No failover necessary if network fails or site
                          becomes unavailable

                    •     Not all applications can handle multi-master
                          •   Applications must avoid con!icts

                          •   Careful testing required

                          •   Restoration of broken systems may not be easy



   ©Continuent 2012                                7


Sunday, February 03, 13
Simple Multi-Master Con!guration

              NYC         Replicator                          Frankfurt
                                              Replicator
                            fra (slave)        fra (master)

                           nyc (master)         nyc (slave)




                               Database-to-Database




   ©Continuent 2012                       8


Sunday, February 03, 13
Three-node
               Multi-Master Con!guration
            NYC           Replicator                      Replicator
                                                                          Frankfurt
                            fra (slave)                    fra (master)

                           nyc (master)                     nyc (slave)

                            lon (slave)                     lon (slave)



                                                              Database-to-Database
                                          lon (master)

                                            fra (slave)

                                           nyc (slave)
                          London
                                          Replicator

   ©Continuent 2012                             9


Sunday, February 03, 13
multi-node star Con!guration
               NYC            Replicator                       Replicator
                                                                                   Frankfurt
                               nyc (master)                     fra (master)

                                   lon (slave)                   lon (slave)




                                                                  Replicator
                                          lon (master)                               Singapore
                                                                    sin (master)
                                            fra (slave)
                                                                    lon (slave)
                                            nyc (slave)
                    London (hub)            sin (slave)

                                         Replicator

   ©Continuent 2012                                       10


Sunday, February 03, 13
CONFLICTS


   ©Continuent 2012           11


Sunday, February 03, 13
What's a con"ict

                    •     Data modi!ed by several sources (masters)

                    •     Creates one or more :
                          • data loss (unwanted delete)
                          • data inconsistency (unwanted update)
                          • duplicated data (unwanted insert)
                          • replication break




   ©Continuent 2012.                            12


Sunday, February 03, 13
Data duplication
                                4        Matt         140



                                    id      name amount           bravo
             alpha                   1       Joe  100
                                     2      Frank 110
                                     3       Sue  100

                                                                    BREAKS
                                                                  REPLICATION
                                     4          Matt        130
                          charlie
   ©Continuent 2012.                             13


Sunday, February 03, 13
auto_increment o#sets
               are not a remedy
                    •     A popular recipe
                          • auto_increment_increment +
                            auto_increment_offset

                    •     They don't prevent con"icts

                    •     They hide duplicates




   ©Continuent 2012.                         14


Sunday, February 03, 13
Hidden data duplication
                          11        Matt         140

                                INSERT
                                                              bravo
          alpha                id      name amount           o#set 2
         o#set 1                1       Joe  100
                                2      Frank 110
                                3       Sue  100

                                      INSERT

                     charlie    13         Matt        130
                     o#set 3
   ©Continuent 2012.                        15


Sunday, February 03, 13
Data inconsistency
                                3        Sue         108

                                    UPDATE

                                    id     name amount           bravo
             alpha                   1      Joe  100
                                     2     Frank 110
                                     3      Sue  100

                                          UPDATE
                                     3         Sue         105
                          charlie
   ©Continuent 2012.                            16


Sunday, February 03, 13
Data loss
                                3        Sue        108

                                    UPDATE

                                    id     name amount    bravo
             alpha                   1      Joe  100
                                     2     Frank 110
                                     3      Sue  100

                                          DELETE           MAY BREAK
                                                          REPLICATION
                                           record #3
                          charlie
   ©Continuent 2012.                           17


Sunday, February 03, 13
con"ict handling strategies
                    •     resolving
                          •   after the fact

                          •   Needs information that is missing in async replication

                    •     avoiding
                          •   requires synchronous replication with 2pc

                    •     preventing                                         used by Tungsten

                          •   setting and enforcing a split sources policy

                    •     Transforming and resolving                              planned for
                          •   all records are converted to INSERTs                  future use

                          •   con!icts are resolved within a given time window

   ©Continuent 2012                                 18


Sunday, February 03, 13
Tungsten Replicator Filters




   ©Continuent 2012.         19


Sunday, February 03, 13
Replicator Pipeline Architecture

                                   Tungsten Replicator Process
                                                   Pipeline
                          Stage                          Stage                  Stage
                          Assign
           Extract        Shard      Apply     Extract    Apply           Extract       Apply
                            ID



                                               Transaction
                                               History Log

         MySQL               filter           THL                  filter       filter
         Binlog                                                                       Slave
                                                                                      DBMS




Sunday, February 03, 13
Restrict replication to some schemas
               and tables
  ./tools/tungsten-installer 
     --master-slave -a 
    ...
    --svc-extractor-filters=replicate 
    "--property=replicator.filter.replicate.do=test,*.foo" 
    ...
    --start-and-report


  # test="test.*" -> same drawback as binlog-do-db in MySQL
  # *.foo = table 'foo' in any database
  # employees.dept_codes,employees.salaries => safest way




   ©Continuent 2012.              21


Sunday, February 03, 13
Multi-master:
                Con!ict prevention


   ©Continuent 2012.      22


Sunday, February 03, 13
Tungsten con"ict prevention
               in a nutshell
                1. de!ne the rules
                          (which master can update which database)

                2. tell Tungsten the rules
                3. de!ne the policy
                          (error, drop, warn, or accept)

                4. Let Tungsten enforce your rules




   ©Continuent 2012.                               23


Sunday, February 03, 13
Tungsten Con"ict prevention facts

                    •     Sharded by database

                    •     De!ned dynamically

                    •     Applied on the slave services

                    •     methods:
                          • error: make replication fail
                          • drop: drop silently
                          • warn: drop with warning


   ©Continuent 2012.                              24


Sunday, February 03, 13
Tungsten con"ict prevention
               applicability
                    •     unknown shards
                          • The schema being updated is not planned
                          • actions: accept, drop, warn, error
                    •     unwanted shards
                          • the schema is updated from the wrong master
                          • actions: accept, drop, warn, error
                    •     whitelisted shards
                          • can be updated by any master

   ©Continuent 2012.                             25


Sunday, February 03, 13
Con"ict prevention directives
  --svc-extractor-filters=shardfilter

  replicator.filter.shardfilter.unknownShardPolicy=error

  replicator.filter.shardfilter.unwantedShardPolicy=error

  replicator.filter.shardfilter.enforceHomes=false

  replicator.filter.shardfilter.allowWhitelisted=false




   ©Continuent 2012.              26


Sunday, February 03, 13
con"ict prevention in a star topology
        alpha updates                                               H
          employees                          Host1                  m
                                   A         master: alpha          d
                                        C    database: employees
                                                                    H
                                                                    m
                                                                    d

                           ✔   A            Host3
                                            master: charlie (hub)
                               B    C
                                            database: vehicles

   Host2

                                   ✔
   master: bravo
   database: buildings B   C

   ©Continuent 2012.               27


Sunday, February 03, 13
con"ict prevention in a star topology
        alpha updates                                               H
           vehicles                          Host1                  m
                                   A         master: alpha          d
                                        C    database: employees
                                                                    H
                                                                    m
                                                                    d



                          ✗    A            Host3
                                            master: charlie (hub)
                               B    C
                                            database: vehicles

   Host2
   master: bravo
   database: buildings B   C

   ©Continuent 2012.               28


Sunday, February 03, 13
con"ict prevention in a all-masters
               topology
                                                                           H
       alpha updates                                Host1                  m
         employees                        A         master: alpha          d
                                  B
                                               C    database: employees
                                                                           H
                                                                           m
                                                                           d

                          ✔           A            Host3
                                                   master: charlie (hub)
                                           C
                          ✔
                                      B
                                                   database: vehicles
                              A
   Host2
   master: bravo
   database: buildings B      C

   ©Continuent 2012.                      29


Sunday, February 03, 13
con"ict prevention in a all-masters
               topology
                                                                           H
      charlie updates                               Host1                  m
         vehicles                         A         master: alpha          d
                                  B
                                               C    database: employees
                                                                           H
                                                                           m
                                                                           d

                          ✔           A            Host3
                                                   master: charlie (hub)
                                           C
                          ✔
                                      B
                                                   database: vehicles
                              A
   Host2
   master: bravo
   database: buildings B      C

   ©Continuent 2012.                      30


Sunday, February 03, 13
con"ict prevention in a all-masters
               topology
                                                                           H
       alpha updates                                Host1                  m
          buildings                       A         master: alpha          d
                                  B
                                               C    database: employees
                                                                           H


                          ✗
                                                                           m
                                                                           d
                                      A            Host3



   Host2
                          ✗   A
                                      B    C       master: charlie (hub)
                                                   database: vehicles


   master: bravo
   database: buildings B      C

   ©Continuent 2012.                      31


Sunday, February 03, 13
con"ict prevention in a all-masters
               topology
                                                                           H
      charlie updates                               Host1                  m
        employees                         A         master: alpha          d
                                  B
                                               C    database: employees
                                                                           H


                          ✗
                                                                           m
                                                                           d
                                      A            Host3



   Host2
                          ✗   A
                                      B    C       master: charlie (hub)
                                                   database: vehicles


   master: bravo
   database: buildings B      C

   ©Continuent 2012.                      32


Sunday, February 03, 13
setting con"ict prevention rules
          trepctl -host host1 -service charlie 
            shard -insert < shards.map

          cat shards.map
          shard_id          master        critical
          personnel         alpha         false
          buildings         bravo         false
          vehicles          charlie       false
          test              whitelisted   false


          # charlie is slave service in host 1


   ©Continuent 2012.              33


Sunday, February 03, 13
setting con"ict prevention rules
          trepctl -host host2 -service charlie 
            shard -insert < shards.map

          cat shards.map
          shard_id          master        critical
          personnel         alpha         false
          buildings         bravo         false
          vehicles          charlie       false
          test              whitelisted   false


          # charlie is slave service in host 2


   ©Continuent 2012.              34


Sunday, February 03, 13
setting con"ict prevention rules
          trepctl         -host host3 -service alpha 
            shard         -insert < shards.map
          trepctl         -host host3 -service bravo 
            shard         -insert < shards.map

          cat shards.map
          shard_id                  master        critical
          personnel                 alpha         false
          buildings                 bravo         false
          vehicles                  charlie       false
          test                      whitelisted   false

          # alpha and bravo are slave services in host 3

   ©Continuent 2012.                      35


Sunday, February 03, 13
Con"ict prevention demo

       •       reminder

       •       Server #1 can update "employees"

       •       Server #2 can update "buildings"

       •       Server #3 can update "vehicles"




   ©Continuent 2012.                   36


Sunday, February 03, 13
Sample correct operation (1)
          mysql #1> create table employees.names( ... )


          # all servers receive the table
          # all servers keep working well




   ©Continuent 2012.              37


Sunday, February 03, 13
Sample correct operation (2)
          mysql #2> create table buildings.homes( ... )


          # all servers receive the table
          # all servers keep working well




   ©Continuent 2012.              38


Sunday, February 03, 13
Sample incorrect operation (1)
          mysql #2> create table employees.nicknames( ... )


          # Only server #2 receives the table
          # slave service in hub gets an error
          # slave service in #1 does not receive anything




   ©Continuent 2012.              39


Sunday, February 03, 13
sample incorrect operation (2)
          #3 $ trepct services | simple_services
          alpha    [slave]
          seqno:           7 - latency:   0.136 - ONLINE

          bravo           [slave]
          seqno:                -1   - latency:        -1.000 - OFFLINE:ERROR

          charlie         [master]
          seqno:                66   - latency:         0.440 - ONLINE




   ©Continuent 2012.                              40


Sunday, February 03, 13
sample incorrect operation (3)
          #3 $ trepct -service bravo status
          NAME                     VALUE
          ----                     -----
          appliedLastEventId     : NONE
          appliedLastSeqno       : -1
          appliedLatency         : -1.0
          (...)
          offlineRequests        : NONE
          pendingError           : Stage task failed: q-to-dbms
          pendingErrorCode       : NONE
          pendingErrorEventId    : mysql-bin.000002:0000000000001241;0
          pendingErrorSeqno      : 7
          pendingExceptionMessage: Rejected event from wrong shard:
          seqno=7 shard ID=employees shard master=alpha service=bravo
          (...)



   ©Continuent 2012.                   41


Sunday, February 03, 13
Fixing the issue
          mysql #1> drop table if exists employees.nicknames;
          mysql #1> create table if exists employees.nicknames ( ... ) ;

          #3 $ trepct -service bravo online -skip-seqno 7




          # all servers receive the new table




   ©Continuent 2012.                   42


Sunday, February 03, 13
Sample whitelisted operation
          mysql #2> create table test.hope4best( ... )

          mysql #1> insert into test.hope4best values ( ... )


          # REMEMBER: 'test' was explicitly whitelisted
          # All servers get the new table and records
          # But there is no protection against conflicts




   ©Continuent 2012.              43


Sunday, February 03, 13
Parting thoughts


                We are hiring !

                http://continuent.com/about/careers


   ©Continuent 2012.             44


Sunday, February 03, 13
blog: http://datacharmer.blogspot.com
                          twitter: @datacharmer



                                 Continuent Website:
                              http://www.continuent.com

                               Tungsten Replicator 2.0:
                             http://tungsten-replicator.org
   ©Continuent 2012                          45


Sunday, February 03, 13

More Related Content

What's hot

Session 49 - Semantic metadata management practical
Session 49 - Semantic metadata management practical Session 49 - Semantic metadata management practical
Session 49 - Semantic metadata management practical
ISSGC Summer School
 
9/28/11 Slides - Introduction to DuraCloud, Slides
9/28/11 Slides - Introduction to DuraCloud, Slides9/28/11 Slides - Introduction to DuraCloud, Slides
9/28/11 Slides - Introduction to DuraCloud, Slides
DuraSpace
 

What's hot (20)

High speed networks and Java (Ryan Sciampacone)
High speed networks and Java (Ryan Sciampacone)High speed networks and Java (Ryan Sciampacone)
High speed networks and Java (Ryan Sciampacone)
 
Scaling Rails Applications In The Cloud
Scaling Rails Applications In The CloudScaling Rails Applications In The Cloud
Scaling Rails Applications In The Cloud
 
Cloud Foundry for Java devs
Cloud Foundry for Java devsCloud Foundry for Java devs
Cloud Foundry for Java devs
 
Large customers want postgresql too !!
Large customers want postgresql too !!Large customers want postgresql too !!
Large customers want postgresql too !!
 
3 Networking CloudStack Developer Day
3  Networking CloudStack Developer Day 3  Networking CloudStack Developer Day
3 Networking CloudStack Developer Day
 
Session 49 - Semantic metadata management practical
Session 49 - Semantic metadata management practical Session 49 - Semantic metadata management practical
Session 49 - Semantic metadata management practical
 
The djatoka Image Server
The djatoka Image ServerThe djatoka Image Server
The djatoka Image Server
 
Toward a practical “HPC Cloud”: Performance tuning of a virtualized HPC cluster
Toward a practical “HPC Cloud”: Performance tuning of a virtualized HPC clusterToward a practical “HPC Cloud”: Performance tuning of a virtualized HPC cluster
Toward a practical “HPC Cloud”: Performance tuning of a virtualized HPC cluster
 
Cloud Foundry: Inside the Machine
Cloud Foundry: Inside the MachineCloud Foundry: Inside the Machine
Cloud Foundry: Inside the Machine
 
Presentation
PresentationPresentation
Presentation
 
Session9part2 Servers Detailed
Session9part2  Servers DetailedSession9part2  Servers Detailed
Session9part2 Servers Detailed
 
9/28/11 Slides - Introduction to DuraCloud, Slides
9/28/11 Slides - Introduction to DuraCloud, Slides9/28/11 Slides - Introduction to DuraCloud, Slides
9/28/11 Slides - Introduction to DuraCloud, Slides
 
HDFS - What's New and Future
HDFS - What's New and FutureHDFS - What's New and Future
HDFS - What's New and Future
 
Cloud Foundry Bootcamp
Cloud Foundry BootcampCloud Foundry Bootcamp
Cloud Foundry Bootcamp
 
High-Availability of YARN (MRv2)
High-Availability of YARN (MRv2)High-Availability of YARN (MRv2)
High-Availability of YARN (MRv2)
 
Charlie Talk - Everything At The Click Of A Button
Charlie Talk - Everything At The Click Of A ButtonCharlie Talk - Everything At The Click Of A Button
Charlie Talk - Everything At The Click Of A Button
 
[AzurePT] Desenvolvimento para o Windows Azure: Diferença para o developer
[AzurePT] Desenvolvimento para o Windows Azure: Diferença para o developer[AzurePT] Desenvolvimento para o Windows Azure: Diferença para o developer
[AzurePT] Desenvolvimento para o Windows Azure: Diferença para o developer
 
IAP09 CUDA@MIT 6.963 - Guest Lecture: CUDA Tricks and High-Performance Comput...
IAP09 CUDA@MIT 6.963 - Guest Lecture: CUDA Tricks and High-Performance Comput...IAP09 CUDA@MIT 6.963 - Guest Lecture: CUDA Tricks and High-Performance Comput...
IAP09 CUDA@MIT 6.963 - Guest Lecture: CUDA Tricks and High-Performance Comput...
 
The Native NDB Engine for Memcached
The Native NDB Engine for MemcachedThe Native NDB Engine for Memcached
The Native NDB Engine for Memcached
 
How LCDS works
How LCDS worksHow LCDS works
How LCDS works
 

Viewers also liked

Resolucion de conflictos bases datos 2
Resolucion de conflictos bases datos 2Resolucion de conflictos bases datos 2
Resolucion de conflictos bases datos 2
Velmuz Buzz
 
Perfil del Dip. Mario Sánchez Ruiz
Perfil del Dip. Mario Sánchez RuizPerfil del Dip. Mario Sánchez Ruiz
Perfil del Dip. Mario Sánchez Ruiz
UNAM
 
Abstractbook GfA 2016-02-10-2016lowres (1)
Abstractbook GfA 2016-02-10-2016lowres (1)Abstractbook GfA 2016-02-10-2016lowres (1)
Abstractbook GfA 2016-02-10-2016lowres (1)
Fred Heller
 
Raviv Melamed (Vayyar Imaging) Making Superman Vision a Reality
Raviv Melamed (Vayyar Imaging) Making Superman Vision a RealityRaviv Melamed (Vayyar Imaging) Making Superman Vision a Reality
Raviv Melamed (Vayyar Imaging) Making Superman Vision a Reality
AugmentedWorldExpo
 

Viewers also liked (9)

Resolucion de conflictos bases datos 2
Resolucion de conflictos bases datos 2Resolucion de conflictos bases datos 2
Resolucion de conflictos bases datos 2
 
Selenium Conf 2014 Opening Talk
Selenium Conf 2014 Opening TalkSelenium Conf 2014 Opening Talk
Selenium Conf 2014 Opening Talk
 
Perfil del Dip. Mario Sánchez Ruiz
Perfil del Dip. Mario Sánchez RuizPerfil del Dip. Mario Sánchez Ruiz
Perfil del Dip. Mario Sánchez Ruiz
 
Abstractbook GfA 2016-02-10-2016lowres (1)
Abstractbook GfA 2016-02-10-2016lowres (1)Abstractbook GfA 2016-02-10-2016lowres (1)
Abstractbook GfA 2016-02-10-2016lowres (1)
 
Raviv Melamed (Vayyar Imaging) Making Superman Vision a Reality
Raviv Melamed (Vayyar Imaging) Making Superman Vision a RealityRaviv Melamed (Vayyar Imaging) Making Superman Vision a Reality
Raviv Melamed (Vayyar Imaging) Making Superman Vision a Reality
 
CocoMake7 and SmartCoconuts
CocoMake7 and SmartCoconutsCocoMake7 and SmartCoconuts
CocoMake7 and SmartCoconuts
 
Zeta Architecture: The Next Generation Big Data Architecture
Zeta Architecture: The Next Generation Big Data ArchitectureZeta Architecture: The Next Generation Big Data Architecture
Zeta Architecture: The Next Generation Big Data Architecture
 
Projekte & Objekte 03-2009
Projekte & Objekte 03-2009Projekte & Objekte 03-2009
Projekte & Objekte 03-2009
 
Content, content, content
Content, content, contentContent, content, content
Content, content, content
 

Similar to Preventing multi master conflicts with tungsten

Database Performance With Proxy Architectures
Database  Performance With  Proxy  ArchitecturesDatabase  Performance With  Proxy  Architectures
Database Performance With Proxy Architectures
PerconaPerformance
 
NFV SDN Summit March 2014 D1 07 kireeti_kompella Native MPLS Fabric
NFV SDN Summit March 2014 D1 07 kireeti_kompella Native MPLS FabricNFV SDN Summit March 2014 D1 07 kireeti_kompella Native MPLS Fabric
NFV SDN Summit March 2014 D1 07 kireeti_kompella Native MPLS Fabric
ozkan01
 

Similar to Preventing multi master conflicts with tungsten (20)

Tungsten University: Geographically Distributed Multi-Master MySQL Clusters
Tungsten University: Geographically Distributed Multi-Master MySQL ClustersTungsten University: Geographically Distributed Multi-Master MySQL Clusters
Tungsten University: Geographically Distributed Multi-Master MySQL Clusters
 
Tungsten University: Setup and Operate Tungsten Replicators
Tungsten University: Setup and Operate Tungsten ReplicatorsTungsten University: Setup and Operate Tungsten Replicators
Tungsten University: Setup and Operate Tungsten Replicators
 
Database Performance With Proxy Architectures
Database  Performance With  Proxy  ArchitecturesDatabase  Performance With  Proxy  Architectures
Database Performance With Proxy Architectures
 
Oracle GoldenGate - Herve Schweitzer, dbi services - Hilton Basel 5/2011
Oracle GoldenGate - Herve Schweitzer, dbi services - Hilton Basel 5/2011Oracle GoldenGate - Herve Schweitzer, dbi services - Hilton Basel 5/2011
Oracle GoldenGate - Herve Schweitzer, dbi services - Hilton Basel 5/2011
 
Build Programming Language Runtime with LLVM
Build Programming Language Runtime with LLVMBuild Programming Language Runtime with LLVM
Build Programming Language Runtime with LLVM
 
OSCON Data 2011 -- NoSQL @ Netflix, Part 2
OSCON Data 2011 -- NoSQL @ Netflix, Part 2OSCON Data 2011 -- NoSQL @ Netflix, Part 2
OSCON Data 2011 -- NoSQL @ Netflix, Part 2
 
Flexible Replication
Flexible ReplicationFlexible Replication
Flexible Replication
 
Set Up & Operate Tungsten Replicator
Set Up & Operate Tungsten ReplicatorSet Up & Operate Tungsten Replicator
Set Up & Operate Tungsten Replicator
 
Setup & Operate Tungsten Replicator
Setup & Operate Tungsten ReplicatorSetup & Operate Tungsten Replicator
Setup & Operate Tungsten Replicator
 
Cp7 rpc
Cp7 rpcCp7 rpc
Cp7 rpc
 
Integration and Batch Processing on Cloud Foundry
Integration and Batch Processing on Cloud FoundryIntegration and Batch Processing on Cloud Foundry
Integration and Batch Processing on Cloud Foundry
 
NFV SDN Summit March 2014 D1 07 kireeti_kompella Native MPLS Fabric
NFV SDN Summit March 2014 D1 07 kireeti_kompella Native MPLS FabricNFV SDN Summit March 2014 D1 07 kireeti_kompella Native MPLS Fabric
NFV SDN Summit March 2014 D1 07 kireeti_kompella Native MPLS Fabric
 
Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!
Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!
Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!
 
Implementing the Future of PostgreSQL Clustering with Tungsten
Implementing the Future of PostgreSQL Clustering with TungstenImplementing the Future of PostgreSQL Clustering with Tungsten
Implementing the Future of PostgreSQL Clustering with Tungsten
 
Tungsten University: Configure and provision Tungsten clusters
Tungsten University: Configure and provision Tungsten clustersTungsten University: Configure and provision Tungsten clusters
Tungsten University: Configure and provision Tungsten clusters
 
OpenLDAP Replication Strategies
OpenLDAP Replication StrategiesOpenLDAP Replication Strategies
OpenLDAP Replication Strategies
 
Peyton jones-2011-parallel haskell-the_future
Peyton jones-2011-parallel haskell-the_futurePeyton jones-2011-parallel haskell-the_future
Peyton jones-2011-parallel haskell-the_future
 
Simon Peyton Jones: Managing parallelism
Simon Peyton Jones: Managing parallelismSimon Peyton Jones: Managing parallelism
Simon Peyton Jones: Managing parallelism
 
Delay Tolerant Streaming Services, Thomas Plagemann, UiO
Delay Tolerant Streaming Services, Thomas Plagemann, UiODelay Tolerant Streaming Services, Thomas Plagemann, UiO
Delay Tolerant Streaming Services, Thomas Plagemann, UiO
 
Ruby for C#-ers (ScanDevConf 2010)
Ruby for C#-ers (ScanDevConf 2010)Ruby for C#-ers (ScanDevConf 2010)
Ruby for C#-ers (ScanDevConf 2010)
 

More from Giuseppe Maxia

More from Giuseppe Maxia (20)

MySQL NDB 8.0 clusters in your laptop with dbdeployer
MySQL NDB 8.0 clusters in your laptop with dbdeployerMySQL NDB 8.0 clusters in your laptop with dbdeployer
MySQL NDB 8.0 clusters in your laptop with dbdeployer
 
Test like a_boss
Test like a_bossTest like a_boss
Test like a_boss
 
Dbdeployer, the universal installer
Dbdeployer, the universal installerDbdeployer, the universal installer
Dbdeployer, the universal installer
 
Test complex database systems in your laptop with dbdeployer
Test complex database systems in your laptop with dbdeployerTest complex database systems in your laptop with dbdeployer
Test complex database systems in your laptop with dbdeployer
 
Dbdeployer
DbdeployerDbdeployer
Dbdeployer
 
Dbdeployer
DbdeployerDbdeployer
Dbdeployer
 
A quick tour of Mysql 8 roles
A quick tour of Mysql 8 rolesA quick tour of Mysql 8 roles
A quick tour of Mysql 8 roles
 
MySQL document_store
MySQL document_storeMySQL document_store
MySQL document_store
 
Replication skeptic
Replication skepticReplication skeptic
Replication skeptic
 
Synchronise your data between MySQL and MongoDB
Synchronise your data between MySQL and MongoDBSynchronise your data between MySQL and MongoDB
Synchronise your data between MySQL and MongoDB
 
Juggle your data with Tungsten Replicator
Juggle your data with Tungsten ReplicatorJuggle your data with Tungsten Replicator
Juggle your data with Tungsten Replicator
 
MySQL in your laptop
MySQL in your laptopMySQL in your laptop
MySQL in your laptop
 
Script it
Script itScript it
Script it
 
Tungsten Replicator tutorial
Tungsten Replicator tutorialTungsten Replicator tutorial
Tungsten Replicator tutorial
 
MySQL high availability power and usability
MySQL high availability power and usabilityMySQL high availability power and usability
MySQL high availability power and usability
 
State of the art of MySQL replication and clustering
State of the art of MySQL replication and clusteringState of the art of MySQL replication and clustering
State of the art of MySQL replication and clustering
 
Testing mysql creatively in a sandbox
Testing mysql creatively in a sandboxTesting mysql creatively in a sandbox
Testing mysql creatively in a sandbox
 
Mysql 5.5 and 5.6 replication
Mysql 5.5 and 5.6 replicationMysql 5.5 and 5.6 replication
Mysql 5.5 and 5.6 replication
 
Lightning talks percona live mysql_2012
Lightning talks percona live mysql_2012Lightning talks percona live mysql_2012
Lightning talks percona live mysql_2012
 
Replication 101
Replication 101Replication 101
Replication 101
 

Recently uploaded

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Recently uploaded (20)

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 

Preventing multi master conflicts with tungsten

  • 1. Preventing con!icts in Multi-master replication with Tungsten Giuseppe Maxia, QA Director, Continuent ©Continuent 2012. 1 Sunday, February 03, 13
  • 2. Introducing Continuent • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Open source, Flexible, high- performance data movement ©Continuent 2012 2 Sunday, February 03, 13
  • 3. What is Tungsten Replicator? • Open source drop-in replacement for MySQL replication, providing: • Global transaction ID • Multiple masters • Multiple sources • Flexible topologies • Heterogeneous replication • Parallel replication • ... and more ©Continuent 2012. 3 Sunday, February 03, 13
  • 4. Tungsten Replicator Overview Master Replicator THL Download transactions via network (Transactions + Metadata) DBMS Logs Slave Replicator THL Apply using JDBC (Transactions + Metadata) ©Continuent 2012 4 Sunday, February 03, 13
  • 5. Tungsten Replication Service Pipeline Stage Stage Stage Extract Filter Apply Extract Filter Apply Extract Filter Apply Master Transaction In-Memory Slave DBMS History Log Queue DBMS ©Continuent 2012 5 Sunday, February 03, 13
  • 6. Multiple Services Per Replicator NYC Replicator Service nyc Replicator Service nyc Service fra Frankfurt Replicator Service fra London ©Continuent 2012 6 Sunday, February 03, 13
  • 7. Multi-Master Replication • Updates on 2+ sites (active-active mode) • Enables geographic distribution of data • No failover necessary if network fails or site becomes unavailable • Not all applications can handle multi-master • Applications must avoid con!icts • Careful testing required • Restoration of broken systems may not be easy ©Continuent 2012 7 Sunday, February 03, 13
  • 8. Simple Multi-Master Con!guration NYC Replicator Frankfurt Replicator fra (slave) fra (master) nyc (master) nyc (slave) Database-to-Database ©Continuent 2012 8 Sunday, February 03, 13
  • 9. Three-node Multi-Master Con!guration NYC Replicator Replicator Frankfurt fra (slave) fra (master) nyc (master) nyc (slave) lon (slave) lon (slave) Database-to-Database lon (master) fra (slave) nyc (slave) London Replicator ©Continuent 2012 9 Sunday, February 03, 13
  • 10. multi-node star Con!guration NYC Replicator Replicator Frankfurt nyc (master) fra (master) lon (slave) lon (slave) Replicator lon (master) Singapore sin (master) fra (slave) lon (slave) nyc (slave) London (hub) sin (slave) Replicator ©Continuent 2012 10 Sunday, February 03, 13
  • 11. CONFLICTS ©Continuent 2012 11 Sunday, February 03, 13
  • 12. What's a con"ict • Data modi!ed by several sources (masters) • Creates one or more : • data loss (unwanted delete) • data inconsistency (unwanted update) • duplicated data (unwanted insert) • replication break ©Continuent 2012. 12 Sunday, February 03, 13
  • 13. Data duplication 4 Matt 140 id name amount bravo alpha 1 Joe 100 2 Frank 110 3 Sue 100 BREAKS REPLICATION 4 Matt 130 charlie ©Continuent 2012. 13 Sunday, February 03, 13
  • 14. auto_increment o#sets are not a remedy • A popular recipe • auto_increment_increment + auto_increment_offset • They don't prevent con"icts • They hide duplicates ©Continuent 2012. 14 Sunday, February 03, 13
  • 15. Hidden data duplication 11 Matt 140 INSERT bravo alpha id name amount o#set 2 o#set 1 1 Joe 100 2 Frank 110 3 Sue 100 INSERT charlie 13 Matt 130 o#set 3 ©Continuent 2012. 15 Sunday, February 03, 13
  • 16. Data inconsistency 3 Sue 108 UPDATE id name amount bravo alpha 1 Joe 100 2 Frank 110 3 Sue 100 UPDATE 3 Sue 105 charlie ©Continuent 2012. 16 Sunday, February 03, 13
  • 17. Data loss 3 Sue 108 UPDATE id name amount bravo alpha 1 Joe 100 2 Frank 110 3 Sue 100 DELETE MAY BREAK REPLICATION record #3 charlie ©Continuent 2012. 17 Sunday, February 03, 13
  • 18. con"ict handling strategies • resolving • after the fact • Needs information that is missing in async replication • avoiding • requires synchronous replication with 2pc • preventing used by Tungsten • setting and enforcing a split sources policy • Transforming and resolving planned for • all records are converted to INSERTs future use • con!icts are resolved within a given time window ©Continuent 2012 18 Sunday, February 03, 13
  • 19. Tungsten Replicator Filters ©Continuent 2012. 19 Sunday, February 03, 13
  • 20. Replicator Pipeline Architecture Tungsten Replicator Process Pipeline Stage Stage Stage Assign Extract Shard Apply Extract Apply Extract Apply ID Transaction History Log MySQL filter THL filter filter Binlog Slave DBMS Sunday, February 03, 13
  • 21. Restrict replication to some schemas and tables ./tools/tungsten-installer --master-slave -a   ...   --svc-extractor-filters=replicate   "--property=replicator.filter.replicate.do=test,*.foo"   ...   --start-and-report # test="test.*" -> same drawback as binlog-do-db in MySQL # *.foo = table 'foo' in any database # employees.dept_codes,employees.salaries => safest way ©Continuent 2012. 21 Sunday, February 03, 13
  • 22. Multi-master: Con!ict prevention ©Continuent 2012. 22 Sunday, February 03, 13
  • 23. Tungsten con"ict prevention in a nutshell 1. de!ne the rules (which master can update which database) 2. tell Tungsten the rules 3. de!ne the policy (error, drop, warn, or accept) 4. Let Tungsten enforce your rules ©Continuent 2012. 23 Sunday, February 03, 13
  • 24. Tungsten Con"ict prevention facts • Sharded by database • De!ned dynamically • Applied on the slave services • methods: • error: make replication fail • drop: drop silently • warn: drop with warning ©Continuent 2012. 24 Sunday, February 03, 13
  • 25. Tungsten con"ict prevention applicability • unknown shards • The schema being updated is not planned • actions: accept, drop, warn, error • unwanted shards • the schema is updated from the wrong master • actions: accept, drop, warn, error • whitelisted shards • can be updated by any master ©Continuent 2012. 25 Sunday, February 03, 13
  • 26. Con"ict prevention directives --svc-extractor-filters=shardfilter replicator.filter.shardfilter.unknownShardPolicy=error replicator.filter.shardfilter.unwantedShardPolicy=error replicator.filter.shardfilter.enforceHomes=false replicator.filter.shardfilter.allowWhitelisted=false ©Continuent 2012. 26 Sunday, February 03, 13
  • 27. con"ict prevention in a star topology alpha updates H employees Host1 m A master: alpha d C database: employees H m d ✔ A Host3 master: charlie (hub) B C database: vehicles Host2 ✔ master: bravo database: buildings B C ©Continuent 2012. 27 Sunday, February 03, 13
  • 28. con"ict prevention in a star topology alpha updates H vehicles Host1 m A master: alpha d C database: employees H m d ✗ A Host3 master: charlie (hub) B C database: vehicles Host2 master: bravo database: buildings B C ©Continuent 2012. 28 Sunday, February 03, 13
  • 29. con"ict prevention in a all-masters topology H alpha updates Host1 m employees A master: alpha d B C database: employees H m d ✔ A Host3 master: charlie (hub) C ✔ B database: vehicles A Host2 master: bravo database: buildings B C ©Continuent 2012. 29 Sunday, February 03, 13
  • 30. con"ict prevention in a all-masters topology H charlie updates Host1 m vehicles A master: alpha d B C database: employees H m d ✔ A Host3 master: charlie (hub) C ✔ B database: vehicles A Host2 master: bravo database: buildings B C ©Continuent 2012. 30 Sunday, February 03, 13
  • 31. con"ict prevention in a all-masters topology H alpha updates Host1 m buildings A master: alpha d B C database: employees H ✗ m d A Host3 Host2 ✗ A B C master: charlie (hub) database: vehicles master: bravo database: buildings B C ©Continuent 2012. 31 Sunday, February 03, 13
  • 32. con"ict prevention in a all-masters topology H charlie updates Host1 m employees A master: alpha d B C database: employees H ✗ m d A Host3 Host2 ✗ A B C master: charlie (hub) database: vehicles master: bravo database: buildings B C ©Continuent 2012. 32 Sunday, February 03, 13
  • 33. setting con"ict prevention rules trepctl -host host1 -service charlie shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # charlie is slave service in host 1 ©Continuent 2012. 33 Sunday, February 03, 13
  • 34. setting con"ict prevention rules trepctl -host host2 -service charlie shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # charlie is slave service in host 2 ©Continuent 2012. 34 Sunday, February 03, 13
  • 35. setting con"ict prevention rules trepctl -host host3 -service alpha shard -insert < shards.map trepctl -host host3 -service bravo shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # alpha and bravo are slave services in host 3 ©Continuent 2012. 35 Sunday, February 03, 13
  • 36. Con"ict prevention demo • reminder • Server #1 can update "employees" • Server #2 can update "buildings" • Server #3 can update "vehicles" ©Continuent 2012. 36 Sunday, February 03, 13
  • 37. Sample correct operation (1) mysql #1> create table employees.names( ... ) # all servers receive the table # all servers keep working well ©Continuent 2012. 37 Sunday, February 03, 13
  • 38. Sample correct operation (2) mysql #2> create table buildings.homes( ... ) # all servers receive the table # all servers keep working well ©Continuent 2012. 38 Sunday, February 03, 13
  • 39. Sample incorrect operation (1) mysql #2> create table employees.nicknames( ... ) # Only server #2 receives the table # slave service in hub gets an error # slave service in #1 does not receive anything ©Continuent 2012. 39 Sunday, February 03, 13
  • 40. sample incorrect operation (2) #3 $ trepct services | simple_services alpha [slave] seqno: 7 - latency: 0.136 - ONLINE bravo [slave] seqno: -1 - latency: -1.000 - OFFLINE:ERROR charlie [master] seqno: 66 - latency: 0.440 - ONLINE ©Continuent 2012. 40 Sunday, February 03, 13
  • 41. sample incorrect operation (3) #3 $ trepct -service bravo status NAME VALUE ---- ----- appliedLastEventId : NONE appliedLastSeqno : -1 appliedLatency : -1.0 (...) offlineRequests : NONE pendingError : Stage task failed: q-to-dbms pendingErrorCode : NONE pendingErrorEventId : mysql-bin.000002:0000000000001241;0 pendingErrorSeqno : 7 pendingExceptionMessage: Rejected event from wrong shard: seqno=7 shard ID=employees shard master=alpha service=bravo (...) ©Continuent 2012. 41 Sunday, February 03, 13
  • 42. Fixing the issue mysql #1> drop table if exists employees.nicknames; mysql #1> create table if exists employees.nicknames ( ... ) ; #3 $ trepct -service bravo online -skip-seqno 7 # all servers receive the new table ©Continuent 2012. 42 Sunday, February 03, 13
  • 43. Sample whitelisted operation mysql #2> create table test.hope4best( ... ) mysql #1> insert into test.hope4best values ( ... ) # REMEMBER: 'test' was explicitly whitelisted # All servers get the new table and records # But there is no protection against conflicts ©Continuent 2012. 43 Sunday, February 03, 13
  • 44. Parting thoughts We are hiring ! http://continuent.com/about/careers ©Continuent 2012. 44 Sunday, February 03, 13
  • 45. blog: http://datacharmer.blogspot.com twitter: @datacharmer Continuent Website: http://www.continuent.com Tungsten Replicator 2.0: http://tungsten-replicator.org ©Continuent 2012 45 Sunday, February 03, 13