SlideShare ist ein Scribd-Unternehmen logo
1 von 64
Downloaden Sie, um offline zu lesen
Advanced sharding techniques with Spider




                          Kentoku SHIBA
             kentokushiba at gmail dot com
How to shard database
    without stopping the service
How to shard database

What is database sharding?
 When the data volume increases or the updating traffic
 increases, your updating database server cannot process
 effectively.
 We often use the technique for dividing data into two or
 more databases to solve the problem. This is database
 sharding.


   Here, I will explain how to shard a data,
        without stopping the service.
Initial Structure

                       tbl_a
Create table tbl_a (
  col_a int,           DB1
  col_b int,
  primary key(col_a)
) engine = InnoDB;




       There is 1 MySQL server without Spider.
Step 1 (for sharding)
                                         col_a%2=0

 Create table tbl_a3 (                               Create table tbl_a (
   col_a int,                   tbl_a     tbl_a        col_a int,
   col_b int,                                          col_b int,
   primary key(col_a)                      DB2         primary key(col_a)
                                                     ) engine = InnoDB;
 ) engine = Spider
 Connection ‘                   tbl_a2
   table “tbl_a”,                        col_a%2=1
   user “user”,                                      Create table tbl_a4 (
   password “pass”                                      col_a int,
 ‘
                                tbl_a3    tbl_a         col_b int,
 partition by list(                                     primary key(col_a)
   mod(col_a, 2)) (
   partition pt1 values in(0)
                                           DB3       ) engine = VP
                                                     Comment ‘
   comment ‘host “DB2”’,                                cit "2",
   partition pt2 values in(1)   tbl_a4                  cil "2",
   comment ‘host “DB3”’                                 ctm “1”,
 );                             DB1                     ist “1”,
                                                        zru “1”,
                                                        tnl “tbl_a2 tbl_a3”
                     Create table on DB2 and DB3.    ‘;

                      Then create tables on DB1.
Step 2
                                  col_a%2=0

                        tbl_a2      tbl_a
                                    DB2
                        tbl_a5
                                  col_a%2=1


                        tbl_a3      tbl_a
                                    DB3
                         tbl_a
                         DB1

                Rename table on DB1.
(rename table tbl_a2 to tbl_a5, tbl_a to tbl_a2, tbl_a4 to tbl_a)
Step 3
                                 col_a%2=0

                      tbl_a2       tbl_a
                                   DB2
                      tbl_a5
                                 col_a%2=1


                       tbl_a3      tbl_a
                                   DB3
                       tbl_a
                       DB1

   Copy data from tbl_a2 to tbl_a3 on DB1.
     (select vp_copy_tables(‘tbl_a’, ‘tbl_a2’, ‘tbl_a3’))
Step 4
                                  col_a%2=0

                        tbl_a2      tbl_a
                                    DB2
                        tbl_a5
                                  col_a%2=1


                         tbl_a      tbl_a
                                    DB3
                        tbl_a4
                         DB1

                Rename table on DB1.
         (rename table tbl_a to tbl_a4, tbl_a3 to tbl_a)
Finish
                            col_a%2=0

                              tbl_a
                              DB2
                            col_a%2=1


                   tbl_a      tbl_a
                              DB3


                   DB1

             Drop table on DB1.
         (drop table tbl_a2, tbl_a4, tbl_a5)
How to re-shard database
      without stopping the service
How to re-shard database

What is re-sharding?
 When the data volume increases or the updating traffic
 increases so much, even if you had your database sharded,
 your updating database server cannot process right again.
 So we solve that problem by increasing the number of
 servers and distributing the load.
 It is called re-sharding to increase the number of servers,
 and to distribute the load.


      Here, I will explain how to re-shard
        without stopping the service.
Initial Structure
                               col_a%2=0   col_a%2=1
Create table tbl_a (
  col_a int,
  col_b int,
  primary key(col_a)
                                tbl_a       tbl_a
) engine = Spider
Connection ‘
                                 DB2         DB3
  table “tbl_a”,
  user “user”,
  password “pass”
‘
partition by list(
  mod(col_a, 2)) (               tbl_a
  partition pt1 values in(0)
  comment ‘host “DB2”’,
  partition pt2 values in(1)
                                 DB1
  comment ‘host “DB3”’
);




 There are 1 MySQL server with Spider and 2 remote
            MySQL servers without Spider.
Step 1 (for re-sharding)
                                col_a%2=0   col_a%2=1   col_a%4=1

 Create table tbl_a3 (
   col_a int,                    tbl_a       tbl_a       tbl_a
   col_b int,
   primary key(col_a)             DB2                     DB4
 ) engine = Spider
 Connection ‘                                tbl_a2
   table “tbl_a”,                                       col_a%4=3
   user “user”,
   password “pass”
 ‘
                                  tbl_a      tbl_a3      tbl_a
 partition by list(
   mod(col_a, 4)) (
   partition pt1 values in(1)     DB1                     DB5
                                                              Create table tbl_a4 (
   comment ‘host “DB4”’,
   partition pt2 values in(3)                tbl_a4              col_a int,
   comment ‘host “DB5”’                                          col_b int,
                                                                 primary key(col_a)
 );                                           DB3             ) engine = VP
                                                              Comment ‘
                                                                 cit "2",
                     Create table on DB4 and DB5.                cil "2",
                                                                 ctm “1”,
                      Then create tables on DB3.                 ist “1”,
                                                                 zru “1”,
                                                                 tnl “tbl_a2 tbl_a3”
                                                              ‘;
Step 2
                       col_a%2=0   col_a%2=1   col_a%4=1

                         tbl_a      tbl_a2      tbl_a
                         DB2                     DB4
                                    tbl_a5
                                               col_a%4=3


                         tbl_a      tbl_a3      tbl_a

                         DB1                     DB5
                                     tbl_a
                                     DB3

                Rename table on DB3.
(rename table tbl_a2 to tbl_a5, tbl_a to tbl_a2, tbl_a4 to tbl_a)
Step 3
                     col_a%2=0   col_a%2=1   col_a%4=1

                       tbl_a      tbl_a2       tbl_a
                       DB2                      DB4
                                  tbl_a5
                                             col_a%4=3


                       tbl_a       tbl_a3      tbl_a

                        DB1                     DB5
                                   tbl_a
                                    DB3

   Copy data from tbl_a2 to tbl_a3 on DB3.
     (select vp_copy_tables(‘tbl_a’, ‘tbl_a2’, ‘tbl_a3’))
Step 4
                                 col_a%2=0   col_a%2=1   col_a%4=1

                                  tbl_a       tbl_a2        tbl_a
Alter table tbl_a
partition by list(                 DB2                       DB4
  mod(col_a, 4)) (
  partition pt1 values in(0,2)
                                              tbl_a5
                                                         col_a%4=3
  comment ‘host “DB2”’,
  partition pt2 values in(1)
  comment ‘host “DB4”’,
  partition pt2 values in(3)       tbl_a      tbl_a         tbl_a
  comment ‘host “DB5”’
);
                                   DB1                       DB5
                                              tbl_a4     Rename table
                                                          tbl_a to tbl_a4,
                                               DB3        tbl_a3 to tbl_a;



                             Rename table on DB3.
                            Then alter table on DB1.
Finish
         col_a%2=0    col_a%4=1

          tbl_a        tbl_a
           DB2          DB4
                      col_a%4=3


           tbl_a       tbl_a

           DB1          DB5




          Drop DB3.
How to add an index
   without stopping the service
How to add an index

If you add an index in MySQL, you cannot
update your data until the process is completed.
When it comes to a big table, it takes
a long time to complete, sometimes you cannot
use the service during the change.



   Here, I will explain how to add an index,
   without stopping the update of your data.
Initial Structure

                              tbl_a
Create table tbl_a (
  col_a int,                  DB1
  col_b int,
  primary key(col_a)
) engine = InnoDB;




                       There is 1 MySQL server.
Step 1 (for adding an index)

Create table tbl_a2 (
  col_a int,                 tbl_a
  col_b int,
  primary key(col_a)
) engine = InnoDB;                    Create table tbl_a4 (
                             tbl_a2      col_a int,
                                         col_b int,
                                         primary key(col_a)
Create table tbl_a3 (                 ) engine = VP
  col_a int,                 tbl_a3   Comment ‘
  col_b int,                             cit "2",
  primary key(col_a),                    cil "2",
  key idx1(col_b)                        ctm “1”,
) engine = InnoDB;           tbl_a4      ist “1”,
                                         zru “1”,
                                         tnl “tbl_a2 tbl_a3”
                             DB1      ‘;



                        Create tables on DB1.
Step 2

                        tbl_a2

                        tbl_a5


                        tbl_a3


                         tbl_a
                         DB1

                Rename table on DB1.
(rename table tbl_a2 to tbl_a5, tbl_a to tbl_a2, tbl_a4 to tbl_a)
Step 3

                      tbl_a2

                      tbl_a5


                      tbl_a3


                       tbl_a
                       DB1

   Copy data from tbl_a2 to tbl_a3 on DB1.
     (select vp_copy_tables(‘tbl_a’, ‘tbl_a2’, ‘tbl_a3’))
Step 4

                        tbl_a2

                        tbl_a5


                         tbl_a


                        tbl_a4
                         DB1

                Rename table on DB1.
         (rename table tbl_a to tbl_a4, tbl_a3 to tbl_a)
Finish




                   tbl_a



                   DB1

             Drop table on DB1.
         (drop table tbl_a2, tbl_a4, tbl_a5)
How to change the schema
      without stopping the service
How to change the schema

If you change schema in MySQL, you cannot
update your data until the process is completed.
When it comes to a big table, it takes
a long time to complete, sometimes you cannot
use the service during the change.



  Here, I will explain how to change schema,
   without stopping the update of your data.
Initial Structure

                              tbl_a
Create table tbl_a (
  col_a int,                  DB1
  col_b int,
  primary key(col_a)
) engine = InnoDB;




                       There is 1 MySQL server.
Step 1 (for adding a column)

Create table tbl_a2 (
  col_a int,                     tbl_a
  col_b int,
  primary key(col_a)
) engine = InnoDB;                        Create table tbl_a4 (
                                 tbl_a2      col_a int,
                                             col_b int,
                                             primary key(col_a)
Create table tbl_a3 (                     ) engine = VP
  col_a int,                     tbl_a3   Comment ‘
  col_b int,                                 cit "2",
  col_c int default null,                    cil "2",
  primary key(col_a)                         ctm “1”,
) engine = InnoDB;               tbl_a4      ist “1”,
                                             zru “1”,
                                             tnl “tbl_a2 tbl_a3”
                                 DB1      ‘;



                            Create tables on DB1.
Step 2

                        tbl_a2

                        tbl_a5


                        tbl_a3


                         tbl_a
                         DB1

                Rename table on DB1.
(rename table tbl_a2 to tbl_a5, tbl_a to tbl_a2, tbl_a4 to tbl_a)
Step 3

                      tbl_a2

                      tbl_a5


                      tbl_a3


                       tbl_a
                       DB1

   Copy data from tbl_a2 to tbl_a3 on DB1.
     (select vp_copy_tables(‘tbl_a’, ‘tbl_a2’, ‘tbl_a3’))
Step 4

                        tbl_a2

                        tbl_a5


                         tbl_a


                        tbl_a4
                         DB1

                Rename table on DB1.
         (rename table tbl_a to tbl_a4, tbl_a3 to tbl_a)
Finish




                   tbl_a



                   DB1

             Drop table on DB1.
         (drop table tbl_a2, tbl_a4, tbl_a5)
How to set up a cluster
  for fault tolerance
     without stopping the service
How to set up a cluster for fault tolerance

Spider can set up a cluster for fault tolerance
by each table.

    Here, I will explain how to set up cluster,
            without stopping service.

'Monitoring node' in this slide is a node that works to observe
the trouble of each node that composes clustering.
'Spider_copy_tables' in this slide is in development , so please
wait for a while to use it.
Initial Structure

                        tbl_a   Create table tbl_a (
                                  col_a int,
                        DB2       col_b int,
                                  primary key(col_a)
Create table tbl_a (            ) engine = InnoDB;
   col_a int,
   col_b int,
   primary key(col_a)
) engine = Spider
Connection ‘            tbl_a
   table “tbl_a”,
   user “user”,
   password “pass”,
                        DB1
   host “DB2”
‘;




 There are 1 MySQL server with Spider and 1 remote
             Mysql servers without Spider.
Step 1 (for clustering)

              tbl_a       tbl_a              tbl_a
              DB2         DB3                 DB4



              tbl_a               Create table tbl_a (
                                    col_a int,
              DB1                   col_b int,
                                    primary key(col_a)
                                  ) engine = InnoDB;




Add new data nodes(DB3 and DB4) and tables.
Step 2

            tbl_a     tbl_a     tbl_a
            DB2       DB3       DB4
                                   Create table tbl_a (
                                      col_a int,
                                      col_b int,
                                      primary key(col_a)
                                   ) engine = Spider
            tbl_a                  Connection ‘
                                      table “tbl_a”,
            DB1         tbl_a
                      DB7             user “user”,
                                      password “pass”,
                       DB6            host “DB2 DB3 DB4”
                         DB5       ‘;




          Add new monitoring nodes
         (DB5, DB6, DB7) and tables.
Step 3
     insert into mysql.spider_link_mon_servers
     (db_name, table_name, link_id, sid, server, scheme, host, port, socket, username, password)
     values
                             tbl_a                   tbl_a                  tbl_a
     ('db_name', 'tbl_a', 0, DB5_sid, null, 'mysql', 'DB5', 3306, null, 'user', 'pass‘),
     ('db_name', 'tbl_a', 0, DB6_sid, null, 'mysql', 'DB6', 3306, null, 'user', 'pass‘),
                              DB2                    DB3                     DB4
     ('db_name', 'tbl_a', 0, DB7_sid, null, 'mysql', 'DB7', 3306, null, 'user', 'pass‘);
Alter table tbl_a
Connection ‘
   table “tbl_a”,
   user “user”,
   password “pass”,
   host “DB2 DB3 DB4”,       tbl_a
   mbk “2”, mkd “2”,
   msi “DB5_sid”,             DB1                      tbl_a
                                                     DB7
   link_status “0 2 2”                                DB6
‘;                                                      DB5


         Register monitornig node information to
              MySQL servers with Spider.
                Then alter table on DB1.
Step 4

           tbl_a         tbl_a                      tbl_a
           DB2            DB3                       DB4



           tbl_a   Select spider_copy_tables(‘tbl_a’, ‘’, ‘’);


           DB1              tbl_a
                          DB7
                           DB6
                             DB5


    Copy data from DB2 to DB3 and DB4.
Finish

                         tbl_a     tbl_a        tbl_a
                         DB2       DB3          DB4
Alter table tbl_a
Connection ‘
   table “tbl_a”,
   user “user”,
   password “pass”,
   host “DB2 DB3 DB4”,   tbl_a
   mbk “2”, mkd “2”,
   msi “DB5_sid”,
   link_status “0 1 1”
                         DB1         tbl_a
                                   DB7
‘;
                                    DB6
                                      DB5


                          Alter table on DB1.
How to add new node
   after failover
    and preparing new server
   without stopping the service
Create a table of a new node to the clustered table

You need to create a new node, in order to
maintain redundancy, when there is a trouble
at the node that composes the cluster.

  Here, I will explain how to add a table of a
  new node, without stopping the service.
'Monitoring node' in this slide is a node that works to observe
the trouble of each node that composes clustering.
'Spider_copy_tables' in this slide is still in development , it will
   be available in future releases.
Initial Structure

               tbl_a         tbl_a        tbl_a
                DB2          DB3           DB4



               tbl_a
                DB1            tbl_a
                             DB7
                              DB6
                                DB5

         There are 4 MySQL servers with Spider
            (include 3 monitoring nodes) and
 3 MySQL servers without Spider (including 1 broken node).
Step 1

     tbl_a             tbl_a   tbl_a     tbl_a
     DB8               DB2     DB3       DB4



Create table tbl_a (   tbl_a
  col_a int,
  col_b int,
                       DB1       tbl_a
                               DB7
  primary key(col_a)
                                DB6
) engine = InnoDB;
                                  DB5


            Add new data node(DB8) and table.
Step 2

  tbl_a       tbl_a      tbl_a     tbl_a
  DB8          DB2       DB3       DB4


                                      Alter table tbl_a
              tbl_a                   Connection ‘
                                         table “tbl_a”,
                                         user “user”,
               DB1         tbl_a
                         DB7             password “pass”,
                          DB6            host “DB2 DB4 DB8”
                            DB5       ‘;




          Alter table on monitoring nodes
               (DB5, DB6 and DB7).
Step 3

      tbl_a              tbl_a     tbl_a        tbl_a
       DB8               DB2       DB3          DB4
Alter table tbl_a
Connection ‘
   table “tbl_a”,
   user “user”,
   password “pass”,
   host “DB2 DB4 DB8”,   tbl_a
   mbk “2”, mkd “2”,
   msi “DB5_sid”,        DB1         tbl_a
                                   DB7
   link_status “0 0 2”              DB6
‘;                                    DB5


                          Alter table on DB1.
Step 4

  tbl_a      tbl_a         tbl_a                      tbl_a
  DB8        DB2            DB3                       DB4



             tbl_a   Select spider_copy_tables(‘tbl_a’, ‘’, ‘’);


             DB1              tbl_a
                            DB7
                             DB6
                               DB5


          Copy data from DB2 to DB8.
Finish

      tbl_a              tbl_a     tbl_a        tbl_a
       DB8               DB2       DB3          DB4
Alter table tbl_a
Connection ‘
   table “tbl_a”,
   user “user”,
   password “pass”,
   host “DB2 DB4 DB8”,   tbl_a
   mbk “2”, mkd “2”,
   msi “DB5_sid”,        DB1         tbl_a
                                   DB7
   link_status “0 0 1”              DB6
‘;                                    DB5


                          Alter table on DB1.
How to avoid table partitioning
 UNIQUE column limitation
         without stopping the service
How to avoid table partitioning UNIQUE column limitation

Right now, there is a restriction of MySQL that
you cannot partition in other columns when
there is a PK or UNIQUE.



Here, I will show you how to partition a table by
 any columns even if there is a PK or
 UNIQUE.
Initial Structure

                              tbl_a
Create table tbl_a (
  col_a int,                  DB1
  col_b int,
  primary key(col_a)
) engine = InnoDB;




                       There is 1 MySQL server.
Step 1 (for avoiding partitioning limitation)
Create table tbl_a3 (
  col_a int,                                  Create table tbl_a2 (
  primary key(col_a)                 tbl_a      col_a int,
) engine = InnoDB                               col_b int,
partition by                                    primary key(col_a)
  linear hash(col_a)                 tbl_a2   ) engine = InnoDB;
  partitions 4;
                                              Create table tbl_a5 (
Create table tbl_a4 (                tbl_a3      col_a int,
  col_a int,                                     col_b int,
  col_b int,                                     primary key(col_a)
  key idx1(col_a),                   tbl_a4   ) engine = VP
  key idx2(col_b)                             Comment ‘
) engine = InnoDB                                ctm “1”, ist “1”,
partition by list(                               zru “1”, pcm “1”
  mod(col_b, 2)) (                   tbl_a5   ‘
  partition pt1 values in(0),                 Connection ‘

);
  partition pt2 values in(1)         DB1      ‘;
                                                 tnl “tbl_a2 tbl_a3 tbl_a4”



                                Create tables on DB1.
Step 2

                        tbl_a2

                        tbl_a6

                        tbl_a3

                        tbl_a4

                         tbl_a
                         DB1

                Rename table on DB1.
(rename table tbl_a2 to tbl_a6, tbl_a to tbl_a2, tbl_a5 to tbl_a)
Step 3

                       tbl_a2

                       tbl_a6

                       tbl_a3

                       tbl_a4

                        tbl_a
                        DB1

 Copy data from tbl_a2 to tbl_a3 and tbl_a4.
  (select vp_copy_tables(‘tbl_a’, ‘tbl_a2’, ‘tbl_a3 tbl_a4’))
Step 4

           tbl_a2

           tbl_a6

           tbl_a3
                     Alter table tbl_a
           tbl_a4    Comment ‘
                        ctm “1”, ist “1”,
                        pcm “1”
                     ‘,
            tbl_a    Connection ‘
                        tnl “tbl_a3 tbl_a4”
            DB1      ‘;



         Alter table tbl_a.
Finish




              tbl_a3

              tbl_a4

               tbl_a
               DB1

              Drop table.
         (drop table tbl_a2, tbl_a6)
Case study
About MicroAd




        MicroAd is an advatising company.

       This company can advertise efficiently
      using "behavioral targeting" technology.



                         【MicroAd, Inc.]
           http://www.microad.jp/english/
The previous architecture

   ……                                                   ……
          AP             AP          AP         AP

                               LVS


         Slave         Slave
          DB            DB      Register new statistical rules
               replication      from batch server
                      Master
                                                Batch
                       DB

Batch processing updates new statistical rules every day.
    (For every advertisers, every advertising medias
                   and every users)
The problem with business expansion

Increase data and request.
  At that time the limit of updates were 20 million
  records a day.
  They needed to update 100 million records a day.

  They also wanted to improve the performance of
  the reference slave by decreasing the amount of
  the update by one slave.

  They did not want to change or modify their
  application to support the increase.

           Then, Spider was used.
The architecture with Spider
 ……         AP                AP                  AP               AP         ……
        with Spider       with Spider         with Spider      with Spider
Spider sharding
           LVS                          LVS                        LVS

   SlaveDB      SlaveDB      SlaveDB      SlaveDB           SlaveDB     SlaveDB
         replication              replication                   replication
         MasterDB                 MasterDB                      MasterDB

Spider sharding                                         Register new
                                  statistical rules from batch server
            SpiderDB
        (MySQL with Spider)                                   Batch


             They created the shards with
              the unit of the replication.
Resolved the problem

As a result,
  They achieved update 100 million records a day
  and improved the performance of the reference.

  They didn't need to change or modify their
  applications so much.

  They are planning in the near future of
  resharding, when they expand the business.
Any Questions?


Thank you for taking
    your time!!



  Kentoku SHIBA (kentokushiba at gmail dot com)
  http://wild-growth.blogspot.com/
  http://spiderformysql.com

Weitere ähnliche Inhalte

Was ist angesagt?

MariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB plc
 
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docx
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docxKeepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docx
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docxNeoClova
 
M|18 How MariaDB Server Scales with Spider
M|18 How MariaDB Server Scales with SpiderM|18 How MariaDB Server Scales with Spider
M|18 How MariaDB Server Scales with SpiderMariaDB plc
 
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptxMySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptxNeoClova
 
Maxscale 소개 1.1.1
Maxscale 소개 1.1.1Maxscale 소개 1.1.1
Maxscale 소개 1.1.1NeoClova
 
Best practices for MySQL/MariaDB Server/Percona Server High Availability
Best practices for MySQL/MariaDB Server/Percona Server High AvailabilityBest practices for MySQL/MariaDB Server/Percona Server High Availability
Best practices for MySQL/MariaDB Server/Percona Server High AvailabilityColin Charles
 
MariaDB 마이그레이션 - 네오클로바
MariaDB 마이그레이션 - 네오클로바MariaDB 마이그레이션 - 네오클로바
MariaDB 마이그레이션 - 네오클로바NeoClova
 
What is new in PostgreSQL 14?
What is new in PostgreSQL 14?What is new in PostgreSQL 14?
What is new in PostgreSQL 14?Mydbops
 
Optimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceOptimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceMariaDB plc
 
Galera cluster for high availability
Galera cluster for high availability Galera cluster for high availability
Galera cluster for high availability Mydbops
 
[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization
[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization
[Pgday.Seoul 2021] 2. Porting Oracle UDF and OptimizationPgDay.Seoul
 
Parallel Replication in MySQL and MariaDB
Parallel Replication in MySQL and MariaDBParallel Replication in MySQL and MariaDB
Parallel Replication in MySQL and MariaDBMydbops
 
PostgreSQL Performance Tuning
PostgreSQL Performance TuningPostgreSQL Performance Tuning
PostgreSQL Performance Tuningelliando dias
 
Replication in PostgreSQL tutorial given in Postgres Conference 2019
Replication in PostgreSQL tutorial given in Postgres Conference 2019Replication in PostgreSQL tutorial given in Postgres Conference 2019
Replication in PostgreSQL tutorial given in Postgres Conference 2019Abbas Butt
 
MySQL GTID 시작하기
MySQL GTID 시작하기MySQL GTID 시작하기
MySQL GTID 시작하기I Goo Lee
 
MySQL_SQL_Tunning_v0.1.3.docx
MySQL_SQL_Tunning_v0.1.3.docxMySQL_SQL_Tunning_v0.1.3.docx
MySQL_SQL_Tunning_v0.1.3.docxNeoClova
 
Tuning Autovacuum in Postgresql
Tuning Autovacuum in PostgresqlTuning Autovacuum in Postgresql
Tuning Autovacuum in PostgresqlMydbops
 
UKOUG - 25 years of hints and tips
UKOUG - 25 years of hints and tipsUKOUG - 25 years of hints and tips
UKOUG - 25 years of hints and tipsConnor McDonald
 
Don’t optimize my queries, optimize my data!
Don’t optimize my queries, optimize my data!Don’t optimize my queries, optimize my data!
Don’t optimize my queries, optimize my data!Julian Hyde
 

Was ist angesagt? (20)

MariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & Optimization
 
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docx
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docxKeepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docx
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docx
 
M|18 How MariaDB Server Scales with Spider
M|18 How MariaDB Server Scales with SpiderM|18 How MariaDB Server Scales with Spider
M|18 How MariaDB Server Scales with Spider
 
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptxMySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
 
Maxscale 소개 1.1.1
Maxscale 소개 1.1.1Maxscale 소개 1.1.1
Maxscale 소개 1.1.1
 
Best practices for MySQL/MariaDB Server/Percona Server High Availability
Best practices for MySQL/MariaDB Server/Percona Server High AvailabilityBest practices for MySQL/MariaDB Server/Percona Server High Availability
Best practices for MySQL/MariaDB Server/Percona Server High Availability
 
Galera Cluster DDL and Schema Upgrades 220217
Galera Cluster DDL and Schema Upgrades 220217Galera Cluster DDL and Schema Upgrades 220217
Galera Cluster DDL and Schema Upgrades 220217
 
MariaDB 마이그레이션 - 네오클로바
MariaDB 마이그레이션 - 네오클로바MariaDB 마이그레이션 - 네오클로바
MariaDB 마이그레이션 - 네오클로바
 
What is new in PostgreSQL 14?
What is new in PostgreSQL 14?What is new in PostgreSQL 14?
What is new in PostgreSQL 14?
 
Optimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceOptimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performance
 
Galera cluster for high availability
Galera cluster for high availability Galera cluster for high availability
Galera cluster for high availability
 
[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization
[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization
[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization
 
Parallel Replication in MySQL and MariaDB
Parallel Replication in MySQL and MariaDBParallel Replication in MySQL and MariaDB
Parallel Replication in MySQL and MariaDB
 
PostgreSQL Performance Tuning
PostgreSQL Performance TuningPostgreSQL Performance Tuning
PostgreSQL Performance Tuning
 
Replication in PostgreSQL tutorial given in Postgres Conference 2019
Replication in PostgreSQL tutorial given in Postgres Conference 2019Replication in PostgreSQL tutorial given in Postgres Conference 2019
Replication in PostgreSQL tutorial given in Postgres Conference 2019
 
MySQL GTID 시작하기
MySQL GTID 시작하기MySQL GTID 시작하기
MySQL GTID 시작하기
 
MySQL_SQL_Tunning_v0.1.3.docx
MySQL_SQL_Tunning_v0.1.3.docxMySQL_SQL_Tunning_v0.1.3.docx
MySQL_SQL_Tunning_v0.1.3.docx
 
Tuning Autovacuum in Postgresql
Tuning Autovacuum in PostgresqlTuning Autovacuum in Postgresql
Tuning Autovacuum in Postgresql
 
UKOUG - 25 years of hints and tips
UKOUG - 25 years of hints and tipsUKOUG - 25 years of hints and tips
UKOUG - 25 years of hints and tips
 
Don’t optimize my queries, optimize my data!
Don’t optimize my queries, optimize my data!Don’t optimize my queries, optimize my data!
Don’t optimize my queries, optimize my data!
 

Mehr von Kentoku

An issue of all slaves stop replication
An issue of all slaves stop replicationAn issue of all slaves stop replication
An issue of all slaves stop replicationKentoku
 
How to migrate_to_sharding_with_spider
How to migrate_to_sharding_with_spiderHow to migrate_to_sharding_with_spider
How to migrate_to_sharding_with_spiderKentoku
 
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介Kentoku
 
Spider storage engine (dec212016)
Spider storage engine (dec212016)Spider storage engine (dec212016)
Spider storage engine (dec212016)Kentoku
 
Spiderストレージエンジンのご紹介
Spiderストレージエンジンのご紹介Spiderストレージエンジンのご紹介
Spiderストレージエンジンのご紹介Kentoku
 
Using spider for sharding in production
Using spider for sharding in productionUsing spider for sharding in production
Using spider for sharding in productionKentoku
 
MariaDB ColumnStore 20160721
MariaDB ColumnStore 20160721MariaDB ColumnStore 20160721
MariaDB ColumnStore 20160721Kentoku
 
Sharding with spider solutions 20160721
Sharding with spider solutions 20160721Sharding with spider solutions 20160721
Sharding with spider solutions 20160721Kentoku
 
Mroonga 20141129
Mroonga 20141129Mroonga 20141129
Mroonga 20141129Kentoku
 
MariaDB Spider Mroonga 20140218
MariaDB Spider Mroonga 20140218MariaDB Spider Mroonga 20140218
MariaDB Spider Mroonga 20140218Kentoku
 
Mroonga 20131129
Mroonga 20131129Mroonga 20131129
Mroonga 20131129Kentoku
 
Newest topic of spider 20131016 in Buenos Aires Argentina
Newest topic of spider 20131016 in Buenos Aires ArgentinaNewest topic of spider 20131016 in Buenos Aires Argentina
Newest topic of spider 20131016 in Buenos Aires ArgentinaKentoku
 
Spiderの最新動向 20131009
Spiderの最新動向 20131009Spiderの最新動向 20131009
Spiderの最新動向 20131009Kentoku
 
Spiderの最新動向 20130419
Spiderの最新動向 20130419Spiderの最新動向 20130419
Spiderの最新動向 20130419Kentoku
 
Mroonga 20121129
Mroonga 20121129Mroonga 20121129
Mroonga 20121129Kentoku
 
Mroonga unsupported feature_20111129
Mroonga unsupported feature_20111129Mroonga unsupported feature_20111129
Mroonga unsupported feature_20111129Kentoku
 
Introducing mroonga 20111129
Introducing mroonga 20111129Introducing mroonga 20111129
Introducing mroonga 20111129Kentoku
 
hs_spider_hs_something_20110906
hs_spider_hs_something_20110906hs_spider_hs_something_20110906
hs_spider_hs_something_20110906Kentoku
 
Charms of MySQL 20101206(DTT#7)
Charms of MySQL 20101206(DTT#7)Charms of MySQL 20101206(DTT#7)
Charms of MySQL 20101206(DTT#7)Kentoku
 
Introducing Spider 20101206(DTT#7)
Introducing Spider 20101206(DTT#7)Introducing Spider 20101206(DTT#7)
Introducing Spider 20101206(DTT#7)Kentoku
 

Mehr von Kentoku (20)

An issue of all slaves stop replication
An issue of all slaves stop replicationAn issue of all slaves stop replication
An issue of all slaves stop replication
 
How to migrate_to_sharding_with_spider
How to migrate_to_sharding_with_spiderHow to migrate_to_sharding_with_spider
How to migrate_to_sharding_with_spider
 
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介
 
Spider storage engine (dec212016)
Spider storage engine (dec212016)Spider storage engine (dec212016)
Spider storage engine (dec212016)
 
Spiderストレージエンジンのご紹介
Spiderストレージエンジンのご紹介Spiderストレージエンジンのご紹介
Spiderストレージエンジンのご紹介
 
Using spider for sharding in production
Using spider for sharding in productionUsing spider for sharding in production
Using spider for sharding in production
 
MariaDB ColumnStore 20160721
MariaDB ColumnStore 20160721MariaDB ColumnStore 20160721
MariaDB ColumnStore 20160721
 
Sharding with spider solutions 20160721
Sharding with spider solutions 20160721Sharding with spider solutions 20160721
Sharding with spider solutions 20160721
 
Mroonga 20141129
Mroonga 20141129Mroonga 20141129
Mroonga 20141129
 
MariaDB Spider Mroonga 20140218
MariaDB Spider Mroonga 20140218MariaDB Spider Mroonga 20140218
MariaDB Spider Mroonga 20140218
 
Mroonga 20131129
Mroonga 20131129Mroonga 20131129
Mroonga 20131129
 
Newest topic of spider 20131016 in Buenos Aires Argentina
Newest topic of spider 20131016 in Buenos Aires ArgentinaNewest topic of spider 20131016 in Buenos Aires Argentina
Newest topic of spider 20131016 in Buenos Aires Argentina
 
Spiderの最新動向 20131009
Spiderの最新動向 20131009Spiderの最新動向 20131009
Spiderの最新動向 20131009
 
Spiderの最新動向 20130419
Spiderの最新動向 20130419Spiderの最新動向 20130419
Spiderの最新動向 20130419
 
Mroonga 20121129
Mroonga 20121129Mroonga 20121129
Mroonga 20121129
 
Mroonga unsupported feature_20111129
Mroonga unsupported feature_20111129Mroonga unsupported feature_20111129
Mroonga unsupported feature_20111129
 
Introducing mroonga 20111129
Introducing mroonga 20111129Introducing mroonga 20111129
Introducing mroonga 20111129
 
hs_spider_hs_something_20110906
hs_spider_hs_something_20110906hs_spider_hs_something_20110906
hs_spider_hs_something_20110906
 
Charms of MySQL 20101206(DTT#7)
Charms of MySQL 20101206(DTT#7)Charms of MySQL 20101206(DTT#7)
Charms of MySQL 20101206(DTT#7)
 
Introducing Spider 20101206(DTT#7)
Introducing Spider 20101206(DTT#7)Introducing Spider 20101206(DTT#7)
Introducing Spider 20101206(DTT#7)
 

Kürzlich hochgeladen

UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 

Kürzlich hochgeladen (20)

UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 

Advanced Sharding Techniques with Spider (MUC2010)

  • 1. Advanced sharding techniques with Spider Kentoku SHIBA kentokushiba at gmail dot com
  • 2. How to shard database without stopping the service
  • 3. How to shard database What is database sharding? When the data volume increases or the updating traffic increases, your updating database server cannot process effectively. We often use the technique for dividing data into two or more databases to solve the problem. This is database sharding. Here, I will explain how to shard a data, without stopping the service.
  • 4. Initial Structure tbl_a Create table tbl_a ( col_a int, DB1 col_b int, primary key(col_a) ) engine = InnoDB; There is 1 MySQL server without Spider.
  • 5. Step 1 (for sharding) col_a%2=0 Create table tbl_a3 ( Create table tbl_a ( col_a int, tbl_a tbl_a col_a int, col_b int, col_b int, primary key(col_a) DB2 primary key(col_a) ) engine = InnoDB; ) engine = Spider Connection ‘ tbl_a2 table “tbl_a”, col_a%2=1 user “user”, Create table tbl_a4 ( password “pass” col_a int, ‘ tbl_a3 tbl_a col_b int, partition by list( primary key(col_a) mod(col_a, 2)) ( partition pt1 values in(0) DB3 ) engine = VP Comment ‘ comment ‘host “DB2”’, cit "2", partition pt2 values in(1) tbl_a4 cil "2", comment ‘host “DB3”’ ctm “1”, ); DB1 ist “1”, zru “1”, tnl “tbl_a2 tbl_a3” Create table on DB2 and DB3. ‘; Then create tables on DB1.
  • 6. Step 2 col_a%2=0 tbl_a2 tbl_a DB2 tbl_a5 col_a%2=1 tbl_a3 tbl_a DB3 tbl_a DB1 Rename table on DB1. (rename table tbl_a2 to tbl_a5, tbl_a to tbl_a2, tbl_a4 to tbl_a)
  • 7. Step 3 col_a%2=0 tbl_a2 tbl_a DB2 tbl_a5 col_a%2=1 tbl_a3 tbl_a DB3 tbl_a DB1 Copy data from tbl_a2 to tbl_a3 on DB1. (select vp_copy_tables(‘tbl_a’, ‘tbl_a2’, ‘tbl_a3’))
  • 8. Step 4 col_a%2=0 tbl_a2 tbl_a DB2 tbl_a5 col_a%2=1 tbl_a tbl_a DB3 tbl_a4 DB1 Rename table on DB1. (rename table tbl_a to tbl_a4, tbl_a3 to tbl_a)
  • 9. Finish col_a%2=0 tbl_a DB2 col_a%2=1 tbl_a tbl_a DB3 DB1 Drop table on DB1. (drop table tbl_a2, tbl_a4, tbl_a5)
  • 10. How to re-shard database without stopping the service
  • 11. How to re-shard database What is re-sharding? When the data volume increases or the updating traffic increases so much, even if you had your database sharded, your updating database server cannot process right again. So we solve that problem by increasing the number of servers and distributing the load. It is called re-sharding to increase the number of servers, and to distribute the load. Here, I will explain how to re-shard without stopping the service.
  • 12. Initial Structure col_a%2=0 col_a%2=1 Create table tbl_a ( col_a int, col_b int, primary key(col_a) tbl_a tbl_a ) engine = Spider Connection ‘ DB2 DB3 table “tbl_a”, user “user”, password “pass” ‘ partition by list( mod(col_a, 2)) ( tbl_a partition pt1 values in(0) comment ‘host “DB2”’, partition pt2 values in(1) DB1 comment ‘host “DB3”’ ); There are 1 MySQL server with Spider and 2 remote MySQL servers without Spider.
  • 13. Step 1 (for re-sharding) col_a%2=0 col_a%2=1 col_a%4=1 Create table tbl_a3 ( col_a int, tbl_a tbl_a tbl_a col_b int, primary key(col_a) DB2 DB4 ) engine = Spider Connection ‘ tbl_a2 table “tbl_a”, col_a%4=3 user “user”, password “pass” ‘ tbl_a tbl_a3 tbl_a partition by list( mod(col_a, 4)) ( partition pt1 values in(1) DB1 DB5 Create table tbl_a4 ( comment ‘host “DB4”’, partition pt2 values in(3) tbl_a4 col_a int, comment ‘host “DB5”’ col_b int, primary key(col_a) ); DB3 ) engine = VP Comment ‘ cit "2", Create table on DB4 and DB5. cil "2", ctm “1”, Then create tables on DB3. ist “1”, zru “1”, tnl “tbl_a2 tbl_a3” ‘;
  • 14. Step 2 col_a%2=0 col_a%2=1 col_a%4=1 tbl_a tbl_a2 tbl_a DB2 DB4 tbl_a5 col_a%4=3 tbl_a tbl_a3 tbl_a DB1 DB5 tbl_a DB3 Rename table on DB3. (rename table tbl_a2 to tbl_a5, tbl_a to tbl_a2, tbl_a4 to tbl_a)
  • 15. Step 3 col_a%2=0 col_a%2=1 col_a%4=1 tbl_a tbl_a2 tbl_a DB2 DB4 tbl_a5 col_a%4=3 tbl_a tbl_a3 tbl_a DB1 DB5 tbl_a DB3 Copy data from tbl_a2 to tbl_a3 on DB3. (select vp_copy_tables(‘tbl_a’, ‘tbl_a2’, ‘tbl_a3’))
  • 16. Step 4 col_a%2=0 col_a%2=1 col_a%4=1 tbl_a tbl_a2 tbl_a Alter table tbl_a partition by list( DB2 DB4 mod(col_a, 4)) ( partition pt1 values in(0,2) tbl_a5 col_a%4=3 comment ‘host “DB2”’, partition pt2 values in(1) comment ‘host “DB4”’, partition pt2 values in(3) tbl_a tbl_a tbl_a comment ‘host “DB5”’ ); DB1 DB5 tbl_a4 Rename table tbl_a to tbl_a4, DB3 tbl_a3 to tbl_a; Rename table on DB3. Then alter table on DB1.
  • 17. Finish col_a%2=0 col_a%4=1 tbl_a tbl_a DB2 DB4 col_a%4=3 tbl_a tbl_a DB1 DB5 Drop DB3.
  • 18. How to add an index without stopping the service
  • 19. How to add an index If you add an index in MySQL, you cannot update your data until the process is completed. When it comes to a big table, it takes a long time to complete, sometimes you cannot use the service during the change. Here, I will explain how to add an index, without stopping the update of your data.
  • 20. Initial Structure tbl_a Create table tbl_a ( col_a int, DB1 col_b int, primary key(col_a) ) engine = InnoDB; There is 1 MySQL server.
  • 21. Step 1 (for adding an index) Create table tbl_a2 ( col_a int, tbl_a col_b int, primary key(col_a) ) engine = InnoDB; Create table tbl_a4 ( tbl_a2 col_a int, col_b int, primary key(col_a) Create table tbl_a3 ( ) engine = VP col_a int, tbl_a3 Comment ‘ col_b int, cit "2", primary key(col_a), cil "2", key idx1(col_b) ctm “1”, ) engine = InnoDB; tbl_a4 ist “1”, zru “1”, tnl “tbl_a2 tbl_a3” DB1 ‘; Create tables on DB1.
  • 22. Step 2 tbl_a2 tbl_a5 tbl_a3 tbl_a DB1 Rename table on DB1. (rename table tbl_a2 to tbl_a5, tbl_a to tbl_a2, tbl_a4 to tbl_a)
  • 23. Step 3 tbl_a2 tbl_a5 tbl_a3 tbl_a DB1 Copy data from tbl_a2 to tbl_a3 on DB1. (select vp_copy_tables(‘tbl_a’, ‘tbl_a2’, ‘tbl_a3’))
  • 24. Step 4 tbl_a2 tbl_a5 tbl_a tbl_a4 DB1 Rename table on DB1. (rename table tbl_a to tbl_a4, tbl_a3 to tbl_a)
  • 25. Finish tbl_a DB1 Drop table on DB1. (drop table tbl_a2, tbl_a4, tbl_a5)
  • 26. How to change the schema without stopping the service
  • 27. How to change the schema If you change schema in MySQL, you cannot update your data until the process is completed. When it comes to a big table, it takes a long time to complete, sometimes you cannot use the service during the change. Here, I will explain how to change schema, without stopping the update of your data.
  • 28. Initial Structure tbl_a Create table tbl_a ( col_a int, DB1 col_b int, primary key(col_a) ) engine = InnoDB; There is 1 MySQL server.
  • 29. Step 1 (for adding a column) Create table tbl_a2 ( col_a int, tbl_a col_b int, primary key(col_a) ) engine = InnoDB; Create table tbl_a4 ( tbl_a2 col_a int, col_b int, primary key(col_a) Create table tbl_a3 ( ) engine = VP col_a int, tbl_a3 Comment ‘ col_b int, cit "2", col_c int default null, cil "2", primary key(col_a) ctm “1”, ) engine = InnoDB; tbl_a4 ist “1”, zru “1”, tnl “tbl_a2 tbl_a3” DB1 ‘; Create tables on DB1.
  • 30. Step 2 tbl_a2 tbl_a5 tbl_a3 tbl_a DB1 Rename table on DB1. (rename table tbl_a2 to tbl_a5, tbl_a to tbl_a2, tbl_a4 to tbl_a)
  • 31. Step 3 tbl_a2 tbl_a5 tbl_a3 tbl_a DB1 Copy data from tbl_a2 to tbl_a3 on DB1. (select vp_copy_tables(‘tbl_a’, ‘tbl_a2’, ‘tbl_a3’))
  • 32. Step 4 tbl_a2 tbl_a5 tbl_a tbl_a4 DB1 Rename table on DB1. (rename table tbl_a to tbl_a4, tbl_a3 to tbl_a)
  • 33. Finish tbl_a DB1 Drop table on DB1. (drop table tbl_a2, tbl_a4, tbl_a5)
  • 34. How to set up a cluster for fault tolerance without stopping the service
  • 35. How to set up a cluster for fault tolerance Spider can set up a cluster for fault tolerance by each table. Here, I will explain how to set up cluster, without stopping service. 'Monitoring node' in this slide is a node that works to observe the trouble of each node that composes clustering. 'Spider_copy_tables' in this slide is in development , so please wait for a while to use it.
  • 36. Initial Structure tbl_a Create table tbl_a ( col_a int, DB2 col_b int, primary key(col_a) Create table tbl_a ( ) engine = InnoDB; col_a int, col_b int, primary key(col_a) ) engine = Spider Connection ‘ tbl_a table “tbl_a”, user “user”, password “pass”, DB1 host “DB2” ‘; There are 1 MySQL server with Spider and 1 remote Mysql servers without Spider.
  • 37. Step 1 (for clustering) tbl_a tbl_a tbl_a DB2 DB3 DB4 tbl_a Create table tbl_a ( col_a int, DB1 col_b int, primary key(col_a) ) engine = InnoDB; Add new data nodes(DB3 and DB4) and tables.
  • 38. Step 2 tbl_a tbl_a tbl_a DB2 DB3 DB4 Create table tbl_a ( col_a int, col_b int, primary key(col_a) ) engine = Spider tbl_a Connection ‘ table “tbl_a”, DB1 tbl_a DB7 user “user”, password “pass”, DB6 host “DB2 DB3 DB4” DB5 ‘; Add new monitoring nodes (DB5, DB6, DB7) and tables.
  • 39. Step 3 insert into mysql.spider_link_mon_servers (db_name, table_name, link_id, sid, server, scheme, host, port, socket, username, password) values tbl_a tbl_a tbl_a ('db_name', 'tbl_a', 0, DB5_sid, null, 'mysql', 'DB5', 3306, null, 'user', 'pass‘), ('db_name', 'tbl_a', 0, DB6_sid, null, 'mysql', 'DB6', 3306, null, 'user', 'pass‘), DB2 DB3 DB4 ('db_name', 'tbl_a', 0, DB7_sid, null, 'mysql', 'DB7', 3306, null, 'user', 'pass‘); Alter table tbl_a Connection ‘ table “tbl_a”, user “user”, password “pass”, host “DB2 DB3 DB4”, tbl_a mbk “2”, mkd “2”, msi “DB5_sid”, DB1 tbl_a DB7 link_status “0 2 2” DB6 ‘; DB5 Register monitornig node information to MySQL servers with Spider. Then alter table on DB1.
  • 40. Step 4 tbl_a tbl_a tbl_a DB2 DB3 DB4 tbl_a Select spider_copy_tables(‘tbl_a’, ‘’, ‘’); DB1 tbl_a DB7 DB6 DB5 Copy data from DB2 to DB3 and DB4.
  • 41. Finish tbl_a tbl_a tbl_a DB2 DB3 DB4 Alter table tbl_a Connection ‘ table “tbl_a”, user “user”, password “pass”, host “DB2 DB3 DB4”, tbl_a mbk “2”, mkd “2”, msi “DB5_sid”, link_status “0 1 1” DB1 tbl_a DB7 ‘; DB6 DB5 Alter table on DB1.
  • 42. How to add new node after failover and preparing new server without stopping the service
  • 43. Create a table of a new node to the clustered table You need to create a new node, in order to maintain redundancy, when there is a trouble at the node that composes the cluster. Here, I will explain how to add a table of a new node, without stopping the service. 'Monitoring node' in this slide is a node that works to observe the trouble of each node that composes clustering. 'Spider_copy_tables' in this slide is still in development , it will be available in future releases.
  • 44. Initial Structure tbl_a tbl_a tbl_a DB2 DB3 DB4 tbl_a DB1 tbl_a DB7 DB6 DB5 There are 4 MySQL servers with Spider (include 3 monitoring nodes) and 3 MySQL servers without Spider (including 1 broken node).
  • 45. Step 1 tbl_a tbl_a tbl_a tbl_a DB8 DB2 DB3 DB4 Create table tbl_a ( tbl_a col_a int, col_b int, DB1 tbl_a DB7 primary key(col_a) DB6 ) engine = InnoDB; DB5 Add new data node(DB8) and table.
  • 46. Step 2 tbl_a tbl_a tbl_a tbl_a DB8 DB2 DB3 DB4 Alter table tbl_a tbl_a Connection ‘ table “tbl_a”, user “user”, DB1 tbl_a DB7 password “pass”, DB6 host “DB2 DB4 DB8” DB5 ‘; Alter table on monitoring nodes (DB5, DB6 and DB7).
  • 47. Step 3 tbl_a tbl_a tbl_a tbl_a DB8 DB2 DB3 DB4 Alter table tbl_a Connection ‘ table “tbl_a”, user “user”, password “pass”, host “DB2 DB4 DB8”, tbl_a mbk “2”, mkd “2”, msi “DB5_sid”, DB1 tbl_a DB7 link_status “0 0 2” DB6 ‘; DB5 Alter table on DB1.
  • 48. Step 4 tbl_a tbl_a tbl_a tbl_a DB8 DB2 DB3 DB4 tbl_a Select spider_copy_tables(‘tbl_a’, ‘’, ‘’); DB1 tbl_a DB7 DB6 DB5 Copy data from DB2 to DB8.
  • 49. Finish tbl_a tbl_a tbl_a tbl_a DB8 DB2 DB3 DB4 Alter table tbl_a Connection ‘ table “tbl_a”, user “user”, password “pass”, host “DB2 DB4 DB8”, tbl_a mbk “2”, mkd “2”, msi “DB5_sid”, DB1 tbl_a DB7 link_status “0 0 1” DB6 ‘; DB5 Alter table on DB1.
  • 50. How to avoid table partitioning UNIQUE column limitation without stopping the service
  • 51. How to avoid table partitioning UNIQUE column limitation Right now, there is a restriction of MySQL that you cannot partition in other columns when there is a PK or UNIQUE. Here, I will show you how to partition a table by any columns even if there is a PK or UNIQUE.
  • 52. Initial Structure tbl_a Create table tbl_a ( col_a int, DB1 col_b int, primary key(col_a) ) engine = InnoDB; There is 1 MySQL server.
  • 53. Step 1 (for avoiding partitioning limitation) Create table tbl_a3 ( col_a int, Create table tbl_a2 ( primary key(col_a) tbl_a col_a int, ) engine = InnoDB col_b int, partition by primary key(col_a) linear hash(col_a) tbl_a2 ) engine = InnoDB; partitions 4; Create table tbl_a5 ( Create table tbl_a4 ( tbl_a3 col_a int, col_a int, col_b int, col_b int, primary key(col_a) key idx1(col_a), tbl_a4 ) engine = VP key idx2(col_b) Comment ‘ ) engine = InnoDB ctm “1”, ist “1”, partition by list( zru “1”, pcm “1” mod(col_b, 2)) ( tbl_a5 ‘ partition pt1 values in(0), Connection ‘ ); partition pt2 values in(1) DB1 ‘; tnl “tbl_a2 tbl_a3 tbl_a4” Create tables on DB1.
  • 54. Step 2 tbl_a2 tbl_a6 tbl_a3 tbl_a4 tbl_a DB1 Rename table on DB1. (rename table tbl_a2 to tbl_a6, tbl_a to tbl_a2, tbl_a5 to tbl_a)
  • 55. Step 3 tbl_a2 tbl_a6 tbl_a3 tbl_a4 tbl_a DB1 Copy data from tbl_a2 to tbl_a3 and tbl_a4. (select vp_copy_tables(‘tbl_a’, ‘tbl_a2’, ‘tbl_a3 tbl_a4’))
  • 56. Step 4 tbl_a2 tbl_a6 tbl_a3 Alter table tbl_a tbl_a4 Comment ‘ ctm “1”, ist “1”, pcm “1” ‘, tbl_a Connection ‘ tnl “tbl_a3 tbl_a4” DB1 ‘; Alter table tbl_a.
  • 57. Finish tbl_a3 tbl_a4 tbl_a DB1 Drop table. (drop table tbl_a2, tbl_a6)
  • 59. About MicroAd MicroAd is an advatising company. This company can advertise efficiently using "behavioral targeting" technology. 【MicroAd, Inc.] http://www.microad.jp/english/
  • 60. The previous architecture …… …… AP AP AP AP LVS Slave Slave DB DB Register new statistical rules replication from batch server Master Batch DB Batch processing updates new statistical rules every day. (For every advertisers, every advertising medias and every users)
  • 61. The problem with business expansion Increase data and request. At that time the limit of updates were 20 million records a day. They needed to update 100 million records a day. They also wanted to improve the performance of the reference slave by decreasing the amount of the update by one slave. They did not want to change or modify their application to support the increase. Then, Spider was used.
  • 62. The architecture with Spider …… AP AP AP AP …… with Spider with Spider with Spider with Spider Spider sharding LVS LVS LVS SlaveDB SlaveDB SlaveDB SlaveDB SlaveDB SlaveDB replication replication replication MasterDB MasterDB MasterDB Spider sharding Register new statistical rules from batch server SpiderDB (MySQL with Spider) Batch They created the shards with the unit of the replication.
  • 63. Resolved the problem As a result, They achieved update 100 million records a day and improved the performance of the reference. They didn't need to change or modify their applications so much. They are planning in the near future of resharding, when they expand the business.
  • 64. Any Questions? Thank you for taking your time!! Kentoku SHIBA (kentokushiba at gmail dot com) http://wild-growth.blogspot.com/ http://spiderformysql.com