MySQL Fabric

1. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Fabric
-­‐
High
Availability
&
Automated
Sharding
for
MySQL
London
October
2014
Mark
Swarbrick
Sales
Consultant
MySQL
UK&I
Sept
2014

2. Safe
Harbor
Statement
The
following
is
intended
to
outline
our
general
product
direcPon.
It
is
intended
for
informaPon
purposes
only,
and
may
not
be
incorporated
into
any
contract.
It
is
not
a
commitment
to
deliver
any
material,
code,
or
funcPonality,
and
should
not
be
relied
upon
in
making
purchasing
decisions.
The
development,
release,
and
Pming
of
any
features
or
funcPonality
described
for
Oracle’s
products
remains
at
the
sole
discrePon
of
Oracle.
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
2

3. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Program
Agenda
1
2
3
4
5
Requirements
for
Next
Gen
Services
Simple,
transparent
High
Availability
Delivering
SQL
&
ACID
at
scale
Where
MySQL
Fabric
fits
with
Oracle’s
other
MySQL
soluPons
Geng
started
3

4. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Program
Agenda
4
1
2
3
4
5
Requirements
for
Next
Gen
Services
Simple,
transparent
High
Availability
Delivering
SQL
&
ACID
at
scale
Where
MySQL
Fabric
fits
with
Oracle’s
other
MySQL
soluPons
Geng
started

5. Business
PrioriPes
for
IT
Focus
on
driving
the
business
rather
than
on
infrastructure
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Deliver
new
services
Real-­‐Time
Insight
from
Data
Simplify
Architecture
Fast
Time
to
Market
5

6. Responsive
&
Agile
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
ApplicaPon
Requirements
High
Availability
Scale
Data
&
User
Loads
OLTP
&
AnalyPcs
ElasPc
6

7. An
extensible
and
easy-­‐to-­‐use
framework
for
managing
a
farm
of
MySQL
server
supporPng
high-­‐availability
and
sharding
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Fabric
7

8. MySQL
Fabric
1.4
High
Availability
+
Sharding-­‐Based
Scale-­‐out
• High
Availability:
• Server
monitoring
with
auto-­‐promoPon
and
transparent
applicaPon
failover
• Fabric-­‐aware
connectors
rather
than
proxy:
Python,
Java
&
PHP
• Lower
latency,
boeleneck-­‐free
• OpPonally
scale-­‐out
through
sharding
• ApplicaPon
provides
shard
key
• Range
or
Hash
• Tools
for
resharding
• Global
updates
&
tables
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Fabric
ApplicaPon
Connector
Read-­‐slaves
mappings
SQL
HA
group
Read-­‐slaves
HA
Group
8

9. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Program
Agenda
9
1
2
3
4
5
Requirements
for
Next
Gen
Services
Simple,
transparent
High
Availability
Delivering
SQL
&
ACID
at
scale
Where
MySQL
Fabric
fits
with
Oracle’s
other
MySQL
soluPons
Geng
started

10. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Layers
of
HA
Redundant
Access
to
Data
Data
Redundancy
Redundant
App
Servers
RouJng
to
the
Data
10

11. MySQL
Fabric
Framework
(HA)
All Data
Primary Secondary
Extra Read
Replicas
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Fabric
Controller
SQL
Queries
State &
Routing Info
HA Group
Coordination
and Control

12. MySQL
ReplicaPon
&
MySQL
Fabric
HA
&
how
this
effects
failover
• MySQL
ReplicaPon
is
the
iniPal
implementaPon
used
in
HA
Groups
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• PRIMARY
=
ReplicaPon
Master
&
receives
all
writes
• Failover
• MySQL
Fabric
detects
failure
of
PRIMARY/Master
• Selects
a
SECONDARY/Slave
and
promotes
it
• Updates
State
Store
• Pushes
state
change
to
Fabric-­‐aware
connectors
12

13. High-­‐Availability
Group
Concept
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Abstract
Concept
• Set
of
servers
• Server
aeributes
• Connector
Aeributes
• ConnecPon
informaPon
• Mode:
read-­‐only,
read-­‐write,
...
• Weight:
distribute
load
• Management
Aeributes
• State:
state/role
of
the
server
State:
Primary
Mode:
Read-­‐Write
Host:
server-­‐1.example.com
13

14. Create
HA
Groups
and
add
Servers
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Define
a
group
mysqlfabric group create my_group
• Add
servers
to
group
mysqlfabric group add my_group server1.example.com
mysqlfabric group add my_group server2.example.com
14

15. Create
HA
Groups
and
add
Servers
• Promote
one
server
to
be
primary
mysqlfabric group promote my_group
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Tell
failure
detector
to
monitor
group
mysqlfabric group activate my_group
15

16. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Program
Agenda
16
1
2
3
4
5
Requirements
for
Next
Gen
Services
Simple,
transparent
High
Availability
Delivering
SQL
&
ACID
at
scale
Where
MySQL
Fabric
fits
with
Oracle’s
other
MySQL
soluPons
Geng
started

17. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
The
Path
to
Scalability
Scaling-­‐Up
can
take
you
a
long
way
Scaling
on
dense,
mulP-­‐
core,
mulP-­‐thread
servers
• 10s
-­‐
100GBs
RAM
• SSDs
Scale
across
cores
within
a
single
instance
You
can
get
a
long
way
with
MySQL
5.6!

18. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Benefits
of
Sharding
• Write
scalability
• Can
handle
more
writes
• Large
data
set
• Database
too
large
• Does
not
fit
on
single
server
• Improved
performance
• Smaller
index
size
• Smaller
working
set
• Improve
performance
UID
10000-­‐20000
UID
20001-­‐40000
18

19. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Fabric
Features
• Connector
API
Extensions
• Support
TransacPons
• Support
full
SQL
• Decision
logic
in
connector
• Reducing
network
load
• Shard
MulPple
Tables
• Using
same
key
• Global
Updates
• Global
tables
• Schema
updates
• Sharding
FuncPons
• Range
• (Consistent)
Hash
• Shard
OperaPons
• Shard
move
• Shard
split
19

20. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Sharding
Architecture
Shards
MySQL
Fabric
Node
ApplicaJon
Global
Group
Global
Updates
Shard
Updates
ReplicaJon
20

21. MySQL
Fabric
(HA
+
Sharding)
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Global Data
Primary Secondary
Shard 1
Primary Secondary
Shard 2
Primary Secondary
MySQL
Fabric
Controller
SQL
Queries
Server/Shard State
& Mapping
Global Group HA Group
Coordination
and Control
HA Group
Extra Read
Replicas
Extra Read
Replicas

22. Shard #1
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
RouPng
TransacPons
Shard #2
Shard #3
Executor
App
Server
Connector
Cache
State
Store
App
Server
Connector
Cache
22

23. MySQL
Fabric:
Sharding
Setup
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Set
up
some
groups
• my_global
–
for
global
updates
• my_group.N
–
for
the
shards
• Add
servers
to
the
groups
• Create
a
shard
mapping
• A
“distributed
database”
• Mapping
keys
to
shards
• Give
informaPon
on
what
tables
are
sharded
• Add
shards
23

24. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Fabric:
Moving
and
Spling
Shards
• Moving
a
shard
(id=5)
from
exisPng
group
to
another
(my_group.8)
• mysqlfabric
sharding
move
5
my_group.8
• Spling
a
shard
(id=5)
into
two
parts
with
new
half
stored
in
group
my_group.6
• mysqlfabric
sharding
split
5
my_group.6
24

25. • Provide
sharding
key
• Property:
key
• Fabric
will
compute
shard
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Connector
API:
Shard
Specific
Query
• Indicate
tables
to
be
used
in
query
• Property:
tables
• Fabric
will
compute
map
conn.set_property(tables=["test.subscribers"], key=sub_no,
mode=fabric.MODE_READWRITE)
cur = conn.cursor()
cur.execute(
"INSERT INTO subscribers VALUES (%s, %s, %s)",
(sub_no, first_name, last_name))
25

26. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Connector
API:
Shard
Specific
Query
• Provide
tables
in
query
– Property:
tables
– Fabric
will
compute
map
conn.set_property(tables=["test.subscribers"], key=sub_no,
mode=fabric.MODE_READONLY)
cur = conn.cursor()
cur.execute(
"SELECT first_name, last_name FROM subscribers "
"WHERE sub_no = %s", (sub_no, )
)
for row in cur:
print row
§ Provide sharding key
– Property: key
– Fabric will compute shard
§ Joins within the shard (or with
global tables) supported
26

27. Connector
API:
Global
Update
• Set
global
scope
– Property:
scope
– Query
goes
to
global
group
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Provide
tables
in
query
– Property:
tables
– Fabric
will
compute
map
– (Likely
to
not
be
needed)
conn.set_property(tables=[’test.subscribers'], scope='GLOBAL')
cur = conn.cursor()
cur.execute("ALTER TABLE test.subscirbers ADD nickname VARCHAR(64)")
27

28. Protocols
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Fabric
Node
MySQL
MySQL
Fabric
Framework
Executor
State
Store
(Persister)
Sh
HA
AMQP
MySQL
XML-­‐RPC
?
?
Extensions
Backing
Store
Extensible
Architecture
28

29. MySQL
Fabric:
Goals
&
Features
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Connector
API
Extensions
• Support
TransacPons
• Support
full
SQL
• Fabric-­‐Aware
Connectors
at
GA:
• PHP
+
Doctrine,
Python,
Java
+
Hibernate
• Decision
logic
in
connector
• Reducing
network
load
• Load
Balancing
• Read-­‐Write
Split
• Distribute
transacPons
• Global
Updates
• Global
tables
• Schema
updates
• Shard
MulPple
Tables
• Using
same
key
• Sharding
FuncPons
• Range
• (Consistent)
Hash
• Shard
OperaPons
• Shard
move
• Shard
split
29

30. MySQL
Fabric
–
Current
LimitaPons
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• RouPng
is
dependent
on
Fabric-­‐aware
connectors
• Currently
Java
(+
Hibernate),
PHP
(+
Doctrine)
&
Python
• MySQL
Fabric
node
is
a
single
(non-­‐
redundant
process)
• HA
Maintained
as
connectors
conPnue
to
route
using
local
caches
• Establishes
asynchronous
replicaPon
• Manual
steps
to
switch
to
semisynchronous
• Sharding
not
completely
transparent
to
applicaPon
(must
provide
shard
key
–
column
from
applicaPon
schema)
• No
cross-­‐shard
joins
or
other
queries
• Management
is
through
CLI
or
XML/RPC
API
• No
GUI
30

31. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Program
Agenda
31
1
2
3
4
5
Requirements
for
Next
Gen
Services
Simple,
transparent
High
Availability
Delivering
SQL
&
ACID
at
scale
Where
MySQL
Fabric
fits
with
Oracle’s
other
MySQL
soluPons
Geng
started

32. Oracle
MySQL
HA
&
Scaling
SoluPons
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
ReplicaJon
MySQL
Fabric
Oracle
VM
Template
Oracle
Clusterware
Solaris
Cluster
Windows
Cluster
DRBD
MySQL
Cluster
App
Auto-­‐Failover
✖
✔
✔
✔
✔
✔
✔
✔
Data
Layer
Auto-­‐Failover
✖
✔
✔
✔
✔
✔
✔
✔
Zero
Data
Loss
MySQL
5.7
MySQL
5.7
✔
✔
✔
✔
✔
✔
Plauorm
Support
All
All
Linux
Linux
Solaris
Windows
Linux
All
Clustering
Mode
Master
+
Slaves
Master
+
Slaves
AcJve/
Passive
AcJve/Passive
AcJve/
Passive
AcJve/
Passive
AcJve/
Passive
MulJ-­‐
Master
Failover
Time
N/A
Secs
Secs
+
Secs
+
Secs
+
Secs
+
Secs
+
<
1
Sec
Scale-­‐out
Reads
✔
✖
✖
✖
✖
✖
✔
Cross-­‐shard
operaPons
N/A
✖
N/A
N/A
N/A
N/A
N/A
✔
Transparent
rouPng
✖
For
HA
✔
✔
✔
✔
✔
✔
Shared
Nothing
✔
✔
✖
✖
✖
✖
✔
✔
Storage
Engine
InnoDB+
InnoDB+
InnoDB+
InnoDB+
InnoDB+
InnoDB+
InnoDB+
NDB
Single
Vendor
Support
✔
✔
✔
✔
✔
✖
✔
✔
32

33. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Program
Agenda
33
1
2
3
4
5
Requirements
for
Next
Gen
Services
Simple,
transparent
High
Availability
Delivering
SQL
&
ACID
at
scale
Where
MySQL
Fabric
fits
with
Oracle’s
other
MySQL
soluPons
Geng
started

34. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Fabric
Resources
• Download
and
try
hep://dev.mysql.com/downloads/fabric/
• DocumentaPon
hep://dev.mysql.com/doc/mysql-­‐uPliPes/1.4/en/fabric.html
• Forum
(MySQL
Fabric,
Sharding,
HA,
UPliPes)
hep://forums.mysql.com/list.php?144
• Tutorial:
MySQL
Fabric
-­‐
adding
High
Availability
and
Scaling
to
MySQL
hep://www.clusterdb.com/mysql-­‐fabric/mysql-­‐fabric-­‐adding-­‐high-­‐availability-­‐and-­‐scaling-­‐to-­‐mysql
• White
Paper:
MySQL
Fabric
-­‐
A
Guide
to
Managing
MySQL
High
Availability
and
Scaling
Out
hep://www.mysql.com/why-­‐mysql/white-­‐papers/mysql-­‐fabric-­‐product-­‐guide
34