Sharding talk from MongoSV 2011.

1. Sharding
Jared Rosoff (@forjared)

2. Overview
• Architecture
• How it works
• Use Cases

3. SHARDING
ARCHITECTURE

4. Architecture
Shards
mongod mongod mongod
...
mongod mongod mongod
mongod mongod mongod
Config
Servers
mongod
mongos mongos ...
mongod
mongod
client client client client

5. mongos
• Shard Router
Shards
• Acts just like a MongoD
mongod mongod mongod
mongod mongod mongod
...
• 1 or as many as you
Config
mongod mongod mongod want
Servers
mongod
• Can run on App
mongod
mongos mongos ...
Servers
• Caches meta-data from
mongod
client client client client
config servers

6. Config Server
• 3 of them
Shards
• Changes use 2 phase
mongod mongod mongod
mongod mongod mongod
... commit
Config
mongod mongod mongod • If any are down, meta
Servers
mongod
data goes read only
mongod
mongos mongos ...
• System is online as
mongod
client client client client
long as 1/3 is up

7. HOW IT WORKS

8. Keys
> db.runCommand( { shardcollection: “test.users”,
key: { email: 1 }} )
{
name: “Jared”,
email: “jsr@10gen.com”,
}
{
name: “Scott”,
email: “scott@10gen.com”,
}
{
name: “Dan”,
email: “dan@10gen.com”,
}

9. Chunks
-∞ +
∞

10. Chunks
-∞ +
∞
dan@10gen.com scott@10gen.com
jsr@10gen.com

11. Chunks
Split!
-∞ +
∞
dan@10gen.com scott@10gen.com
jsr@10gen.com

12. Chunks
This is a Split! This is a
chunk chunk
-∞ +
∞
dan@10gen.com scott@10gen.com
jsr@10gen.com

13. Chunks
-∞ +
∞
dan@10gen.com scott@10gen.com
jsr@10gen.com

14. Chunks
-∞ +
∞
dan@10gen.com scott@10gen.com
jsr@10gen.com

15. Chunks
Split!
-∞ +
∞
dan@10gen.com scott@10gen.com
jsr@10gen.com

16. Chunks
Min Key Max Key Shard
-∞ adam@10gen.com 1
adam@10gen.com jared@10gen.com 1
jared@10gen.com scott@10gen.com 1
scott@10gen.com +∞ 1
• Stored in the config servers
• Cached in MongoS
• Used to route requests and keep cluster
balanced

17. Balancing
mongos
config
balancer
config
Chunks!
config
1 2 3 4 13 14 15 16 25 26 27 28 37 38 39 40
5 6 7 8 17 18 19 20 29 30 31 32 41 42 43 44
9 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48
Shard 1 Shard 2 Shard 3 Shard 4

18. Balancing
mongos
config
balancer
config
Imbalance
Imbalance config
1 2 3 4
5 6 7 8
9 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48
Shard 1 Shard 2 Shard 3 Shard 4

19. Balancing
mongos
config
balancer
config
Move chunk 1 config
to Shard 2
1 2 3 4
5 6 7 8
9 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48
Shard 1 Shard 2 Shard 3 Shard 4

20. Balancing
mongos
config
balancer
config
config
1 2 3 4
5 6 7 8
9 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48
Shard 1 Shard 2 Shard 3 Shard 4

21. Balancing
mongos
config
balancer
config
Chunk 1 now lives
on Shard 2 config
2 3 4
5 6 7 8 1
9 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48
Shard 1 Shard 2 Shard 3 Shard 4

22. ROUTING

23. Routed Request
1
1. Query arrives at
4
mongos MongoS
2. MongoS routes query
to a single shard
3. Shard returns results
of query
2
4. Results returned to
client
3
Shard 1 Shard 2 Shard 3

24. Scatter Gather
1
4 1. Query arrives at
mongos MongoS
2. MongoS broadcasts
query to all shards
3. Each shard returns
results for query
2 4. Results combined
2 2 and returned to client
3
3 3
Shard 1 Shard 2 Shard 3

25. Distributed Merge Sort
1
6 1. Query arrives at
mongos MongoS
2. MongoS broadcasts
5
query to all shards
3. Each shard locally
sorts results
2 4. Results returned to
2 2 mongos
4 4 4 5. MongoS merge sorts
individual results
3 6. Combined sorted
3 3
result returned to
Shard 1 Shard 2 Shard 3 client

26. Writes
Inserts Requires shard db.users.insert({
key name: “Jared”,
email: “jsr@10gen.com”})
Removes Routed db.users.delete({
email: “jsr@10gen.com”})
Scattered db.users.delete({name: “Jared”})
Updates Routed db.users.update(
{email: “jsr@10gen.com”},
{$set: { state: “CA”}})
Scattered db.users.update(
{state: “FZ”},
{$set:{ state: “CA”}} )

27. Queries
By Shard Routed db.users.find(
{email: “jsr@10gen.com”})
Key
Sorted by Routed in order db.users.find().sort({email:-1})
shard key
Find by non Scatter Gather db.users.find({state:”CA”})
shard key
Sorted by Distributed db.users.find().sort({state:1})
merge sort
non shard
key

28. EXAMPLES

29. User Profiles
{
• Shard by email
name: “Jared”,
email: “jsr@10gen.com”, • Lookup by email hits
addresses: [ 1 node
{state: “CA”}
] • Index on
} {“addresses.state”:1}

30. Activity Stream
{
• Shard by user_id
user_id: “jsr@10gen.com”,
event_id: “Logged in”, • Looking up a stream
data: “…” hits 1 node
}
• Writing is evenly
distributed
• Index on {“event_id”:1}
for deletes

31. Photos
{
• What’s the right key?
photo_id: ???,
data: BinData(…) – Auto Increment?
} – MD5( data )
– Now() + MD5(data)
– Month() + MD5(data)