1. Jeff Miller
Principal Program Manager
Jaeque (JQ) Yoon
Software Development Manager
Building Expedia’s
Travel Graph
using MongoDB

2. What you imagine…
…what you get.
https://flic.kr/p/6h4u72

3. How many combinations?

4. Actual existing travel planning tool.

5. Mental image…
https://flic.kr/p/5nduiR
…reality.
https://flic.kr/p/8bxbnE

6. Competitive marketplace
Changing all the time
Remember and connect everything.
Travel Graph Vision:

7. Scratchpad Vision: No more paper notes.
Automatic notetaking
Live updates
Multiple devices

8. Functional requirements

9. Photo © 2012 Jonny Ross (https://flic.kr/p/cQamT9)
20-40M writes/day
<1 sec latency
40-60M reads/day
Non-Functional Requirements
Six months to market

10. Scratchpad stack
Listen +
Distribute
Buffer
Store
Retrieve
Amazon SQS
MongoDB
History Service
Writer
Activity Broker
Record

11. • Three-node replicaset
– Multiple availability zones
– Reads from secondaries
• TTL Index to enforce data
retention limit
• Working set not in memory
• Single shard
Scratchpad: target launch configuration
Config
Config
Config
Mongo-s
Shard 1
Primary
Secondary Secondary
History Service
Mongo-s

12. June
Projected delivery timeline
August
July
Complete
Design
April May
Release
Target
Cluster
Build out
Enable Write
Traffic

13. Scratchpad data structures: v1
{
"_id" : "110c71e5-8476-4325-bec7-91659720faa6",
"key" : "110c71e5-8476-4325-bec7-91659720faa6-1",
"uid" : "110c71e5-8476-4325-bec7-91659720faa6",
"timestamp" : 12332323,
"hotel" : {
"details" : [
{
"rooms" : [
{
"roomtype": null,
"numadults": 1,
"numchildren": null
}],
"hotelid" : 87,
"checkindate" : "07-24-2014",
"checkoutdate" : "07-26-2014",
"price" : 255.00,
"timestamp" : 12332323
}],
},
"flight" : {
"searches" :
[ FlightSearchInteraction ],
}
}

14. {
"_id" : "110c71e5-8476-4325-bec7-91659720faa6-
hotels-detail-87-2807186400000",
"key" : "110c71e5-8476-4325-bec7-91659720faa6-
1"
"rooms" : [
{
"roomtype" : null,
"numadults" : 1,
"children" : null
}
],
"checkindate" : "07-24-2014",
"checkoutdate" : "07-26-2014",
"price" : 255.00,
"hotelid" : 87,
"regionId" : 800077,
"product" : "HOTEL",
"uid" : "110c71e5-8476-4325-bec7-91659720faa6",
"timestamp" : 12332323,
"type" : "DETAIL",
"ttl" : ISODate("2014-05-13T15:14:35.916Z")
}
Scratchpad data structures: v2

15. Complete
Design
June
Post-launch timeline
August
July
Live Traffic System Falls
Over
April
Cluster
Build out
Perf Testing
May

16. IOPS
Increased IOPS
IO Saturation

17. TTL deletes
Increased Delete Volume
Response Time Variation
TTL Deletes Trim Job Diff %Change
112 69 -43 -38.4%
History Average Response Time

18. July
Optimized for production
October
Aug
Live Write
Traffic Performance
Optimized
April
Official Release:
Live Read Traffic
• Provisioned IOPS
• Additional shard
• Reduced Write Load
Complete
Design

19. Scratchpad in the wild

20. My notes, everywhere.

21. My notes, everywhere.

22. No, wait, they were right here…

23. Photo © 2012 Vic De Leon (https://flic.kr/p/ebH68m)
Wait, they were here just a minute ago…

24. Easy come, easy go!
Config
Config
Config
Mongo-s
Shard 1
Primary
Secondary Secondary
History Service
Mongo-s
Shard 2
Primary
Secondary Secondary
Uh… now what?

25. Config
Mongo-s
EBS volumes, but no servers
Shard 1
EBS EBS EBS
Shard 2
EBS EBS EBS

26. Config
Mongo-s
New Cluster
Shard 1
EBS EBS EBS
Shard 2
EBS EBS EBS
Shard 1B
EBS EBS EBS
Shard 2B
EBS EBS EBS
sh.setBalancerState(false)

27. Config
Mongo-s
Swap Shards
Shard 1
EBS EBS EBS
Shard 2
EBS EBS EBS
Shard 1B
EBS EBS EBS
Shard 2B
EBS EBS EBS
db.shards.update(
{
_id: "travelgraphuhs"} ,
{$set: {"host" : "travelgraphuhs/
10.0.21.107:27017,10.0.21.225:27017,10.0.
21.189:27017”
}
})

28. Config
Mongo-s
Unmount Volumes
Shard 1
EBS EBS EBS
Shard 2
EBS EBS EBS
Shard 1B
EBS EBS EBS
Shard 2B
EBS EBS EBS

29. Config
Mongo-s
Swap Volumes
Shard 1
Shard 2
Shard 1B
EBS EBS EBS
Shard 2B
EBS EBS EBS

30. Config
Mongo-s
Restored
Shard 1B
EBS EBS EBS
Shard 2B
EBS EBS EBS

31. Scratchpad Today/Tomorrow
Config
Config
Config
Mongo-s
History Service
Mongo-s
Shard 1
Primary
Secondary
Secondary
Analytics
Shard 2
Primary
Secondary
Secondary
Analytics
Shard 3
Primary
Secondary
Secondary
Analytics
• 3 shards serving ~80
million reads/writes
daily
• 4 node replicaset
• Increased data
retention
• Geo distribution of
secondary reads

32. Special-Purpose Tools Joined Together
Travel Graph Ecosystem

33. Jeff Miller
Principal Program Manager
Jaeque (JQ) Yoon
Software Development Manager
THANK YOU!