Talk given at DevOps Berlin Usergroup January 2015 http://www.meetup.com/blndevops/events/219636613/

1. Mapbox.com
Globally serving maps from 8
datacenters
@freenerd
johan@mapbox.com
Berlin DevOps 2015-01
1

2. • What is Mapbox
• Serving maps from 8 regions
• Deploying to 8 regions
2

3. 3

4. 4

5. 5

6. 6

7. 7

8. 8

9. 9

10. Application requirements
• Global high availability
• Global low latency
10

11. Application requirements
• Global high availability
• Global low latency
Usage pattern
• "The long tail of map requests"
11

12. What is a map?
12

13. 13

14. Serving tiles (layer-by-layer)
14

15. 15

16. Client
• Browser loads an embedded map
• Browser loads tile-by-tile
https://tiles.mapbox.com/v4/
map.id/17/70428/42997.png?access_token=pk.xxx
16

17. Client
https://tiles.mapbox.com/v4/
map.id/17/70428/42997.png?access_token=pk.xxx
17

18. Client
DNS
• Route53: Amazon DNS service
https://tiles.mapbox.com
18

19. Client → DNS
CDN (Content Distribution
Network)
• Cloudfront: AWS CDN
• Terminate TLS
• Cache based on url & query string
https://tiles.mapbox.com/v4/
map.id/17/70428/42997.png?access_token=pk.xxx
19

20. Client → DNS
CDN (Content Distribution
Network)
• find the nearest edge location
20

21. Client → DNS
CDN (Content Distribution
Network)
21

22. Client → DNS
CDN (Content Distribution
Network)
• find the nearest edge location
• TTL of 5 minutes
• ~35% cache hits
22

23. Client → DNS
CDN (Content Distribution
Network)
• On cache miss:
• Continue request to origin server
23

24. 24

25. Client → DNS → CDN
DNS
• Route53: Amazon DNS
25

26. Client → DNS → CDN
DNS
• Route53: Amazon DNS
• Mapbox.com serves maps from 8 regions
• Route to "best region for request based on latency"
26

27. Client → DNS → CDN
DNS
27

28. Client → DNS → CDN -> DNS
Load Balancer
• ELB: Amazon Elastic Load Balancer
• Terminate TLS
• Route requests to application servers
28

29. Client → DNS → CDN -> DNS -> Load Balancer
Application Servers
• EC2: Amazon Elastic Compute Cloud
• Virtual Machines
• EC2 instances are provisioned via Auto Scaling Group
• Auto Scaling is based on instance CPU load
• Scale up/down if CPU load over/under 55%/20% for 2
minutes
29

30. Client → DNS → CDN -> DNS -> Load Balancer
Application Servers
• Node.js/Express
• Authentication
• Determine location of tiles
• Fetch tiles and return them
30

31. Client → DNS → CDN -> DNS -> Load Balancer
Application Servers
Fetching tiles
• check simultanously in local cache (redis) and object store
(s3)
• return from where is found first
• if only found in object store, update local cache
31

32. Client → DNS → CDN -> DNS -> Load Balancer
Application Servers
Cache vs. Object Store
• redis: 50% cache hits
• (60% non-cdn = 30% total requests)
• redis is used as least-recently used cache, thus popular tiles
for a region are usually cached
• redis is provisioned by AWS via ElastiCache
32

33. 33

34. 34

35. 35

36. 36

37. Deploy to many regions
37

38. Deploy to many regions
• constantly have around 300 EC2 instances running
• last months spike was 1000 EC2 instances (batch jobs)
• can't care for individual EC2 instances
• every developer handles deployment themselves
38

39. 39

40. Deploying cloudformation stacks
mapbox-cli
• the tool, the infrastructure team gives to developers
• abstracts and automates common tasks
• encodes our conventions for using AWS
40

41. Screencast
41

42. Screencast
Link to video on youtube
https://www.youtube.com/watch?v=wmDvcDvc6Jw
42

43. Deployment strategies
• Sequential
• Parallel
• Probed
• one region first
• wait for alerts
• all other regions parallel
43

44. Deploys with interruptions
• Take region out of Route 53
• Wait for traffic to cease
• Update stack
• Put region back into Route 53
44

45. • mapbox-cli is built with node.js
• mostly based around the aws-js sdk
• github.com/mapbox
• cfn-config
• cfn-stack-event-stream
• as-replace-instances
45

46. Future directions
• deploybot
• send some traffic to staging/canary stacks
• docker for speeding up instance start time
46

47. We are hiring
• work asynchronously
• github & hipchat
• no emails
47

48. Thanks
• github.com/mapbox
• cfn-config
• cfn-stack-event-stream
• as-replace-instances
@freenerd
johan@mapbox.com
48