In his All Your Base talk, David Mytton (founder of Server Density) will talk you through our experiences in handling large scale MongoDB deployments.

1. Experiences from production
Deployment, performance, failure
David Mytton
All Your Base - Oct 2014
blog.serverdensity.com

2. David Mytton

3. serverdensity.com/allyourbase

4. Slides: twitter.com/davidmytton

5. Agenda
● Architecture
● Performance
● Where to host?
● Downtime
● Preparation

6. Server Density Architecture

7. Server Density Architecture
● ~100 servers - Ubuntu 12.04

8. Server Density Architecture
● ~100 servers - Ubuntu 12.04
● 50:50 virtual/dedicated

9. Server Density Architecture
● ~100 servers - Ubuntu 12.04
● 50:50 virtual/dedicated
● 200TB/m processed data

10. Server Density Architecture
● ~100 servers - Ubuntu 12.04
● 50:50 virtual/dedicated
● 200TB/m processed data
● Nginx, Python, MongoDB

11. Server Density Architecture
● ~100 servers - Ubuntu 12.04
● 50:50 virtual/dedicated
● 200TB/m processed data
● Nginx, Python, MongoDB
● Softlayer > 1TB RAM, 5TB SSDs

12. Two choices for deployment

13. Two choices for deployment
● Virtualized
● Bare metal

14. Advantages of virtualization
● Easy to manage

15. Advantages of virtualization
● Easy to manage
● Fast boot

16. Advantages of virtualization
● Easy to manage
● Fast boot
● Easier to resize/migrate

17. Advantages of virtualization
● Easy to manage
● Fast boot
● Easier to resize/migrate
● Templating/snapshots

18. Advantages of virtualization
● Easy to manage
● Fast boot
● Easier to resize/migrate
● Templating/snapshots
● Containment

19. Disadvantages of virtualization
● Another layer

20. Disadvantages of virtualization
● Another layer
● Hypervisor overhead

21. Disadvantages of virtualization
● Another layer
● Hypervisor overhead
● Host contention

22. Disadvantages of virtualization
● Another layer
● Hypervisor overhead
● Host contention
● i/o performance

23. Advantages of bare metal
● Dedicated resources

24. Advantages of bare metal
● Dedicated resources
● Direct access to hardware

25. Advantages of bare metal
● Dedicated resources
● Direct access to hardware
● Customisable specs

26. Advantages of bare metal
● Dedicated resources
● Direct access to hardware
● Customisable specs
● Performance

27. Disadvantages of bare metal
● Build/deploy time

28. Disadvantages of bare metal
● Build/deploy time
● More difficult to resize

29. Disadvantages of bare metal
● Build/deploy time
● More difficult to resize
● Difficult to migrate/snapshot

30. Disadvantages of bare metal
● Build/deploy time
● More difficult to resize
● Difficult to migrate/snapshot
● Capex/lifetime

31. Performance problems?

32. Performance problems?
Easy answer: move to bare metal!

33. Key performance factors
● Network

34. Key performance factors
● Network
● EC2: Cluster compute,
high memory, high i/o,
high storage
● GCE: Higher CPU
instances

35. Key performance factors
● Network

36. Key performance factors
● Network
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Pacific 150ms
Europe-Japan 300ms

37. Networking performance
AWS
GCE
bit.ly/googlevsamazon

38. Key performance factors
● Memory

39. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

40. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

41. Key performance factors
● Memory is expensive

42. Key performance factors
● Disk
● SSDs!

43. Key performance factors
● Disk
● SSDs!
GCE: 256GB = $83.20/m
EC2: 256GB = $35.32/m
SL: 200GB = $81/m

44. Why cloud?
● Flexible

45. Why cloud?
● Flexible
● Unlimited resources

46. Why cloud?
● Flexible
● Unlimited resources
● Cheap to get started

47. Why cloud?
● Flexible
● Unlimited resources
● Cheap to get started
● Other products

48. Why colo?

49. Why colo?
● Vastly cheaper

50. Why colo?
● Vastly cheaper
● Complete control

51. Let’s talk about downtime

52. 2013 Spend: ~$5bn

53. 2013 Spend: ~$6bn

54. 2013 Spend: ~$4bn

55. How much do you spend?
You will have downtime

56. Preparation

57. Preparation - On Call
● Rotations

58. Preparation - On Call
● Rotations
● Off call

59. Preparation - On Call
● Rotations
● Off call
● Reachability - Train, 3G/4G
(edge?!), Do Not Disturb mode,
system updates
● Work the next day?

60. Preparation - On Call
● Rotations
● Off call
● Reachability - Train, 3G/4G
(edge?!), Do Not Disturb mode,
● Work system the updates
next day?
● Work the next day?

61. Preparation - Documentation

62. Preparation - Documentation
● Searchable

63. Preparation - Documentation
● Searchable
● Easy to edit

64. Preparation - Documentation
● Searchable
● Easy to edit
● Independent of your infrastructure

65. Preparation - Documentation
● Searchable
● Easy to edit
● Independent of your infrastructure
● Up to date

67. Unexpected failures

68. Unexpected failures
● Communication systems

69. Unexpected failures
● Communication systems
● Network connectivity

70. Unexpected failures
● Communication systems
● Network connectivity
● Access to support

71. ALERT!

72. ALERT!
1. Load up incident response checklist

73. ALERT!
1. Load up incident response checklist
2. Log incident in JIRA

74. ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room

75. ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room
4. Public status post

76. ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room
4. Public status post
5. Initial investigation

77. Key response principles

78. Key response principles
● Log everything

79. Key response principles
● Log everything
● Frequent public status updates

80. Key response principles
● Log everything
● Frequent public status updates
● Gather the team

81. Key response principles
● Log everything
● Frequent public status updates
● Gather the team
● Escalate!

82. Summary
● Architecture
● Performance
● Where to host?
● Downtime
● Preparation

83. どもありがとうございます
＠davidmytton
david@serverdensity.com
blog.serverdensity.com
serverdensity.com/allyourbase