The slidedeck of my talk at MongoMunich 2012

1. Lean & Agile with MongoDB
MongoMunich 2012
#MongoDBMunich
@comsysto

2. About us
2

3. About us
• first partner of 10gen in Germany (January 2012)
3

4. About me
• Lead DevOps Engineer at comsysto
• @loomit
• Data Nerd
• 3 years of high performance web ops
• joined comSysto in March 2012
4

5. Questions
• Please ask during the presentation!
5

6. Lean?
6

7. Lean?
Continuous Innovation 7

8. Lean?
• Instant feedback from customers about
features
• eliminate waste
8

9. Eliminate waste
9

10. Agile?
• Iterative and incremental
10

11. SCRUM
• Scrum is a framework for developing and
sustaining complex products
11

12. Kanban
• Pull from a work queue
• originated at Toyota in the 1950s
12

13. Agile Adoption
• Ken Schwaber
13

14. Agile Adoption
• “There is no SCRUM police”
14

15. Agile Adoption
• “Use your intelligence”
15

16. Agile Adoption
• Dogmatic Slumber
16

17. Don’t be the little girl
17

18. Don’t be the Joker
18

19. Cross functional teams
19

20. Cross functional teams
20

21. 8 hats
21

22. Co-location
22

23. Appreciation for simplicity
• “Everything should be as simple as possible,
but not simpler”
• paraphrased Albert Einstein
23

24. Look familiar?
24

25. NOSQL
25

26. Schema Free
“Your data schema is a direct corollary with how you view your
business’ direction and tech goals. When you pivot, especially if it’s
a significant one, your data may no longer make sense in the
context of that change. Give yourself room to breath. A schema-less
data model is MUCH easier to adapt to rapidly changing
requirements than a highly structured, rigidly enforced schema.”
from:
http://www.cleverkoala.com/2010/08/why-your-startup-should-be-
using-mongodb/
26

27. Emergent Architectures
27

28. Move fast and break things
28

29. NOSQL
29

30. Scale out
30

31. AWS
• MongoDB mostly I/O bound
• Storage matters
31

32. AWS
• EBS (anywhere from 70 to 300 ops/sec)
• EBS provisioned IOPS (stable)
• Ephemeral
• SSD (much higher ops/sec but costly)
• use RAID on EC2 (or not?)
32

33. MongoDB AWS Storage
33

34. AWS
• Naming really matters
– combine with Route 53
– ec2-174-129-227-92.compute-1.amazonaws.com?
34

35. Sharded Setup
35

36. MongoDB on AWS
36

37. Infrastructure as code
37

38. Use Cases
• Real-Time Analytics Software
• Operational Intelligence
• High Volume Data Feeds
• Hadoop
38

39. Patterns
• Pre Aggregation
• Batch
– Hadoop
– MapReduce (in MongoDB)
– Aggregation Framework
39

40. Pre-Aggregation
• Problem:
– You require up-to-the minute data, or up-to-the-second if
possible
– The queries for ranges of data (by time) must be as fast as
possible
40

41. Pre-Aggregation
• Best practises
– $inc and upsert are your friend
– pre-allocate documents
– use REST interface
41

42. Batch
• MapReduce
• Aggregation Framework
• Mongo-Hadoop Connector
42

43. Mongo Hadoop Connector
Data Storage Data Processing
43

44. Projects
• What we have done so far...
44

45. Real Time Twitter Heatmap
45

46. Real Time Twitter Heatmap
• The bubbles in the sea?
Friendly Floatees!
46

47. Friendly Floatees
47

48. Flow
48

49. Real Time Twitter Heatmap
• MongoDB Capped Collections
• Flask
• Redis
• Google Maps
• heatmaps.js
• Server-Sent Events
• http://bit.ly/Ou5SsP
49

50. Pizza Quattro Shardoni
50

51. Quattro Shardoni
• Technology Showcase Product
• Complete End2End stack
• Real Time Charting
• Batch Reporting based on Hadoop
51

52. Quattro Shardoni
52

53. Quattro Shardoni
53

54. Quattro Shardoni
54

55. Quattro Shardoni
• Vortrag heute 12:15 BallSaal A
Tom Zorc Bernd Zuther 55

56. Operational Intelligence
56

57. Operational Intelligence
• Analyze behavior of users in web shop
• Recommend NBA for business
• Real Time Analytics
57

58. Online Shop
REST
58

59. Operational Intelligence
• Next best activity for support/callcenter
• interpret user session
• e.g. “RaspberryPi - strong interest”
• exp. 2000 events per second
59

60. Operational Intelligence
60

61. Operational Intelligence
61

62. It’s Real Time!
62

63. Big Data Project
• “which analyzes and visualizes data of mobile
networks”
63

64. Big Data Project
64

65. Big Data Project
65

66. Big Data Project
66

67. Big Data Project
• started as prototype, in production now ;-)
66

68. Big Data Project
• started as prototype, in production now ;-)
• “beyond agile”
66

69. Big Data Project
• started as prototype, in production now ;-)
• “beyond agile”
• going from
66

70. Big Data Project
• started as prototype, in production now ;-)
• “beyond agile”
• going from
– fetch all, calculate in service layer
66

71. Big Data Project
• started as prototype, in production now ;-)
• “beyond agile”
• going from
– fetch all, calculate in service layer
– use MongoDB MapReduce on single node
66

72. Big Data Project
• started as prototype, in production now ;-)
• “beyond agile”
• going from
– fetch all, calculate in service layer
– use MongoDB MapReduce on single node
– use MongoDB MapReduce on 5 shards
66

73. Big Data Project
• started as prototype, in production now ;-)
• “beyond agile”
• going from
– fetch all, calculate in service layer
– use MongoDB MapReduce on single node
– use MongoDB MapReduce on 5 shards
– use MongoDB MapReduce on 24 shards (2
hi1.4xlarge instances)
66

74. Big Data Project
• started as prototype, in production now ;-)
• “beyond agile”
• going from
– fetch all, calculate in service layer
– use MongoDB MapReduce on single node
– use MongoDB MapReduce on 5 shards
– use MongoDB MapReduce on 24 shards (2
hi1.4xlarge instances)
– use EMR (around 10 m2.4xlarge instances)
66

75. Big Data Project
67

76. Big Data Project
68

77. Big Data Project
• why not use Aggregation Framework?
– we started with 2.0.6
– would have had to change data model
– M/R seemed the way to go (data size)
69

78. Big Data Project
• Numbers
– data comes in weekly increments
– 2TB raw data
– 14GB / week (into MongoDB)
– data grows in direct proportion to polygon count
– currently 1 replica set of 3 m2.4xlarge instances
70

79. MongoDB on AWS
71

80. Big Data Project
• Geo Spatial Features
– $within queries (bounding box)
– $near queries
72

81. Big Data Project
73

82. Big Data Project
Raw Data MapReduce
74

83. Big Data Project
• more polygons -> more data
– key length can become an issue
• using polygons to display cell metrics
• tried different types of visualizations
75

84. Big Data Project
• key-size per doc: 1.8KB
– bad: {very_descriptive_long_key : “yay”}
– good { v : “yay”}
76

85. Big Data Project
100000 polygons 500000 polygons
0 100.0 200.0 300.0 400.0
62
GB / year
308
77

86. Big Data Project
78

87. Big Data Project
• 308GB of EBS storage => 332$ per year
– backups / snapshot not considered
79

88. Big Data Project
• Future Plans
– new Use Case
– expecting about 1TB of data / week
80

89. Conclusion
• rapidly changing business needs
• ease of collecting huge amounts of data
• infrastructure as part of code
• MongoDB provides flexibility
81

90. Comments?
• @comsysto
• #MongoMunich2012
• http://blog.comsysto.com
• Don’t forget the hallway track
• Mongo User Group Munich
– http://www.meetup.com/Muenchen-MongoDB-
User-Group/
82

91. We are hiring!
• http://careers.comsysto.com
83

92. Lean & Agile with MongoDB
MongoMunich 2012
#MongoDBMunich
@comsysto