The document discusses how Infochimps refactored their Cube analytics platform to scale from terabytes to petabytes of data. They measured performance to identify bottlenecks, harmonized the system design to better utilize resources like separating events and metrics into different databases, and tuned configurations like using capped collections. This allowed them to scale to billions of events per day while keeping metrics calculations fast. They also discussed techniques for handling increased volume like using queues, caching, and cloud infrastructure.

1. Harmony in Tune
How we Refactored Cube
to Terabyte Scale
Philip (flip) Kromer
Huston Hoburg
infochimps.com
Feb 15 2013

2. Big Data for All

3. Big Data for All

4. why dashboards?

5. Lightweight Dashboards
• Understand what’s happening
• Understand data in context
• NOT exploratory analytics
• real-time insight...
but not just about real-time
mainline: j.mp/sqcube
hi-scale branch: j.mp/icscube

6. The “Church of Graphs”

7. Predictive Kvetching

8. Lightweight Dashboards

9. Approach to Tuning
• Measure: “Why can’t it be faster?”
• Harmonize: “Use it right”
• Tune: “Align it to production resources”

10. cube is awesome

11. What’s so great?
• Streaming, real-time
• Ad-hoc data: write whatever you want
• Ad-hoc queries: make up new queries whenever
• Efficient (“pyramidal”) calculations

12. Event Stream
•{ time: "2013-02-15T01:02:03Z",
type: "webreq", data: {
path: "/order", method: "POST",
duration: 50.7, status: 400,
ua:"...MSIE 6.0..." } }
•{ time: "2013-02-15T01:02:03Z",
type: "tweet", id: 8675309, data: {
text: "MongoDB talk yay",
retweet_count: 121,
user: { screen_name: "infochimps",
followers_count: 7851,
lang: "en", ...} } }

13. Events vs Metrics
Event:
•{ time: "2013-02-15T01:02:03Z",
type: "tweet", id: 8675309, data: {
text: "MongoDB talk yay",
retweet_count: 121,
user: { screen_name: "infochimps",
followers_count: 7851,
lang: "en", ...} } }
Metrics:
• “# of tweets in 10s bucket at 1:02:10 on 2013-02-15”
• “# of non-english-language tweets in 1hr bucket at ...”

14. Events vs Metrics
Event:
•{ time: "2013-02-15T01:02:03Z",
type: "webreq", data: {
path: "/order", method: "POST",
duration: 50.7, status: 400,
ua:"...MSIE 6.0..." } }
Metrics:
• “# of requests in 10s bucket at 3:05:10 on 2013-02-15”
• “Average duration of requests with 4xx status in the 5
minute bucket at 3:05:00 on 2013-02-15”

15. Events vs Metrics
• Events: { time: "2013-02-15T01:02:03Z",
type: "webreq",
• baskets of facts
data: {
path: "/order",
method: "POST",
• narcissistic duration: 50.7,
status: 400,
ua:"...MSIE 6.0..." } }
• LOTS AND LOTS

16. Events vs Metrics
• Events: { time: "2013-02-15T01:02:03Z",
type: "webreq",
• baskets of facts
data: {
path: "/order",
method: "POST",
• narcissistic duration: 50.7,
status: 400,
ua:"...MSIE 6.0..." } }
• LOTS AND LOTS
• Metrics:
• a timestamped number
• look like the graph{ time: "2013-02-15T01:02:03Z",
value: 90 }
• one per time bucket

17. billions and billions

18. 3000 events/second

19. tuning methodology

20. Monkey See Monkey Do
Google for
the #s the
cool kids use

21. Spinal Tap
Turn
everything
to 11!!!!

22. Hillbilly Mechanic
Rewrite for
memcached
HBase on
Cassandra!!!

23. Moneybags
SSD plz
Moar CPU
Moar RAM
Moar Replica

24. Tuning How to do it
• Measure: “Why can’t it be faster?”
• Harmonize: “Use it right”
• Tune: “Align it to production resources”

25. see through
the magic

26. • Why can’t it be faster than it is now?

27. • dstat (http://j.mp/dstatftw): dstat -drnycmf -t 5
• htop
• mongostat

28. Grok: client-side
• Made a sprayer to inject data
• invalidate a time range at max speed
• writes variously-shaped data: noise, ramp, sine, etc
• Or just reach into the DB and poke
• delete range of metrics, leave events
• delete range of events, leave metrics

29. Fault injection
• raise when packet comes in with certain flag
•{ time: "2013...", data: {...},
_raise:"db_write" }
• (only in development mode, obvs.)

30. app-side tracing
metalog.event('connect',
{ method: 'ws',
ip: connection.remoteAddress,
path: request.url }, 'minor');
• “Metalog” announces lifecycle progress:
• writes to log...
• ... or as cube metrics!

31. app-side tracing

32. fits on machine

33. 3000 events/second
• Rate:
• 3000 ev/sec ≈ 250 M ev/day ≈ 2 BILLION/wk
• Expensive. Difficult.
• 250 GB accumulated per day (@1000 bytes/ev)
• 95 TB accumulated per year (@1000 bytes/ev)

34. Metrics
• Rate:
• 3M tensec/year (π· 10 sec/year)
7
• < 100 bytes/metric ...
• Manageable!
• a 30 metric dashboard is ~ 10 GB/year @10sec
• a 30 metric dashboard is ~ 170 MB/year @ 5min

35. 20% gains are boring
At scale, your first barriers are either:
• Easy
• Impossible
Metrics: 10 GB/year
Events: 10 TB/month

36. Scalability sí
Performance no

37. Still CPU and Memory Use
• Problem
• Mongo seems to be working
• but high resident memory and fault rate
• Memory-mapped Files
• 1Tb data served by 4Gb ram is no good

38. Capped Collections
• Fixed size circular queue
• records are in order of insertion
A B C D A E F
• oldest records are discarded when full
...G H C D A E F G ...

39. Capped Collections
• Extremely efficient on write
A B C D A E F
• Extremely efficient for insertion-order reads
• Very efficient if queries are ‘local’
• events in same timebucket
typically arrived at nearby times
and so are nearby on disk

40. don’t like the answer?
change the question.

41. mainline
uncapped events
capped metrics:
metrics are a view on data

42. hi-scale branch
capped events
uncapped metrics:
events are ephemeral

43. Harmony
• Make your pattern of access
match your system’s strengths and rhythm

44. Validate Mental Model

45. Easy fixes
• Duplicate requests = duplicate calculations
• Cube patch for request queues exists
• Easy fix!
• Non-pyramidal are inefficient
• Remove until things are under control
• ( solve paralyzing problems first )

46. cube 101

47. Cube Systems

48. Collector
• Receives events
• writes to MongoDB
• marks metrics for re-calculation (“invalidates”)

49. Evaluator
• receives, parses requests for metrics
• calculates metrics “pyramidally”
• then stores them, cached

50. Pyramidal Aggregation
90
5min
10 20 15 25 10 10
1min
1 5 2 0 2 0 6 4 7 1 0 2 2 3 2 4 2 2 5 5 4 6 4 1 2 7 0 0 0 1 6 0 0 1 0 3
10s
ev ev ev ev ev ev ...

51. Pyramidal Aggregation
5min
1min
1 5 2 0 2 0 6 4 7 1 0 2 2 3 2 4 2 2
10s
ev ev ev ev ev ev ...

52. Uses Cached Results
5min
10 20 15 25 10
1min
1 5 2 0 2 0 6 4 7 1 0 2 2 3 2 4 2 2 5 5 4 6 4 1 2 7 0 0 0 1
10s
ev ev ev ev ev ev ...

53. Pyramidal Aggregation
• calculates metrics...
• from metrics and constants ... from metrics ...
• from events
• (then stores them, cached)
5 min
1 min
10 sec
ev ev ev ev ev....

54. fast writes

55. how fast can we write?

56. how fast can we write?
FAST
streaming writes: way efficient

57. locked out

58. Writes and Invalidations

59. Inserts Stop Every 5s
• working
• working
• ANGRY
• ANGRY
• working
• working

60. Thanks, mongostat!
• working
• working
• ANGRY
...
• ANGRY
• working
• working
(simulated)

61. Inserts Stop Every 5s
Events Collection
...G H C D A E F G ...
hi-speed writes localized reads
Metrics Collection
. . . . . . . . . . . . . . . . . .
.
.
. x x xx x x
. . . . . .
. . . . . .
x .
randomish hi-speed
reads deletes
updates

62. Inserts Stop Every 5s
Events Collection
...G H C D A E F G ...
hi-speed writes localized reads
Metrics Collection
. . . . . . . . . . . . . . . . . .
.
.
. x x xx x x
. . . . . .
. . . . . .
x .
randomish hi-speed
reads deletes
updates

63. Inserts Stop Every 5s
• What’s really going on?
• Database write locks
• Events and metrics have conflicting locks
• Solution: split the databases
Events Collection
...G H C D A E F G ...
hi-speed writes localized reads
Metrics Collection
. . . . . . . . . . . . . . . . . .
.
.
. x x xx x x
. . . . . .
. . . . . .
x .
randomish hi-speed
reads deletes

64. fast reads

65. Pre-cache Metrics
• Keep metrics fresh (Warmer)
• Only calculate recent updates (Horizons)

66. fancy metrics

67. Non-pyramidal Aggregates
• Can’t calculate from warmed metrics
• Store values with counts in metrics
• Counts can be vivified for aggregations
• Smaller footprint than full events
• Works best for dense, finite values

68. finally, scaling

69. Multicore
• MongoDB
• Writes limited to single core
• Requires sharding for multicore

70. Multicore
• Cube (node.js)
• Concurrent, but not multi-threaded
• Easy solution
• Multiple collectors on different ports
• Produces redundant invalidations
• Requires external load balancing

71. Multicore

72. Hardware
• High Memory
• Capped events size scale with memory
• CPU
• Mongo / cube not optimized for multicore
• Faster cores
• EC2 Best value: m2.2xlarge
• < $700/mo, 34.2GB RAM, 13 bogo-hertz

73. Cloud helps
• Tune machines to application
• Dedicating databases for each application makes life
a lot easier

74. Cloud helps
• Tune machines to application
•

75. jobs@infochimps.com
github.com/
infochimps-labs

76. good ideas that
didn’t help

77. Queues
• Different queueing methods
• Should optimize metric calculations
• No significant improvement

78. Locks: update VS remove
• Uncapped metrics allow ‘remove’ as
invalidation option
• Remove doesn’t help with database locks
• It was a stupid idea anyway: that’s OK
• “Hey, poke it and see what happens!”

79. Mongo Aggregations
• Mongo has aggregations!
• Node ends up working better
• Mongo aggregations aren’t faster
• Less flexible
• Would require query language rewrite

80. Why not Graphite?
• Data model
• Metrics-centric vs Events-centric
(metrics code not intertwingled with app code)
• Environment familiarity
• Cube: d3, node.js, mongo
• Graphite: Django, Whisper, C