1. Monitoring the
Clouds
Gary Dusbabek @gdusbabek

2. Outline
Motivation &
Features
Technology
Prognosticate

3. Motivation

4. Thousands
of servers

5. Complex
Event
pipeline

6. Existing Solutions New Expensive
but
old
!
Lessons from
Cloudkick

7. Our Goals

8. Internal
vs
external
Millions of signals
!

9. Geographic
distribution
&
Redundancy

10. Simultaneous!
Redundant Processing
& analysis

11. Fail!!
safely

12. Results
What we ended up with

13. Concurrent & redundant
event analysis

14. Robust!
metrics
collection

15. Remote collectors
in five datacenters
!
Agent collectors anywhere!
https://github.com/racker/virgo

16. Advancing state of the art

17. How we did it
Team!&!
Culture

18. How we did it
No idea!
is!!
too stupid

19. How we did it
Slippery!
iteration

20. How we did it
Reduce
Friction

21. How we did it
Well-greased
Deployments
https://github.com/etsy/deployinator
https://github.com/racker/dreadnot

22. How we did it
Well-greased
Deployments
30 Days
146 Deploys
5 Datacenters

23. Nuts!&!Bolts!

24. Nuts!&!Bolts!
Cassandra!
!
5x!replica5on!
3!Datacenter!

25. Nuts!&!Bolts!
Node.js!

26. Nuts!&!Bolts!
Reconnoiter!
+!
Custom!ingestor!

27. Nuts!&!Bolts!
Esper!
+!
Magic!sauce!

28. Nuts!&!Bolts!
Our!code!
!
With!help!from!Zookeeper!

29. Nuts!&!Bolts!
Virgo!
hGps://github.com/racker/virgo!
!
Lua!
Luvit!even5ng!
Node.js!backend!

30. Nuts!&!Bolts!
Metrics!
inges5on!&!
processing!
!
Java!
20M!signals/min!

31. Nuts!&!Bolts!
Mul5!DC!

32. Nuts!&!Bolts!
Message!
Redundancy!

33. Nuts!&!Bolts!
Agent!
crossRDC!

34. Prognostication

35. 2011!Gartner!
Big Data!!
“Emerging Technology”

36. Ingestion
Solved problem

37. Data!!
vs
Knowledge

38. Insight !
powerful
rare

39. Correlation
Prediction
Automation!
Aggregation

40. Bitrot
So much data
can never change
databases

41. It will
get
worse

42. Questions?
@gdusbabek

43. Image!Credits!
!
All!images!for!this!presenta5on!come!from!the!Flickr!commons!collec5on!
hGp://www.flickr.com/commons/
Clouds ! ! !hGp://www.flickr.com/photos/sdasmarchives/7316444746!
Biplane ! ! !hGp://www.flickr.com/photos/sdasmarchives/7316444552!
Umbrellas ! ! !hGp://www.flickr.com/photos/statelibraryofnsw/4944459226!
servers ! ! !hGp://www.flickr.com/photos/na5onaalarchief/4026288398!
complex ! ! !hGp://www.flickr.com/photos/lselibrary/3990093924!
pyramid ! ! !hGp://www.flickr.com/photos/brooklyn_museum/2488800747!
kites ! ! ! !hGp://www.flickr.com/photos/usna5onalarchives/5506532956!
target! ! ! !hGp://www.flickr.com/photos/nlireland/5963130143!
map ! ! ! !hGp://www.flickr.com/photos/sdasmarchives/5018415335!
safety! ! ! !hGp://www.flickr.com/photos/usna5onalarchives/3660777810!
reports ! ! !hGp://www.flickr.com/photos/usna5onalarchives/6990370962!
apparatus ! ! !hGp://www.flickr.com/photos/library_of_congress/2179184290!
robust ! ! !hGp://www.flickr.com/photos/twm_news/5863492976!
old!computer ! !hGp://www.flickr.com/photos/nasacommons/7605906886!
dogs ! ! ! !hGp://www.flickr.com/photos/smithsonian/4010655153!
team ! ! ! !hGp://www.flickr.com/photos/floridamemory/6730955597!
oil ! ! ! !hGp://www.flickr.com/photos/usna5onalarchives/7065894867!
classroom ! ! !hGp://www.flickr.com/photos/library_of_congress/2179067948!
horses ! ! !hGp://www.flickr.com/photos/powerhouse_museum/2581833315!
skeleton ! ! !hGp://www.flickr.com/photos/powerhouse_museum/2980051095!
washing!machine ! !hGp://www.flickr.com/photos/na5onaalarchief/4397795513!
genie ! ! ! !hGp://www.flickr.com/photos/library_of_congress/7267723488!
waste! ! ! !hGp://www.flickr.com/photos/nlscotland/4699984659!
truck!waste ! ! !hGp://www.flickr.com/photos/usna5onalarchives/3928424307!
ques5ons ! ! !hGp://www.flickr.com/photos/nlscotland/4698823475!