Slides from a lighting talk presented at the January Sydney DevOps Meetup.

1. LATENCY
THE SILENT
MONITORING SYSTEM
KILLER
Saturday, 21 January 2012

2. #MONITORINGSUCKS
Saturday, 21 January 2012

3. SCALABILITY
Saturday, 21 January 2012

4. execute large volumes of monitoring
checks under a variety of conditions
(good + bad) with a consistent throughput
Saturday, 21 January 2012

5. CONSISTENT
THROUGHPUT
Saturday, 21 January 2012

6. SHORT CHECK
EXECUTION TIME
Saturday, 21 January 2012

7. WHAT
INTRODUCES
VARIABILITY?
Saturday, 21 January 2012

8. LATENCY INTRODUCED
c SYNCHRONOUS CALLS
Saturday, 21 January 2012

9. Saturday, 21 January 2012

10. = 1 monitoring check
Saturday, 21 January 2012

11. Saturday, 21 January 2012

12. 150 monitoring checks
each executed every 300 seconds
each takes 1 second
checks are executed serially
Saturday, 21 January 2012

13. all checks executed in 150 seconds
monitoring system at 50% capacity
Saturday, 21 January 2012

14. DOUBLE THE CHECKS
Saturday, 21 January 2012

15. Saturday, 21 January 2012

16. all checks executed in 300 seconds
monitoring system at 100% capacity
Saturday, 21 January 2012

17. DOUBLE THE
EXECUTION TIME
Saturday, 21 January 2012

18. Saturday, 21 January 2012

19. all checks executed in 600 seconds
monitoring system at 200% capacity
only 50% of the checks are “on time”
Saturday, 21 January 2012

20. CHECK LATENCY
Saturday, 21 January 2012

21. Saturday, 21 January 2012

22. HOW DO
WE FIX THIS!?
Saturday, 21 January 2012

23. PERFORMANCE
ANALYSIS!
Saturday, 21 January 2012

24. AN ANALOGY
Saturday, 21 January 2012

25. MONITORING CHECK
==
“ACTION” ON MVC WEB APP
Saturday, 21 January 2012

26. Saturday, 21 January 2012

27. Saturday, 21 January 2012

28. SEPARATE DATA
COLLECTION
FROM
THRESHOLDING
f NOTIFICATIONS
Saturday, 21 January 2012

29. Saturday, 21 January 2012

30. THIS SHIFTS
LATENCY
Saturday, 21 January 2012

31. IT DOES NOT
ELIMINATE IT!
Saturday, 21 January 2012

32. Saturday, 21 January 2012

33. RRDTOOL IS EVIL
Saturday, 21 January 2012

34. USE SOMETHING
BETTER!
Saturday, 21 January 2012

35. USE
OPENTSDB
OR
GANGLIA
WITH
CHECK_TSDB
OR
CHECK_GMOND
Saturday, 21 January 2012

36. Saturday, 21 January 2012

37. A DIFFERENT SET a
PROBLEMS
Saturday, 21 January 2012

38. STORAGE WILL
GO AWAY
Saturday, 21 January 2012

39. CHAOS WILL ENSUE
Saturday, 21 January 2012

40. PAGERS WILL MELT
Saturday, 21 January 2012

41. SET UP “META-
PARENTING”
Saturday, 21 January 2012

42. BUILD A KILL
SWITCH
Saturday, 21 January 2012

43. Saturday, 21 January 2012

44. READ MORE
ABOUT THIS:
Saturday, 21 January 2012

45. bit.ly/yN4mdy
Saturday, 21 January 2012