Talk from Icinga Camp San Diego in October 2016

1. The Unrealized
Role of:
Monitoring
& Alerting
@jasonhand | VictorOps | #DevOpsDays

2. Jason
HandDevOps Evangelist
VictorOps
@jasonhand | VictorOps | #DevOpsDays

3. SCaLE 14x
Southern
California
Linux
Expo
@jasonhand | VictorOps | #DevOpsDays

4. @jasonhand | VictorOps | #DevOpsDays

5. 2015
Monitoring
Survey@jasonhand | VictorOps | #DevOpsDays

6. @jasonhand | VictorOps | #DevOpsDays

7. WhyAreYou CollectingThis Data?
NOTE: You may choose more than one
» Performance analysis and trending
» Fault and Anomaly detection
» Capacity Planning
» A/B Testing
» We don’t do anything with collected metrics
@jasonhand | VictorOps | #DevOpsDays

8. The Results
NOTE: Respondents may have chose more than one
» Performance analysis and trending - 63%
» Fault and Anomaly detection - 53%
» Capacity Planning - 45%
» A/B Testing - 11%
» We don’t do anything with collected metrics - 3%
@jasonhand | VictorOps | #DevOpsDays

9. Tyranny of the
S.L.A.(Service Level Agreement)
@jasonhand | VictorOps | #DevOpsDays

10. HighAvailabilityPrediction & Prevention
@jasonhand | VictorOps | #DevOpsDays

11. @jasonhand | VictorOps | #DevOpsDays

12. @jasonhand | VictorOps | #DevOpsDays

13. That's Important
... but...@jasonhand | VictorOps | #DevOpsDays

14. @jasonhand | VictorOps | #DevOpsDays

15. @jasonhand | VictorOps | #DevOpsDays

16. Business
Objectives?
@jasonhand | VictorOps | #DevOpsDays

17. HappyCamper
@jasonhand | VictorOps | #DevOpsDays

18. Customerswant more than just
99.999% Uptime
@jasonhand | VictorOps | #DevOpsDays

19. @jasonhand | VictorOps | #DevOpsDays

20. Where'sthe
Innovation?
@jasonhand | VictorOps | #DevOpsDays

21. ? =@jasonhand | VictorOps | #DevOpsDays

22. ? =Continuous
Improvement
@jasonhand | VictorOps | #DevOpsDays

23. HowImportantis
Learning &
Innovation?
@jasonhand | VictorOps | #DevOpsDays

24. @jasonhand | VictorOps | #DevOpsDays

25. @jasonhand | VictorOps | #DevOpsDays

26. @jasonhand | VictorOps | #DevOpsDays

27. @jasonhand | VictorOps | #DevOpsDays

28. The result of underutilizing monitoring & alerting
is that the IT department and the organization have
no chance to...
learn,
improve, or
innovate.@jasonhand | VictorOps | #DevOpsDays

29. Continuallyunderstanding &
respondingtothe feedback
from
monitoring, logging, & alerting
allows you to use information about events in the
past to drive future actions.
@jasonhand | VictorOps | #DevOpsDays

30. SwitchingGears
@jasonhand | VictorOps | #DevOpsDays

31. @jasonhand | VictorOps | #DevOpsDays

32. @jasonhand | VictorOps | #DevOpsDays

33. It's not just about
Prediction
& Prevention
@jasonhand | VictorOps | #DevOpsDays

34. Respond &
Repair
...Quickly
@jasonhand | VictorOps | #DevOpsDays

35. Nope
@jasonhand | VictorOps | #DevOpsDays

36. MTTRRather Than
MTBF@jasonhand | VictorOps | #DevOpsDays

37. Failure Is
Inevitable
@jasonhand | VictorOps | #DevOpsDays

38. us·er/ˈyoozər/
Distributed faultinjectiontestsuite for
production.
credit: Leon Fayer (@papa_fire)
@jasonhand | VictorOps | #DevOpsDays

39. Successis a result of
Failure@jasonhand | VictorOps | #DevOpsDays

40. Understand
Learn
Innovate
@jasonhand | VictorOps | #DevOpsDays

41. re·sil·ient/rəˈzilyənt/
The ability to resist, absorb, recover from or
successfully adapt to adversity or a change in
conditions
@jasonhand | VictorOps | #DevOpsDays

42. Change
can cause failure
but innovation requires
Change
@jasonhand | VictorOps | #DevOpsDays

43. Conflict
@jasonhand | VictorOps | #DevOpsDays

44. ChangeRequired
@jasonhand | VictorOps | #DevOpsDays

45. “Withoutdeviation
fromthe norm,
progress is not
possible ”
Frank Zappa
@jasonhand | VictorOps | #DevOpsDays

46. What Did You
LearnFrom the Recovery Efforts?
(including monitoring & alerting)
@jasonhand | VictorOps | #DevOpsDays

47. Postmortems / Learning Reviews:
Stories of:
Whattook placeleading up to & during
the disruption & recovery efforts
@jasonhand | VictorOps | #DevOpsDays

48. Whowas
involved?
@jasonhand | VictorOps | #DevOpsDays

49. Whatdidthey
see?@jasonhand | VictorOps | #DevOpsDays

50. Whatwas
said?@jasonhand | VictorOps | #DevOpsDays

51. What
actionsweretaken?
jhand.co/chatopsbook
@jasonhand | VictorOps | #DevOpsDays

52. Howdo events &
actions
correlateovertime?
@jasonhand | VictorOps | #DevOpsDays

53. 5Why's
@jasonhand | VictorOps | #DevOpsDays

54. Whatisthe "cause"
ofthe Problem?
Root Cause is ...
@jasonhand | VictorOps | #DevOpsDays

55. Our...
obsession with
"Root
Cause"
@jasonhand | VictorOps | #DevOpsDays

56. Asking "why"
.. leads to ..
Blame
@jasonhand | VictorOps | #DevOpsDays

57. Blamingleadsto..
operators hiding relevant &
important information
@jasonhand | VictorOps | #DevOpsDays

58. We must
believethat our operators are doing their best given the
constraints of the "system"
@jasonhand | VictorOps | #DevOpsDays

59. "We are here to"
LearnFrom Failure
(and success)
@jasonhand | VictorOps | #DevOpsDays

60. Ratherthan ..
@jasonhand | VictorOps | #DevOpsDays

61. AvoidFailure
@jasonhand | VictorOps | #DevOpsDays

62. What'sthe
Story?
@jasonhand | VictorOps | #DevOpsDays

63. InnovateLearning from both success & failure
to develop & implement
small incremental improvements
is critical.
@jasonhand | VictorOps | #DevOpsDays

64. Learning
Organization
@jasonhand | VictorOps | #DevOpsDays

65. Learning does NOT come from
Reading
&
Listening@jasonhand | VictorOps | #DevOpsDays

66. Learning comes from
Doing@jasonhand | VictorOps | #DevOpsDays

67. Real Learning comes from:
Observing
Orienting
Deciding
Acting
John Boyd's OODA Loop
@jasonhand | VictorOps | #DevOpsDays

68. Example:
Learningto playthe
Dobro Guitar
@jasonhand | VictorOps | #DevOpsDays

69. @jasonhand | VictorOps | #DevOpsDays

70. Learning
@jasonhand | VictorOps | #DevOpsDays

71. Why?
Go from knowing...
to understanding...
to learning
NOTE:
(Requires making mistakes)
@jasonhand | VictorOps | #DevOpsDays

72. @jasonhand | VictorOps | #DevOpsDays

73. “We will trade some uptime in exchange for innovation
-Dave Hahn (Netflix)”
DevOpsDays Boise 2016
(today)
@jasonhand | VictorOps | #DevOpsDays

74. Are We Doing
itRight?@jasonhand | VictorOps | #DevOpsDays

75. What do your
Postmortemslook like?
Are they setting you up to learn?
@jasonhand | VictorOps | #DevOpsDays

76. "The Story"
-Timeline
-WhoWas Involved
-Context
(Seeing, Saying, Executing)
-Action Items
(Small Incremental
Improvements)
@jasonhand | VictorOps | #DevOpsDays

77. Shiftour gaze
from:
maintaining
& protecting
@jasonhand | VictorOps | #DevOpsDays

78. LearningWhich leads to...
Improving
& Innovating
@jasonhand | VictorOps | #DevOpsDays

79. we increasevalue
of monitoring & alerting
of the IT teams
of Products & Services
& of the Organization.
@jasonhand | VictorOps | #DevOpsDays

80. Hypothesize
Explore
Stretch
Experiment
Fail
Learn
Try Again
@jasonhand | VictorOps | #DevOpsDays

81. @jasonhand | VictorOps | #DevOpsDays

82. Learning & Innovating
leads to uncovering new ways of
building, deploying,and
maintaining software &
infrastructure
Which leads to...
@jasonhand | VictorOps | #DevOpsDays

83. ResilientSystems
@jasonhand | VictorOps | #DevOpsDays

84. The
By-product
of a highly
resilient
system is ...
@jasonhand | VictorOps | #DevOpsDays

85. @jasonhand | VictorOps | #DevOpsDays

86. HighlyAvailable
system@jasonhand | VictorOps | #DevOpsDays

87. The Unrealized
Role of:
Monitoring
& Alerting is ....
@jasonhand | VictorOps | #DevOpsDays

88. Learning&
Innovation
@jasonhand | VictorOps | #DevOpsDays

89. Thank
YouBe Victorious!
@jasonhand | VictorOps | #DevOpsDays

90. References:
Monitoring Survey: https://kartar.net/2015/08/
monitoring-survey-2015---metrics/
Firefighter: https://www.learyfirefighters.org/wp-
content/uploads/2013/09/cover-slide-1.jpg
Mechanic: https://upload.wikimedia.org/wikipedia/
commons/4/4b/Flickr_-_Israel_Defense_Forces_-
_Airplane_Technician,_March_2010.jpg
Gnome Plan: http://www.nerdfitness.com/wp-content/
uploads/2012/04/Screen-Shot-2012-03-30-at-3.15.38-
AM-1024x7591.jpg
NOC: https://upload.wikimedia.org/wikipedia/commons/
@jasonhand | VictorOps | #DevOpsDays

91. References:
Kodak: http://file.answcdn.com/answ-cld/image/upload/
v1/tk/brand_image/b59911fc/
91d6e71d30a0878dfe3cb30a22751cb874a3ea8c.jpeg
VW Camper: https://upload.wikimedia.org/wikipedia/
commons/d/d7/VW_Camper.jpg
Blockbuster: https://
jordanandeddie.files.wordpress.com/2013/11/
blockbuster-feature.jpg
Borders: http://smashingtops.com/wp-content/uploads/
2012/06/borders_logo1.jpg
@jasonhand | VictorOps | #DevOpsDays

92. References:
Chained Hands: https://www.google.com/url?
sa=i&rct=j&q=&esrc=s&source=images&cd=&ved=0ahUKEwjgr
NCDh5TMAhXJs4MKHaoZDssQjBwIBA&url=http%3A%2F
%2Fwww.publicdomainpictures.net%2Fdownload-
picture.php%3Fadresar%3D50000%26soubor%3Dhands-in-
chains.jpg%26id%3D40426&bvm=bv.
119745492,d.amc&psig=AFQjCNFIdnDPzSqiLA-
znIW5SCTCUHhqEw&ust=1460926880336203
Inevitable: http://vignette4.wikia.nocookie.net/
matrix/images/5/51/SMITH.png/revision/latest?
cb=20110214092002
@jasonhand | VictorOps | #DevOpsDays

93. References:
Accident Free:http://www.compliancesigns.com/media/
digital-scoreboard/1000/Safety-Awareness-Sign-
DSE-195271000.gif
Stewie:
http://chroniclesofredmark.com/wp-content/uploads/
2014/01/Stewie.gif
change: http://i.imgur.com/EQyC6N3.gif
Hard drive: https://i.imgur.com/pWsKSEf.gif
Change: https://farm6.staticflickr.com/
5208/5270199049df99b234e9od.jpg
Value: https://d13yacurqjgara.cloudfront.net/users/
@jasonhand | VictorOps | #DevOpsDays

94. @jasonhand | VictorOps | #DevOpsDays