Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2S6mtg5. Damon Edwards takes a look at the techniques that high-performing operations organizations are using to finally transform how they identify, mobilize, and respond to incidents. Filmed at qconsf.com. Damon Edwards is a Co-Founder of Rundeck Inc., the makers of Rundeck, the popular open source runbook automation. He has spent the past 19 years working with both the technology and business ends of IT Operations and is noted for being a leader in porting cutting-edge DevOps techniques to large-scale enterprise organizations.

1. Damon Edwards
Incident Management
in the Age of DevOps and SRE
November 12, 2019

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
incident-management-devops-sre/

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Damon Edwards
Incident Management
in the Age of DevOps and SRE
@damonedwards
November 12, 2019

6. Assertion:
The ability to respond to and resolve incidents is the true
indicator of an organization’s operational capabilities

8. Assertion 2:
Everybody now works in “Operations"

9. What Is an Incident?
An unplanned disruption impacting
customers or business operations

10. What Is an Incident?
An unplanned disruption impacting
customers or business operations
Outages
Service Degradation

11. What Is an Incident?
An unplanned disruption impacting
customers or business operations
Outages
Service Degradation
Work interruption
Delay/Waiting
“Short-Notice” Requests

15. Board

16. Integrated
Board

17. Integrated
Responsive
Board

18. Integrated
Responsive
Everywhere
Board

19. Integrated
Responsive
Everywhere
Always
Board

20. Integrated
Responsive
Everywhere
Always
Board
Tech Org Execution

21. Integrated
Responsive
Everywhere
Always
Board
Tech Org Execution

23. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad

24. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad

25. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad

26. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad
SAIL/cornell.edu

27. Adrian Cockcroft
Developer
Developer
Developer
Developer
Developer
Old Release Still
Running
Release Plan
Release Plan
Release Plan
Release Plan
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Bugs
Deploy
Feature to
Production
Immutable microservice deployment
scales, is faster with large teams and
diverse platform components
DockerCon EU 2014 Architecture enables speed.
Speed is the advantage.

30. The Three Ways (2013)

31. The Three Ways (2013) The Five Ideals (2019)

33. DEV

34. Go! Go! Go!DEV

35. Go! Go! Go!DEV …OPS?

36. 0000
Go! Go! Go!DEV …OPS?

37. 0000
Go! Go! Go!DEV …OPS?
Operations:
The Last Mile

40. 1. SRE needs Service Level Objectives, with consequences
2. SREs have time to make tomorrow better than today
3. SRE teams have the ability to regulate their workload
Principles of SRE

41. 1. SRE needs Service Level Objectives, with consequences
2. SREs have time to make tomorrow better than today
3. SRE teams have the ability to regulate their workload
Principles of SRE

42. 1. SRE needs Service Level Objectives, with consequences
2. SREs have time to make tomorrow better than today
3. SRE teams have the ability to regulate their workload
Principles of SRE

44. DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +

45. DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +

46. Dev Ops
Cross-Functional Team
Cross-Functional Team
DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +

47. Dev Ops
Cross-Functional Team
Cross-Functional Team
DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +
“Value-Aligned” and Self-Regulating
Shared
Responsibility
Model

49. Traditional ITSM

50. Traditional ITSM
ITIL
1989 - ?

51. Traditional ITSM
ITIL
1989 - ?

52. Traditional ITSM
Unintentionally Encourages Silos
ITIL
1989 - ?

53. Traditional ITSM
X X X XX X
Unintentionally Encourages Silos
ITIL
1989 - ?

54. Traditional ITSM
X X X XX X
Unintentionally Encourages Silos
Encourages command
& control management
ITIL
1989 - ?

55. Traditional ITSM
X X X XX X
Unintentionally Encourages Silos
Encourages command
& control management
ITIL
1989 - ?

56. Old Way
New Way

57. Old Way
New Way

59. +

73. REDeploy.io

74. There is no root cause.
(That’s just a political distinction)
REDeploy.io

75. Why?
Why?
Why?
Why?
Why?
There is no root cause.
(That’s just a political distinction)
REDeploy.io

76. Why?
Why?
Why?
Why?
Why?
There is no root cause.
(That’s just a political distinction)
Right,
Wrong,
Safety II,
and You.
REDeploy.io

77. Why?
Why?
Why?
Why?
Why?
There is no root cause.
(That’s just a political distinction)
Right,
Wrong,
Safety II,
and You.
Incidents = unplanned investments
REDeploy.io

79. You
Not

85. 18Million
IT Ops
22.3Million
Developers

88. Col. John Boyd
OODA Loop

91. Monitoring
Spotting the knowns

92. Monitoring
Spotting the knowns
Observability
Interrogating the unknowns

93. Observability
Interrogating the unknowns

94. Observability
Interrogating the unknowns
Logging: The event

95. Observability
Interrogating the unknowns
Logging: The event
Metrics: Data points over time

96. Observability
Interrogating the unknowns
Logging: The event
Metrics: Data points over time
Tracing: Events in context of a single request

97. Observability
Interrogating the unknowns
Logging: The event
Metrics: Data points over time
Tracing: Events in context of a single request

98. Automated Governance
Objective automated attestation of
GRC controls

99. Automated Governance
Objective automated attestation of
GRC controls

100. Automated Governance
Objective automated attestation of
GRC controls

101. Monitoring
Observability
Governance
Everyone
Everyone
Everyone
Everyone

102. Incident Command
Mobilization, Coordination, Communication

103. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

104. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

105. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

106. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

107. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)
GitHub: PagerDuty/incident-response-docs

108. Ops = Platform Eng + SRE
Divide and conquer

109. Ops = Platform Eng + SRE
Divide and conquer

110. Ops Platform Eng + SRE
Divide and conquer
SRE: Expert Operators (distributed)
Platform Eng: Build and Operate Platform Services (centralized)

111. Ops Platform Eng + SRE
Divide and conquer
SRE: Expert Operators (distributed)
Platform Eng: Build and Operate Platform Services (centralized)

112. Ops Platform Eng + SRE
Divide and conquer
SRE: Expert Operators (distributed)
Platform Eng: Build and Operate Platform Services (centralized)

113. New Views on Escalations
Avoid… but swarm if you do
Support at
the edge
Swarm

114. Diagnose: Health checks, exploratory actions
Take Action!
Restore: Restart, repair actions, rollback

115. The Return of Runbooks
Awhile ago Not that long ago Now

116. The Return of Runbooks
Awhile ago Not that long ago Now
Runbooks
(Mostly Manual)

117. The Return of Runbooks
Awhile ago Not that long ago Now
Runbooks
(Mostly Manual) …

118. The Return of Runbooks
Awhile ago Not that long ago Now
Runbooks
(Mostly Manual)
Runbooks
(Automate!…How?)…
Thanks SRE!

119. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

120. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

121. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

122. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.
Moving the bits is the easy part!

123. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

124. Empower those closest to the action!
Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

125. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

126. De-risk!
Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

127. Before Runbook Automation…

128. Before Runbook Automation…
3 options:

129. 1. Decipher the wiki
Before Runbook Automation…
3 options:

130. 1. Decipher the wiki
2.Ad-hoc tool/script usage
Before Runbook Automation…
3 options:

131. 1. Decipher the wiki
2.Ad-hoc tool/script usage
3.ESCALATE!
Before Runbook Automation…
3 options:

132. …with Runbook Automation

133. Shorter Incidents. Fewer Escalations.
Before RBA

134. Shorter Incidents. Fewer Escalations.
Before RBA

135. With RBA
Shorter Incidents. Fewer Escalations.

136. With RBA
Shorter Incidents. Fewer Escalations.

137. Before RBA
Shorter Incidents. Fewer Escalations.

138. With RBA
Shorter Incidents. Fewer Escalations.

139. Solve Difficult Security & Compliance Problems
Before RBA

140. Solve Difficult Security & Compliance Problems
With RBA

141. Everything Through a SDLC
Promote

142. Runbooks as a Service

147. Incidents = unplanned investments …the ROI is up to you.

148. Recap!
Elevate the Human.

149. @damonedwards
damon@rundeck.com
Let’s talk…
Special thanks to

150. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
incident-management-devops-sre/