Move past the jargon. See how DevOps plays into incident management and resolution. Join guest Forrester Analysts and experts from local Colorado companies for a ½ day event focused on the latest and greatest DevOps practices for those tasked with maintaining uptime.

1. DENVER - SEATTLE - SAN FRANCISCO - MINNEAPOLIS - NEW YORK CITY

2. #DevOpsRoadTrip

3. #DevOpsRoadTrip
JASON HAND |
DevOps Evangelist
• 18 years of experience as a developer,
system administrator, and support specialist
• Fully emerged into the world of agile
development and the DevOps movement
• DevOps Evangelist (VictorOps)
• Author of “ChatOps For Dummies”
• Organizer – DevOpsDays Rockies
• Contributor to Wired, TechBeacon, InfoQ,
and many other technology publications
• Co-host of “Community Pulse” – podcast
on building Community in Tech
@jasonhand

4. Agenda
12:00 - 1:00 - Registration & Lunch
1:00 - Opening Remarks | Jason Hand,
DevOps Evangelist, VictorOps
1:15 - Expert Panel + Q&A
From Start to Finish Continuous
Improvement: The DevOps Journey at
Craftsy
Matthew Boeckman, VP of Infrastructure at
Craftsy
The Why and How of DevOps at SendGrid
–
Mary Moore-Simmons, Engineering and
Operations Manager at Sendgrid
2:15 - BREAK
#DevOpsRoadTrip
2:30 - Topical Breakout Sessions
3:30 - BREAK
3:40 - Avoiding the DevOps Pitfalls
Kurt Bittner, Principal Analyst - Forrester
Research
Amy Demartine, Sr. Analyst - Forrester Research
4:25 - Closing Remarks
4:45 - Happy Hour & Stadium Tours

5. DevOps History
101

37. #DevOpsRoadTrip
MATTHEW BOECKMAN |
Vice President , Infrastructure
• Serial entrepreneur with 18 years in
technical and operations management
• Works in enterprise systems
architecture, process optimization,
and operational intelligence

38. From Start to Finish Continuous
Improvement:
The DevOps Journey at Craftsy
Matthew Boeckman
VP - Infrastructure
Craftsy
@matthewboeckman

39. #30 on Forbes' 2015 list of Most Promising Companies
10+MM registered members, 11+MM enrolled courses
350 course enrollments/hour

40. Wow DevOps!

42. Culture
ToolsOncall

44. Reward Failure
Celebrate Success
Blameless Postmortems
Personal Responsibility
Avoid Process

45. Configuration
Management
CI/CD
Git
Clouds

48. Dev shares oncall
Actionable alerts
Runbooks

49. Runbooks:
● System overview
● Escalation path
● Metrics
● Alert descriptions
● Common failure conditions
● Known recovery procedures
● Incident history

50. Incident
Post-Mortem
Tools
Runbook
Reward

51. Thank you!
@matthewboeckman

52. #DevOpsRoadTrip
MARY MOORE-SIMMONS
Engineering Operations Manager
• Engineering Operations Manager
• Began career in chemical
engineering before switching to
the world of DevOps
• Focuses on meaningful impact
on company success in the fast-
paced world of software
engineering

53. The Why and How of DevOps at SendGrid
March 2016

54. Why DevOps?

55. Scale, yo! It’s hard!
2010
2.1K Customers
4.8B Emails
2011
7.3K Customers
20.6B Emails
2012
14.3K Customers
54B Emails
2013
22.6K Customers
99.8B Emails
2014
28.5K Customers
164B Emails
Today
34K Customers
>200B Emails in 2015
Now: 1B Emails/day

56. Champions of Quality

57. But I Want to be Friends!

58. Once Upon a Time…. It Was Terrible

59. Now Everything is Unicorns and
Rainbows
…..Kindof

61. Constant Vigilance! (Analysis)

62. Dedicate People to the Problem

64. Set Goals, Get Leadership On Board

65. Q A&
Questions?
THANKS FOR PUTTING UP
WITH SILLY JPGS

66. Q&A

67. #DevOpsRoadTrip
BREAK TIME

68. DENVER - SEATTLE - SAN FRANCISCO - MINNEAPOLIS - NEW YORK CITY

69. Breakout Sessions
 Devs On-Call, How and Why to Get Started
Kurt Bittner, Principal Analyst - Forrester
Research
 Doing DevOps Safely (governance & security
mindfulness)
Amy Demartine, Sr. Analyst - Forrester
Research
 Learning from Failure
Jason Hand, DevOps Evangelist, VictorOps
 Measure Everything: What to & What Not to
Alert On
Matthew Boeckman, Craftsy
 What does your “DevOps Ecosystem” look
like? #DevOpsRoadTrip

70. #DevOpsRoadTrip
BREAK TIME

71. #DevOpsRoadTrip
KURT BITTNER| Principal Analyst
• Principal Analyst Serving
Application Development and
Delivery
• Focuses on extracting broader
business value from organizational
and cultural shifts
• Former CTO at Ivar Jacobson
International

72. #DevOpsRoadTrip
AMY DEMARTINE| Senior Analyst
• Senior Analyst Serving
Infrastructure & Operations
Professionals
• Currently researching strategy,
design, organization, and
implementation of modern service
delivery
• Formerly worked with BMC and HP
on bringing new enterprise
software products to worldwide
markets

73. Avoid the DevOps Pitfalls
Amy DeMartine, Senior Analyst
Kurt Bittner, Principal Analyst

74. © 2015 Forrester Research, Inc. Reproduction Prohibited 74
Common DevOps Pitfalls To Avoid
1. Going too far too fast
2. Lack of learning culture
3. Measuring people with old system metrics
4. Not being able to reduce mean time to know
5. Keeping change management the same
6. Making DevOps all about Dev and Ops
7. Letting complexity continue

75. Pitfall 1: Going too far too fast

76. © 2015 Forrester Research, Inc. Reproduction Prohibited 76
Anti-Patterns
› Impatience; wanting to “skip to the solution”
› Wanting different results but not really wanting to
change
› Not taking time to understand barriers
› Thinking of “DevOps” as a methodology

77. © 2015 Forrester Research, Inc. Reproduction Prohibited 77
Risks of “Big Batch” Change
› A lot of energy, time and money are
invested before any result is achieved
› Forced change meets resistance
› Coordination complexity
› Progress stops when external energy is
no longer being supplied
› Inevitable disappointment when
progress is less than expected
› “Blame the approach”, or even “Blame
the person who got us into this”
Benefits of “Small Batch” change
› Quick results with small investment
› “Opt-in” change lets people choose
when and how to change
› Low complexity as long as
dependencies are minimized
› Scale-up when ready
› Not everyone needs to change, at least
at first
› Lead by example
Big batches of work don’t work for application delivery;
Why would they work for Agile transformation?

78. © 2015 Forrester Research, Inc. Reproduction Prohibited 78
http://alivecampus.com/wp-content/uploads/2014/08/shortcut.jpg

79. © 2015 Forrester Research, Inc. Reproduction Prohibited 79
How To Scale DevOps:
http://www.netsolutionsindia.com/blog/wp-content/uploads/2013/08/Core-Agile-Team.jpg
Expand Team By Team…
…

80. © 2015 Forrester Research, Inc. Reproduction Prohibited 80
While Building A Supporting Ecosystem
› Common pipeline, process, and tools
› Common platforms and services
› Communities of practice
› Servant-leadership
› A customer-obsessed culture

81. Pitfall 2: Lack of a learning culture

82. Failure not seen as opportunity to learn
Source: “Six Trends Shape DevOps Adoption, Q1 2015”, Forrester report

83. Ops is not getting included in life cycle
Source: “Six Trends Shape DevOps Adoption, Q1 2015”, Forrester report

84. Pitfall 3: Measuring people with old
system metrics

85. © 2015 Forrester Research, Inc. Reproduction Prohibited 85
38% of I&O DevOps respondents didn’t know if
customer experience was measured at their
company.
Source: “Six Trends Shape DevOps Adoption, Q1 2015”, Forrester report

86. © 2015 Forrester Research, Inc. Reproduction Prohibited 86
Too many I&O pros aren’t measured on
customer experience goals.

87. Pitfall 4: Not being able to reduce mean
time to know

88. © 2015 Forrester Research, Inc. Reproduction Prohibited 88
Single Source Of Truth Lacking In Many
Orgs – 95% only most of the time or less
Source: April 15, 2015 “Six Trends That Will Shape DevOps Adoption”, Forrester report

89. © 2015 Forrester Research, Inc. Reproduction Prohibited 89
…Leads To Customers Finding the
Problem First
Source: “Six Trends Shape DevOps Adoption, Q1 2015”, Forrester report

90. © 2015 Forrester Research, Inc. Reproduction Prohibited 90
Where do we waste time, energy and
resources?
MTTI MTTK MTTF MTTV
MTTR
Mean Time To Repair
Mean time to identify
Mean time to know
The War Room
Mean time to fix
Mean time to validate

91. Why a war in the war room?
• Wrong tools for the tasks
• Obsolete or inadequate monitoring (infrastructure monitoring rather than
performance management)
• Too many specialized tools
• Too much data, and the wrong types
• Silo mentality
• Exoneration rather than cooperation
• No real process
• Wrong skills applied to the wrong level

92. Pitfall 5: Keeping change management
the same

93. © 2015 Forrester Research, Inc. Reproduction Prohibited 93
Changes Require Too Much Touch by
72%
Source: “Six Trends Shape DevOps Adoption, Q1 2015”, Forrester report

94. Pitfall 6: Making DevOps all about Dev
and Ops

95. © 2015 Forrester Research, Inc. Reproduction Prohibited 95
1 out of every 16 open
source component
download request is for
a component with a
known vulnerability
97% of the successfully exploited
vulnerabilities in 2014 trace back to 10
common vulnerabilities and exposures, eight
of which have been patched for 10 to 12
years
median number of days that
threatening groups were
present on a victim's network
before detection was 205 in
2014, and only 31% of victims
discovered the breach
themselves
47% of web applications have
unprotected file issues due to web
server misconfigurations

96. © 2015 Forrester Research, Inc. Reproduction Prohibited 96
The Delivery-Centric Organization
Integrated
Product
Teams
Communities of Practice
Servant-
Leadership
LOB
CMO
CIO …

97. © 2015 Forrester Research, Inc. Reproduction Prohibited 97
Operational Platform Services
Customers
Software Platform
Services Teams
Shared Service
Teams
Integrated __
Product Teams___
Business Offering Teams
Project
Teams
S
o
u
r
c
i
n
g
• One or more products
• Integrated customer experience
• Integrated business processes
• Single release
• Shared resources
• Finite scope
• Ongoing releases
• Dedicated resources
• Aligned with business offering
funding
• Ideally deliver via automated self-service
• Resources shared across products and
projects
• Ideally staffed or automated for “zero wait-
time”
• Common components and
services
• Common architectural
patterns
• Computing as-a-utility
The Modular Customer-Focused Application
Delivery Organization

98. © 2015 Forrester Research, Inc. Reproduction Prohibited 98
skill
Dev I&O QA Architects CX
Product
Managers
Cyber
Security
Discovering outcomes ✔︎ ✔︎
Designing capabilities ✔︎ ✔︎ ✔︎ ✔︎
Writing code/scripts ✔︎ ✔︎ ✔︎ ✔︎ ✔︎
Testing code/scripts ✔︎ ✔︎ ✔︎ ✔︎ ✔︎
Assessing usability ✔︎ ✔︎ ✔︎
Optimizing experiences ✔︎ ✔︎ ✔︎ ✔︎
Optimizing application &
operational platforms ✔︎ ✔︎
Reviewing analytic data ✔︎ ✔︎ ✔︎ ✔︎ ✔︎ ✔︎ ✔︎
Securing applications ✔︎ ✔︎ ✔︎ ✔︎
Architecting solutions, patterns,
and IaC ✔︎ ✔︎ ✔︎
Product Team Skills and Roles

99. © 2015 Forrester Research, Inc. Reproduction Prohibited 99
Successful change is bottom-up, with support from leadership to clear obstacles and change culture
Patterns
› Clearing obstacles, including people who get in the
way
› Measuring cycle time, customer satisfaction and
throughput to determine effectiveness
› Promoting a culture of shared accomplishment
across Business, Application Development, QA and
Ops
› Encouraging innovation and experimentation, and
tolerating mistakes so long as learning results
› Dissolving the walls between roles and functional
areas
› “The customer doesn’t always know what they want,
but they know what they don’t want when they see it.”
Anti-patterns
› The executive cheerleader - encouraging but not
involved in the game
› Measuring utilization to determine effectiveness
› Promoting a culture of individual accomplishment and
“accountability”; promoting “heroes”
› Rewarding predictability, and punishing mistakes
› Encouraging an “us-them” culture
› “The customer is always right.”
The Role of Management
Leadership matters

100. Pitfall 7: Letting complexity continue

101. © 2015 Forrester Research, Inc. Reproduction Prohibited 101
Tackling the risk of 3rd party software including open
source
› Use new components
› Use components that do not have any reported CVEs
› Create component library
› Reduce number of versions of a single component
› Don’t forget middleware, OS, network, database, and performance management tools
› Use continuous delivery pipeline tools to catalog which 3rd party software is used and
where it’s located
And when a vulnerability is identified, use the continuous delivery pipeline
to find all affected applications, quickly generate a fix and deploy

102. © 2015 Forrester Research, Inc. Reproduction Prohibited 102
Recommendations for DevOps Success
1. Start small and grow
2. Create a high trust environment
3. Measure based on customer experience
4. Create meaningful, common dashboards
5. Left shift control points
6. Include expertise from key groups such as QA, EA, Security
7. Standardize

103. Thank you
forrester.com
Amy DeMartine
+ademartine@forrester.com
@AmyDeMartine
Kurt Bittner
kbittner@forrester.com
@ksbittner

104. Q&A

123. INSERT JASON CLOSING SLIDES

124. #DevOpsRoadTrip

125. #DevOpsRoadTrip

126. #DevOpsRoadTrip

127. #DevOpsRoadTrip

128. #DevOpsRoadTrip

129. #DevOpsRoadTrip

130. #DevOpsRoadTrip

131. #DevOpsRoadTrip

132. #DevOpsRoadTrip

133. #DevOpsRoadTrip

134. #DevOpsRoadTrip

135. #DevOpsRoadTrip

136. #DevOpsRoadTrip

137. #DevOpsRoadTrip

138. #DevOpsRoadTrip

139. #DevOpsRoadTrip4.21 – 4.22Devopsdaysrox.org

140. #DevOpsRoadTrip

141. DENVER - SEATTLE - SAN FRANCISCO - MINNEAPOLIS - NEW YORK CITY