Top Lessons Learned While Researching and Writing The DevOps Handbook
In this webinar, Gene Kim shares his top insights discovered while co-authoring The DevOps Handbook with Jez Humble, Patrick Debois, and John Willis, including:
• Informative DevOps transformation case studies around continuous integration and delivery
• Jez Humble’s latest definitions of continuous delivery vs. deployment
• How Conway’s Law and architecture can both hinder and enable success
• Concrete techniques to build a culture of continuous experimentation and learning – including those from Google, Etsy, Nordstrom, and Capital One
2. ‘tis the season for town hall debate formats
Mark Tomlinson
Performance Sherpa
@mark_on_task
Andi Grabner
Performance Advocate
@grabnerandi
Gene Kim, CTO
Researcher and Author
@RealGeneKim
3. Gene, thoughts on companies that move
to more rapid deployment mode and
their success?
4. The DevOps Handbook Is Here!
•5+ Years & Now Avail!
•23 Chapters
•48 Case Studies
•48 Images
192 footnotes
503 endnotes
98,124 words
5. High Performers Are More Agile
200x 2,555x
more frequent
deployments
faster lead times
than their peers
Source: Puppet Labs 2015 State Of DevOps Report: https://puppet.com/resources/white-paper/2016-state-of-devops-report
6. High Performers Are More Reliable
3x 24x
lower change
failure rate
faster meantime to
recover (MTTR)
Source: Puppet Labs 2015 State Of DevOps Report: https://puppet.com/resources/white-paper/2016-state-of-devops-report
7. High Performers Are More Secure and Controlled
2x 29%
less time spent
remediating
security issues
more time spent
on new work
Source: Puppet Labs 2015 State Of DevOps Report: https://puppet.com/resources/white-paper/2016-state-of-devops-report
8. High Performers Win in the
Marketplace
2x 50%more likely to
exceed profitability,
market share &
productivity goals
higher market
capitalization growth
over 3 years*
Source: Puppet Labs 2015 State Of DevOps Report: https://puppet.com/resources/white-paper/2016-state-of-devops-report
10. @RealGeneKim
Increased Flow of High Quality Value
Test Driven Development
Automated Deployments
Infrastructure as Code
Virtual/Cloud/PaaS/Containers
Break the Monolith to
Reduce Code & Dependency
Complexity
Shift-Left Performance
Automated Feedback Loops
@grabnerandi
11. @RealGeneKim
Real User Feedback: Building the RIGHT thing RIGHT!
Experiment &
innovate on
new ideas
Optimizing what is
not perfect
Removing
what
nobody
needs
@grabnerandi
20. 2011: APM about to be disrupted!
• Migrate from On-Prem to VM, Cloud, Containers and PaaS
• Architectures include micro-services, on-demand scaling, self-
healing
• ”Cloud Natives“ demand SaaS based solutions
• Digital Transformers demand Analytics for Biz, Dev, Ops & Sec
• Many new players on the market
@grabnerandi
21. Challenges to master!
• Bridging the gap between ”New Stack“ and “Enterprise Stack“
• Deploying the same way our customers do: Continuously!
• Not disrupting current operations and slower moving customers
• Aligning 300+ engineers across 3 different geos
@grabnerandi
22. Believe in the mission impossible
6months
major/minor release
+ intermediate updates
+ weeks to months
rollout delay
sprint releases
(continuous-delivery)
1h: code to production
@grabnerandi
24. #1: Going from 6 Months to 1 Month On Premise Updates
• Challenge: Monolith download too big for our customers
• Impact: Update Process was error prone and “All or Nothing“
• Solution: Componentize, Automate Rollout/Rollback Capability,
A/B Rollout Model
Increased velocity uncovered bottlenecks!
@grabnerandi
25. #2: Education on Frequent Updates
• Challenge: Release Education used to happen 60-90 Days
after the release
• Impact: Upgrade to latest version happened very late
• Solution: Education Integrated into Continuous Delivery:
Dev Blogs, YouTube Videos...
Increased velocity uncovered bottlenecks!
@grabnerandi
26. #3: Availabilty of Development / Test Environments
• Challenge: Supporting many different tech stack makes it hard
to maintain it
• Impact: Long running support tickets and long feature
development
• Solution: Infrastructure as Code gives “On Demand“ access to
these enviornments
Increased velocity uncovered bottlenecks!
@grabnerandi
27. • Sprint Reviews Done on “dynaSprint“
• Daily Builds get deployed on “dynaDay“. Sprint builds to “dynaSprint“
• If you can only show it “on your dev machine“ its NOT DONE!
• Deploy Sprint Builds into our internal Production Enviornment
• We monitor Website, Support, Licensing, Community ... With Dynatrace
• If we break our own backoffice software we ALL feel the pain right away
Confidential, Dynatrace, LLC
Tip #1: Increasing Sprint Quality
@grabnerandi
29. • Which Features to Optimize? Which Features to „Phase Out“
• Allows Reducing Technical and Business Debt
Confidential, Dynatrace, LLC
Tip #3: Feedback Loops in SaaS & On-Premise
@grabnerandi
31. DevOps Enterprise: Lessons Learned
• On Oct 19-21, we held the second DevOps Enterprise Summit, a
conference for horses, by horses
• Speakers included fifty leaders from:
• Macy’s, Disney, Target, GE Capital, Western Union, Sherwin Williams, Blackboard,
Nordstrom, Telstra, US Department of Homeland Security, CSG, Raytheon, IBM,
Ticketmaster, MITRE, Marks and Spencer, Barclays Capital, Microsoft, Nationwide
Insurance, Capital One, Gov.UK, Fidelity, Rally Software, Neustar, Walmart, PNC,
ADP, …
32. Observations
• They were using the same technical practices and getting the same sort of
metrics as the unicorns
• Target: 100+ deploys per week, < 10 incidents per month, enabled 53 business
initiatives
• Capital One: 100s of deploys per day, lead time of minutes
• Macy’s: 1,500 manual tests every 10 days, now 100Ks automated tests run daily
• Disney: Has embedded nearly 100 Ops engineers into LOB teams across the
enterprise
• Nationwide Insurance: Retirement Plans app (COBOL on mainframe)
• Raytheon: testing and certification from months to a day
• US CIS: security and compliance testing run every code commit
35. Dr. Steven Spear
“While designing
perfectly safe systems is
likely beyond our abilities,
safe systems are close to
achievable” when the
four following conditions
are met…”
Source: Dr. Steven Spear
36. @RealGeneKim
Dr. Steven Spear’s Four Capabilities
1. See problems as they occur
2. Swarm and solve problems to create new
knowledge
3. Spread new knowledge throughout the
organization
4. Leaders create new leaders
Source: Dr. Steven Spear
37. @RealGeneKim
Open / sharing cross team culture
e.g: Capital One
open sourcing
their Hygiea
Dashboards
Larger Enterprises
sharing their
stories, pitfalls and
lessons learned
via Blog or
Conferences (such
as DOES, DOD,
Meetups, etc.)
Tip: Start internal DevOpsDays to promote
lessons learned within the organization
@grabnerandi
39. @RealGeneKim
Q & A
Mark Tomlinson
Performance Sherpa
@mark_on_task
Andi Grabner
Performance Advocate
@grabnerandi
Gene Kim, CTO
Researcher and Author
@RealGeneKim
40. Lots of cool stuff!
Join us, stay connected, learn more!
Order the “DevOps
Handbook” now on Amazon,
click HERE
The “DevOps Handbook”
catalogues “high performing
DevOps organizations” and
what they have in common.
It then provide prescriptive
guidance so that other
organizations can replicate
their results.
Follow the Dynatrace Devops
story, join upcoming Webinar
“Lessons Learned Moving
Dynatrace from an On-Prem
Company to One that is
Cloud Native” click HERE
41. Lots of cool stuff!
Join us, stay connected, learn more!
Learn more about the
DevOps Enterprise Summit,
click HERE
The goal of this event is to
give leaders the tools and
practices they need to
develop and deploy software
faster and to win in the
marketplace.
Join our PurePerformance
Podcast, click HERE
Gene Kim will join us live to
take us thorugh more details
of top insights discovered
while co-authoring The
DevOps Handbook