What’s the value proposition of DevOps? Does culture change show up in the bottom line? What practices predict high IT performance? We hear many stories to inspire and inform us, but the plural of anecdote is not data. Let’s dive into the research and find out which DevOps practices drive optimal IT and business outcomes. The data shows that the best IT performers have the highest throughput and reliability while contributing to organizational profitability, productivity, and market share goals. Industry trends around security, containers, continuous delivery, and lean management relate to IT performance and quality: let’s talk about how. Management and practitioners alike will leave with a better understanding of how to achieve the best outcomes, while armed with the data they need to make the case for change.

1. Nicole Forsgren, PhD
Director of Organizational Performance & Analytics, Chef
Research Affiliate, Clemson University
The Data on DevOps:
Making the Case for Awesome

2. “IT doesn’t matter.”
-- Nicholas Carr, 2003
@nicolefv

3. IT Does Matter
• Times – and IT – have changed
• DevOps is good for Organizations
• DevOps is good for IT
• And then some detail: What drives this change?
• Technical practices (hint: Continuous Delivery)
• Management practices (hint: from Lean Management)
• Culture and identity
@nicolefv

4. @nicolefv

5. @nicolefv

6. DevOps is
Technical practices
seen in Continuous Delivery,
Management practices
seen in Lean Management principles, and
Organizational Culture and Identity
@nicolefv
Research shows that these drive IT Performance
and Organizational Performance

7. DevOps is good for organizations
@nicolefv

8. High Performing IT organizations
2x
More likely to exceed
Profitability,
Market share, and
Productivity goals
50%
Higher market cap
growth over 3 years*
The 2014 and 2015 State of DevOps Report
@nicolefv

9. Devops is good for IT
Measuring DevOps and IT Performance
- Deploy frequency (Note: NOT delivery)
- Lead Time for Changes
- Mean Time to Recover (MTTR)
- Change Fail Rate
@nicolefv

10. High Performing DevOps teams
More agile
The 2016 State of DevOps Report
@nicolefv

11. High Performing DevOps teams
More reliable
The 2016 State of DevOps Report
@nicolefv

12. IT Performance over the years
The 2016 State of DevOps Report
@nicolefv
Deploy Frequency Change LeadTime MeanTime to Recover

13. DevOps promises – and delivers
More throughput
More stability
In tandem. Without the tradeoffs that
ITIL calls for.
Let’s talk about what this means for us
@nicolefv

14. High Performing DevOps teams
More agile What does this mean for:
200x
More frequent
deployments
The 2015 DevOps Survey of Practice and its resulting database are the property of Puppet Labs, Inc. and Gene Kim and Associates, LLC. Allrights reserved.
New content delivery
Value/savings around A/B testing
Value around speed to market
Compliance / regulatory
Security2,555x
Faster lead times
@nicolefv

15. Evaluating well-designed and executed
experiments that were designed to
improve a key metric, only about 1/3
were successful at improving the key
metric!
Online Experimentation at Microsoft, Kohavi et al http://stanford.io/130uW6X
@nicolefv

16. High Performing DevOps teams
More reliable What does this mean for:
3x
Fewer deploy
failures
The 2016 State of DevOps Report
Value/savings around reliability
Value/savings around uptime
Compliance
Security
Reputation around uptime,
compliance & security
24x
Faster MTTR
@nicolefv

17. @nicolefv

18. "Fixing it as soon as possible or
having compensating controls in
place days before could have
saved this entire breach from
occurring in the first place."

19. We know:
• IT Performance is comprised of throughput and stability,
and both are possible without tradeoffs
• IT Performance contributes to org performance ($$$)
So:
What drives IT and Organizational Performance?
@nicolefv

20. IT
Performance
Org
Performance???
@nicolefv

21. IT
Performance
Org
Performance???
1.
@nicolefv

22. IT
Performance
Org
Performance???
2. 1.
@nicolefv

23. IT
Performance
Org
Performance???
2. $$$1.
@nicolefv

24. Continuous Delivery makes our work better…
The 2016 State of DevOps Report
@nicolefv

25. “We never had testability before. We have it now. We have this experience
and know this stuff is working, and working with controls.”
– Product Owner for Yahoo Chef implementation
Automated configuration and
deployment of 250,000 nodes
Can deploy up to 140k node
configurations in 8 hours.
Can patch entire infrastructure within 6
hours of a patch being made available
@nicolefv

26. Continuous Delivery makes our work better… and
makes us feel better!
@nicolefv
https://devops-research.com
Forsgren, N., J. Humble (2016).
"The Role of Continuous Delivery in
IT and Organizational
Performance." In the Proceedings
of the Western Decision Sciences
Institute (WDSI) 2016, Las Vegas,
NV. Available at SSRN:
http://ssrn.com/abstract=2681909

27. Microsoft Engineering: DevOps Lessons
Thiago Almeida -- DevOps Days London, 2016
@nicolefv
Work/Life Scores
Before CD: 38%
After CD: 75%
Source: https://vimeo.com/165184757

28. Continuous Delivery makes our work better…
The 2016 State of DevOps Report
@nicolefv

29. But what else drives IT Performance?
The 2015 DevOps Survey of Practice and its resulting database are the property of Puppet Labs, Inc. and Gene Kim and Associates, LLC. Allrights reserved.
@nicolefv

30. Lean Management practices
The 2015 State of DevOps Report
@nicolefv

31. “If it moves, graph it.”
- Michael Rembetsy, Vice President Operations, Etsy
@nicolefv

32. “I was trying to figure out why my team was working
themselves to death but not getting anything done… By
implementing WIP limits, we were able to focus on our work.
Finishing work feels better than sprinting and feeling like a
hero in the moment, because that’s only a moment.”
- Julia Wester,
Development Manager
for Turner Sports,
Turner Broadcasting
@nicolefv

33. Lean Product Management practices
The 2016 State of DevOps Report
@nicolefv

34. Organizational Culture
@nicolefv
Pathological
Power-oriented
Bureaucratic
Rule-oriented
Generative
Performance-oriented
Low cooperation Modest cooperation High cooperation
Messengers shot Messengers neglected Messengers trained
Responsibilities shirked Narrow responsibilities Risks are shared
Bridging discouraged Bridging tolerated Bridging encouraged
Failure leads to
scapegoating
Failure leads to justice Failure leads to inquiry
Novelty crushed Novelty leads to
problems
Novelty implemented

35. • I am glad I chose to work for this organization rather than another
company.
• I talk of this organization to my friends as a great company to work for.
• I am willing to put in a great deal of effort beyond what is normally
expected to help my organization to be successful.
• I find that my values and my organization's values are very similar.
• In general, the people employed by my organization are working toward
the same goal.
• I feel that my organization cares about me.
Adapted from adapted from Atreyi Kankanhalli, Bernard C.Y. Tan, and Kwok-Kee Wei (2005),
“Contributing Knowledge to Electronic Knowledge Repositories: An Empirical Investigation,“ MIS
Quarterly, 29, 113-143.
Identity
@nicolefv

36. Google Team
Performance
Surprise! No magical
formula for what makes
the perfect team

37. Intuit
“By installing a rampant innovationculture, we performed
165 experiments in the peak three months of tax season.
Our business result? Conversion rate of the website is up
50%. Employee result? Everyone loves it, because their
new ideas can make it to market. ”
- Scott Cook, Intuit founder
@nicolefv

38. Amazon
“I think buildingthis culture is the key to innovation.
Creativity must flow from everywhere. Whether you are a
summer intern or the CTO, any good idea must be able to
seek an objective test, preferably a test that exposes the idea
to real customers. Everyone must be able to experiment,
learn, and iterate.”
- Greg Linden
@nicolefv

39. IT Does Matter
• Times – and IT – have changed
• DevOps is good for Organizations
• DevOps is good for IT
• And then some detail: What drives this change?
• Technical practices (hint: Continuous Delivery)
• Management practices (hint: from Lean Management)
• Culture and identity
@nicolefv

40. For more information:
@nicolefv
Sign up for our ROI
whitepaper & get peer-
reviewed research
devops-research.com

41. nicole@chef.io
@nicolefv
nicolefv.com
Thank you

42. Appendix

43. 10 deploys per day
Dev & ops cooperation at Flickr
John Allspaw & Paul Hammond
Velocity 2009
That was then…
@nicolefv

44. Amazon Deployment Stats
(production & host environments only)
This is now…
1,079Max deploys
In a single hour
Every 11.6 seconds!
10,000Mean # hosts receiving
Deploys simultaneously
30,000Max # hosts receiving
Deploys simultaneously
@nicolefv

45. Etsy Code Deployment
What once required 6-14 hours and an “Army”
…Now takes 15 minutes and 1 person
This is now…
2013 Mike Brittain,Continuous Deployment:The Dirty Details
3/2014 Daniel Schauenberg ,Qcon London
4/2014 tweet@philkates
30+
Deploys
per day
2013
50
Deploys per day
March 2014
QCon London
80-90
Deploys per day
April 2014
Chef Conf
@nicolefv