When we discuss DevOps from a software delivery perspective, all that we hear about is the need for speed: faster tests, more frequent deployments, cloud auto-scaling, and so on. Does this mean we are going to break things more often as well? Maybe. But not necessarily. To go faster efficiently, we need to frequently deliver smaller changes with low risk. High performing organizations achieve this flow by applying a set of principles and practices that we will cover in this talk. And, of course, they measure themselves!

1. Software Quality and DevOps
Friends or Foes?
Manuel Pais
@manupaisable | manuelpais.net

2. About me
Manuel Pais
MS Software Eng
@manupaisable
me@manuelpais.net
http://manuelpais.net
DevOps Consultant & Trainer
Co-author of “Team Topologies”
Focused on teams and flow
2@manupaisable | manuelpais.net

3. Agenda
1. The Need for Speed (aka DevOps)
2. Failure = Quality in Complex Systems
3. Survival of High-Performing Cultures

4. Agile… Scrum

5. Agile… Scrum

6. Scrum, Scrum, Scrum

7. Scrum, Scrum, Scrum

8. Scrum, Scrum, Scrum

9. Scrum, Scrum, Scrum

10. Wall of confusion

11. “DevOps brought to the attention
that two worlds, typically apart in a
company, need to collaborate and
that actually gives you a
competitive edge”
–Patrick Debois
infoq.com/interviews/debois-devops

12. “Agile System Administration was
too long and too narrow… ”
–Patrick Debois
infoq.com/interviews/debois-devops

14. “ability to get changes of all types,
into production, or into the hands of
users, safely and quickly in a
sustainable way”
–Jez Humble
continuousdelivery.com

16. Risk-based activities (short & wide pipeline)

17. 17

18. Speed lead time
deployment frequency
Quality change failure rate
Operability MTTR
18

19. 19

20. Test Question #1
What is the total percentage of software releases
without bugs across the industry?
● 0-2%
● 3-5%
● 6%-15%
● > 15%

23. The lawsuit states that in early 2016, Hertz began an ambitious project to transform its digital identity.
Lacking the internal expertise and resources to carry out the work itself, Hertz picked Accenture from a list of
potential candidates to design, build, test, and deploy Hertz’s new website and apps.
As reported by The Register, the revamped website was supposed to go live in December 2017, but the
deadline was delayed to January 2018 then pushed back again to April 2018, which was also missed.
By May 2018, Hertz lost patience and terminated Accenture’s services. The lawsuit claims the car firm no
longer had any confidence that Accenture was capable of completing the project.

24. “Accenture did not perform tests on many components of the system.
When Accenture did perform tests, they were seriously inadequate,
to the point of being misleading.”
The lawsuit states that in early 2016, Hertz began an ambitious project to transform its digital identity.
Lacking the internal expertise and resources to carry out the work itself, Hertz picked Accenture from a list of
potential candidates to design, build, test, and deploy Hertz’s new website and apps.
As reported by The Register, the revamped website was supposed to go live in December 2017, but the
deadline was delayed to January 2018 then pushed back again to April 2018, which was also missed.
By May 2018, Hertz lost patience and terminated Accenture’s services. The lawsuit claims the car firm no
longer had any confidence that Accenture was capable of completing the project.

28. Little’s Law
L = λW
“work noom/flow-efficiency-a-great-metric-you-probably-arent-using
@manupaisable | manuelpais.net 28

29. Little’s Law for Software
L = λW
“work noom/flow-efficiency-a-great-metric-you-probably-arent-using
@manupaisable | manuelpais.net 29

30. Little’s Law for Software
L = λW
Work In Progress (WIP) = Throughput x Cycle Time
noom/flow-efficiency-a-great-metric-you-probably-arent-using
@manupaisable | manuelpais.net 30

31. Work In Progress (WIP) = Throughput x Cycle Time
@manupaisable | manuelpais.net 31
What happens when we start more work?
Work takes longer to finish!

32. Work In Progress (WIP) = Throughput x Cycle Time
@manupaisable | manuelpais.net 32
What happens when we start more work?
Work takes longer to finish!

33. Agenda
1. The Need for Speed (aka DevOps)
2. Failure = Quality in Complex Systems
3. Survival of High-Performing Cultures

39. Systems of systems
Complex run time dependencies
Vulnerable build time dependencies
Failure is endemic

43. “The zero-error fallacy”
Researchers at MIT have shown that:
a) the more incidents an airline has, the lower the
passenger mortality risk
b) construction sites with relatively more incidents in a
given year have fewer worker deaths than those with
zero incidents.
https://techbeacon.com/zero-error-fallacy-what-really-counts-devops-teams

44. “The zero-error fallacy”
Researchers at MIT have shown that:
a) the more incidents an airline has, the lower the
passenger mortality risk
b) construction sites with relatively more incidents in a
given year have fewer worker deaths than those with
zero incidents.
https://techbeacon.com/zero-error-fallacy-what-really-counts-devops-teams

45. Learning from Failure
Greatest illusion is that the difference between excellent
and crappy operations is the number of errors or failures

46. Learning from Failure
What makes a difference is the presence of positive
capacities—in people, in teams, in the organization.
Greatest illusion is that the difference between excellent
and crappy operations is the number of errors or failures

47. Learning from Failure
What makes a difference is the presence of positive
capacities—in people, in teams, in the organization.
Greatest illusion is that the difference between excellent
and crappy operations is the number of errors or failures
A safety culture is one in which the boss actually invites
bad news, and may even reward it.

48. time between failures
time to repair

49. time between failures
time to repair

53. Development vs Maintenance

54. Development vs Maintenance
TEAM builds, deploys, runs, monitors and fixes
+ Ops provides platform

55. DONE = deployed

56. DONE = deployed
DONE = monitored in production

57. DONE = deployed
DONE = monitored NOT in production

58. Incident reviews
http://www.slideshare.net/jhand2/its-not-your-fault-blameless-post-mortems

61. Chaos engineering
https://medium.com/netflix-techblog/chaos-engineering-upgraded-878d341f15fa

62. Wrong incentives
Rewarding zero defects / fixing defects

63. Wrong incentives
Focus on simple/single metric
Rewarding zero defects / fixing defects

64. Wrong incentives
Focus on simple/single metric
Rewarding zero defects / fixing defects
Different IT teams with different goals

66. Right incentives
Reward along business objectives

67. Right incentives
Combination of metrics (e.g. lead time + time to repair)
Reward along business objectives

68. Right incentives
Combination of metrics (e.g. lead time + time to repair)
Reward along business objectives
All IT teams share same objectives

69. Agenda
1. The Need for Speed (aka DevOps)
2. Failure = Quality in Complex Systems
3. Survival of High-Performing Cultures

70. source: http://continuousdelivery.com/implementing/culture
Culture Types

71. Blame Culture
Resistance to Change
Lack of Collaboration

72. Blame Culture
Resistance to Change
Lack of Collaboration

73. Blame Culture
Resistance to Change
Lack of Collaboration

74. source:
http://continuousdelivery.com/implementing/culture
Culture Types

75. Blameless
Continuous Learning
High Collaboration

76. Blameless
Continuous Learning
High Collaboration

77. Blameless
Continuous Learning
High Collaboration

79. It’s obvious there is only one *right* way
Stop when...

80. It’s obvious there is only one *right* way
Good enough is not good enough for you
Stop when...

81. It’s obvious there is only one *right* way
Good enough is not good enough for you
You see industry leaders as “rare birds”
Stop when...