2019-06-12 DevOpsCon Spring 2019 https://devopsconference.de/business-company-culture/how-weve-killed-devops-by-creating-a-dedicated-devops-team/ In the early days of Netguru, we were a bunch of developers and didn’t even think of having a dedicated ops role - life was simple and easy. The company grew, so did our requirements and challenges, and that’s why we’ve decided to change this by introducing The DevOps Team. That’s also how we’ve entered a very dangerous path with a lot of obstacles related to technology, people, and culture. This presentation is about real scenarios which we’ve encountered and the way we handled them. This is a story about money, expectations, tickets, sweat, and long hours spent making everybody happy. If you’re thinking of hiring a “DevOps Specialist”, then this talk is just for you. Don’t wake up one day and realise that there’s a team in your organisation which takes your focus and energy in the wrong direction.

1. How we killed DevOps by creating
a dedicated DevOps team
Adam Nowak

2. 🌃

3. 😤

4. 📳

5. 🔥

6. 👍

7. 🏁

8. ☎

9. 👶

10. ❌

11. Hello Berlin!
Hello DevOpsCon!

12. I’m Adam 👋

13. _
How did we kill DevOps?

14. Let’s go back in time

15. Pre-DevOps team era
Pre-DevO
ps team

16. •Small web-dev agency
•Simple infrastructures
•No dedicated "ops" role
•Developers manage servers and deploy on their
own
Pre-DevOps team era

17. •Small community
•Open meetings
•Common tools & best practices
•Part-time Ops
Developer manage servers…

18. DevOps definition v1
People who can do servers and update them via scripts
and they automate things.

19. 🤝,
🤯⚙
multiple contexts ad-hoc help
tools & automation "heavy devops"
Challenges

20. •We need someone to take us through the cloud
& container journey
•We want to automate everything
•There is more "heavy DevOps" to handle
Things evolve

21. We need him!

22. Create A proper Team

23. DevOps Team
Pre-DevO
ps team
DevO
ps team

24. 📦☁
1♾
Cloud migration Dockerization
CI/CD DevOps Guardians
Cloud, Docker, CI…

25. Growth happens

26. 34567🧟9: ;<=>?@ABCDEFGHIJKLM👩OPQR
STUVWXYZ[]🧓_`ab🎅defghij👷lmnopqrs
👧uvw🧑yz{|}~•€•‚ƒ„…†‡ˆ‰Š‹Œ•Ž•👵‘’“”
•–—🧙™š›🧜•žŸ ¡¢£¤¥¦§¨©👳«¬-®¯°±²³´🤵
¶£·🦹¹º»¼👦¾¿ÀÁÂÃÄÅÆÇÈÉ👨ËÌÍÎÏÐÑÒÓ🧕Õ
Ö×ØÙÚÛ Ü💂Þßàáâ👸äåæçèéêëìíîïðñòAóô
õö÷ø🧒ú👴üýþÿ!"#$%&'()*+👲-./0123456
789:;<=>ê?@ABCDEF🤶HIJKLMNOPQ🧚STU1
VWXYZ[]^_`abcdefghijklmnopqrstuv
w🧛yz🤴|}~•€•‚ƒ🕵👱†‡ˆ‰Š🧔Œ•Ž••1‘’“”•–
—˜™š›œ🦸žŸ ¡¢£¤🧞¦§¨©Aª«¬-👰¯°±²³👮¿
•sdŸVs|³×Т§Ù
…🧟–µ«👨=®¶🤵👸Ó🧓
€•iIg·ÞŸR_<¯
*>ov-h’ñz`±@¸

27. Something is not right…

28. The DevOps team
A£þê1

29. Something is not right…
This is what
devops do! ☝
I’m here to do
dev stuff! 😤
I don’t know
how to… º
The checklist
doesn’t work ☹
I can’t access the
server 😡
I’m getting crazy about
waiting for 3-4 weeks for
simple tasks! 🕰 🛑

30. 🤔💩❓

31. _
What’s going on?

32. We can’t keep up
with work

33. DevOps vs Projects
Project Project Project
Project Project Project
Project Project Project
Project Project Project
Project Project Project
Project Project Project
Improvement
DevOps
Project Project Project
😡
😠
🤬
👎
🔥
😱

34. _
The result?

35. The Ops knowledge among
the team decreases

36. The tools we created
are for Ops, not Devs

37. Everything is maintained
from one central place

38. The tension between teams
becomes an issue

39. Tension becomes an issue
hey @, check which devs f*cked it up and
tell their leaders about it. I don’t want to
turn the other cheek. This time, it’s them
who should get the a*s beaten.
🔥
The waiting time is a joke.
🚨

40. Let’s hire
More DevOps engineers!

41. Not so easy

42. •No earnings for the company
• 💰 is very important
•DevOps team is not good at 💸
•Nope 🚫
Can we hire more DevOps engineers?

43. Overtime is the new normal

44. On-call is just yet another shift

45. We found that when we woke up developers at 2am, defects got
fixed faster than ever
Patrick Lightbody, SVP Product Management
New Relic

46. 💰👔
💯☝
More serious clients More money involved
Demanding projects 100% commitment
Business landscape changes

47. 👀🆘
🧯🛠
Help needed Watch team’s back
Ongoing
maintenance
Putting fires
At the same time the team still needs our help

48. •There’s a huge demand for the DevOps services
•DevOps is the bottleneck
State of DevOps

49. 🔥🔥🔥
🚒🔥
🧯🔥
🔥
🔥
🔥
🔥
🔥
🔥🔥🔥
🔥 🔥
🔥
🔥
🔥

50. No one is happy

51. 💩❗

52. Desperate times, desperate measures

53. #devops-wartime
Pre-DevO
ps team
DevO
ps team
#devops-wartim
e

54. Communication during #devops-wartime
⛺🔥
✅

55. 📝🧤👣
Daily meetings & status updates

56. 🖖Ø🙌

57. Cooperation with other departments

58. Focus!

59. •NPS from Service Desk tickets >= 50
•Average ticket lead time < 2 work days
•The staging environment is created in less than 1
hour from raising a ticket
Sample goals

60. •Sense of urgency
•Cross-company collaboration and engagement
•Possibility of doing things differently
•Focus
#devops-wartime gave us

61. Something has to change

62. Everyone thinks of changing the world, but no one thinks of
changing himself.
Leo Tolstoy
Novelist, short story writer

63. •We need to collaborate
•We need a goal, a destination
•We need to automate tedious tasks
•We need to iterate and experiment
•We need to communicate better
What do we need?

64. NPS
NPS - NET PROMOTER SCORE
How likely is it that you would recommend our DevOps services to a friend or
colleague?

65. Positive feedback!
I really appreciate
the efforts to
reorganise structure
and processes in the
DevOps team.
DevOps: Guys are doing
massive overtime, their
processes are a to be
cleaned-up - I'm really
hoping they will figure this
out in upcoming months,
fingers crossed!
I think that the situation between PM
and DevOps goes in the right direction,
recent news about the reorganisation
are very encouraging!

66. We’re just getting started

67. DevOps enlightenment
Pre-DevO
ps team
DevO
ps team
#devops-wartim
e
DevO
ps enlightenm
ent

68. The "AHA!" moment

69. Slack conversation with my friend Bartosz

71. DevOps?

72. DevOps!

73. •We need to collaborate
•We need a goal, a destination
•We need to automate tedious tasks
•We need to iterate and experiment
•We need to communicate better
What do we need?

74. DevOps definition v2
Combination of cultural philosophies, practices, and tools
that increases an organization’s ability to deliver
applications and services at high velocity.
HTTPS://AWS.AMAZON.COM/DEVOPS/WHAT-IS-DEVOPS/

76. What does it mean to have a good DevOps team?
😱 Ú

77. DevOps culture
Pre-DevO
ps team
DevO
ps team
#devops-wartim
e
DevO
ps enlightenm
ent
DevO
ps culture

78. Diamond Edwards & John Willis
CALMS
Culture
Automation
Lean
Measurement
Sharing

79. Culture
Focus on people
CALMS
Conversations with
all teams
Part of company’s
quarterly themes
No blaming
Help people
understand DevOpsMeasuring culture
Continuous
Improvement

80. •Ron Westrum
•The organizational culture predicts the way
information flows through an organization
•Information is the Û
•3 types of organizations
๏ Pathological (power oriented)
๏ Bureaucratic (rule oriented)
๏ Generative (performance oriented)
CALMS: CULTURE
Three Cultures Model

81. CALMS: CULTURE
Three Cultures Model (cont.)
How well you identify with the following statements?
• Responsibilities are shared
• Information is actively sought
• New ideas are welcomed
• …

82. CALMS: CULTURE
Westrum + DevOps
+ +
=
Higher levels of software delivery performance
= higher levels of job satisfaction
= higher organizational performance
🔁

83. Automation
Continuous Delivery
CALMS
Slack bot

84. Automation
Continuous Delivery
CALMS
Automation
Slack Bot

85. Automation
Continuous Delivery
CALMS
Slack bot
Prometheus
Single source of
truth
Continuous Delivery as
key project metric
Environment setup

86. Continuous Delivery
CALMS
Automation
Single source
of truth

87. ◎ Lean
Focus on the value
CALMS
Better questions
Lead time
Product demos
Case studies

88. | Measurement
Measure and show the progress
CALMS

89. |
CALMS
Measurement
Collecting
data

90. |
CALMS
Measurement
Netguru
Consulting
Index

91. |
CALMS
Measurement
Netguru
Consulting
Index
Business Value Roadmap
Deployment
frequency Cycle time
Self assessment
Ownership
… and other areas
(~20 in total)

92. |
CALMS
Measurement
Netguru
Consulting
Index

93. |
CALMS
Measurement
DevOps health

94. Sharing
Collaboration and communication
CALMS

95. 
CALMS
Sharing

96. Future of DevOps
Ops? Platform? SRE? All?

97. Key takeawaysfrom our journey

98. Be aware of
DevOps teams

99. We need to work
together

100. Good change
takes time

101. Continuous
*

102. The power of 1% improvements

103. We all are
DevOps

104. DevOps is dead
Long live DevOps!

105. DevOps is a culture,
a movement, a philosophy

106. Thanks!
Adam Nowak
Development Process Practice Lead
@lubieniebieski | adam@netguru.com