The document discusses strategies for improving the experience of engineers who are on-call to address technical issues. It suggests focusing on ensuring alerts are meaningful by only sending ones where action can be taken, and building support structures like runbooks to document procedures and enable calling for backup. It also advocates compensating engineers for extra work, conducting blameless post-mortems to learn from issues, and shifting culture away from a hero mentality to emphasize teamwork. The overall goal is to help prevent burnout among on-call staff.

1. Aaron Aldrich
@CrayZeigh @CageData
THE HUMANE TREATMENT OF
ON-CALL ENGINEERS
STOP BURNOUT WHILE FIREFIGHTING

2. HI!
AARON ALDRICH
▸ @CrayZeigh
▸ aaldrich@cagedata.com
▸ Cage Data, Inc.
▸ @CageData
▸ Husband, Father, Musician, Board
Gamer

3. Aaron Aldrich @CrayZeigh
@CageData
THE BURNOUT
PROBLEM

4. Aaron Aldrich @CrayZeigh
@CageData

5. Aaron Aldrich @CrayZeigh
@CageData

6. Aaron Aldrich @CrayZeigh
@CageData
ON CALL
ONE PIECE OF THE
PUZZLE

7. Aaron Aldrich @CrayZeigh
@CageData
Humans can’t work all the time. We need
food and sleep and free time.

8. Aaron Aldrich @CrayZeigh
@CageData

9. Aaron Aldrich @CrayZeigh
@CageData
On-Call:
an Emergency Response Team for your
technology assets

10. Aaron Aldrich @CrayZeigh
@CageData

11. Aaron Aldrich @CrayZeigh
@CageData
MAKING ON CALL
MEANINGFUL
ACTIONABLE ALERTS

12. Aaron Aldrich @CrayZeigh
@CageData
Being on call should mean something
more than just being woken up in the
middle of the night.

13. Aaron Aldrich @CrayZeigh
@CageData

14. Aaron Aldrich @CrayZeigh
@CageData
ACTIONABLE ALERTS
Does it Matter?
Can it wait?
Can the alerted respond?

15. Aaron Aldrich @CrayZeigh
@CageData
Does it Matter?

16. Aaron Aldrich @CrayZeigh
@CageData

17. Aaron Aldrich @CrayZeigh
@CageData
If there’s nothing to do, log it for later

18. Aaron Aldrich @CrayZeigh
@CageData
Can it wait?

19. Aaron Aldrich - @CrayZeigh
IT SOUNDS
PLAUSIBLE ENOUGH
TONIGHT, BUT WAIT
UNTIL TOMORROW.
WAIT FOR THE
COMMON SENSE OF
THE MORNING.
HG Wells, The Time Machine

20. Aaron Aldrich @CrayZeigh
@CageData

21. Aaron Aldrich @CrayZeigh
@CageData
Can the alerted respond?

22. Aaron Aldrich @CrayZeigh
@CageData
The person receiving an alert should be
equipped to deal with it

23. Aaron Aldrich @CrayZeigh
@CageData

24. Aaron Aldrich @CrayZeigh
@CageData
BUILDING A SUPPORT
STRUCTURE
SHIFTING
DEPARTMENTS

25. Aaron Aldrich @CrayZeigh
@CageData

26. Aaron Aldrich @CrayZeigh
@CageData

27. Aaron Aldrich @CrayZeigh
@CageData

28. Aaron Aldrich @CrayZeigh
@CageData
Runbooks are the bread and butter of the
late-night work process.

29. Aaron Aldrich @CrayZeigh
@CageData
Runbooks tell us What
and How

30. Aaron Aldrich @CrayZeigh
@CageData

31. Aaron Aldrich @CrayZeigh
@CageData
Good Runbooks make
automation possible

32. Aaron Aldrich @CrayZeigh
@CageData
Calling for Backup is OK

33. Aaron Aldrich @CrayZeigh
@CageData
CHANGING THE WAY WE
TALK ABOUT ON CALL
SHIFTING CULTURE

34. Aaron Aldrich @CrayZeigh
@CageData
/r/notinteresting

35. Aaron Aldrich @CrayZeigh
@CageData
Good stories contain struggle

36. Aaron Aldrich @CrayZeigh
@CageData

37. Aaron Aldrich @CrayZeigh
@CageData

38. Aaron Aldrich - @CrayZeigh
SOFTWARE
DEVELOPMENT, IT TURNS
OUT, IS A TEAM SPORT…
AND WHAT’S WORSE,
ENCOURAGING THE HERO
MENTALITY LEADS TO
CORROSIVE DYSFUNCTION
IN SOFTWARE TEAMS.
Rob Mee, Pivotal Labs

39. Aaron Aldrich @CrayZeigh
@CageData
COMPENSATION FOR EXTRA UNPLANNED WORK
We need to compensate our
engineers for completing
Extra-work

40. Aaron Aldrich @CrayZeigh
@CageData
Blameless Postmortems

41. Aaron Aldrich @CrayZeigh
@CageData
WHAT IS A BLAMELESS POSTMORTEM?
▸ Team members are accountable but not responsible
▸ Complete Transparency
▸ Deeper look at circumstances
▸ What happened and how to improve it (specific details)
▸ Real conditions of failure in complex systems
@jasonhand
http://www.slideshare.net/jhand2/its-not-your-fault-blameless-post-mortems

42. Aaron Aldrich @CrayZeigh
@CageData
WHAT’S IN A NAME?
LEARNING
REVIEW

43. Aaron Aldrich @CrayZeigh
@CageData
LET ME SUM UP.
…THERE IS TOO MUCH

44. PEOPLE CHANGE ONLY
THEMSELVES
TAKE THE FIRST STEP

45. Aaron Aldrich @CrayZeigh
@CageData

46. Aaron Aldrich @CrayZeigh
@CageData

47. MAKE ON-CALL GREAT AGAIN NOT SUCK
AARON ALDRICH
@CrayZeigh
aaldrich@cagedata.com
Cage Data, Inc.
@CageData
http://devops.cagedata.com
?