As applications become more distributed and complex, so do our failure modes. In this presentation, I’ll share why you shouldn’t just embrace failure, but why you should induce it to intentionally cause and learn from failure. Together with the audience, I'll run a Chaos experiment to show how they can start their own Chaos engineering and make their systems more resilient.

1. CHAOS!
Breaking your systems to make them unbreakable

2. –Barack Obama
“You can’t let your failures define you.
You have to let your failures teach you.”

3. – Corey Bertram
“Hope is not a strategy.”

4. http://bit.ly/netflix-5-things

5. Chaos Monkey

6. J A S O N Y E E
Te c h n i c a l E v a n g e l i s t
Tr a v e l H a c k e r
W h i s k e y H u n t e r
P o k e m o n Tr a i n e r
Tw : @ g i t b i s e c t
E m : j y e e @ d a t a d o g h q . c o m

7. D ATA D O G
S a a S - b a s e d o b s e r v a b i l i t y
p l a t f o r m :
• M e t r i c s
• Tr a c e s ( A P M )
• L o g s
• S y n t h e t i c s
Tw : @ d a t a d o g h q
We ’ re h i r i n g :
j o b s . d a t a d o g h q . c o m

8. –Netflix Technology Blog, 2011
http://bit.ly/netflix-chaos
“By running Chaos Monkey in the middle of a
business day, in a carefully monitored environment
with engineers standing by to address any problems,
we can still learn the lessons about the weaknesses
of our system, and build automatic recovery
mechanisms to deal with them.
So next time an instance fails at 3 am on a Sunday,
we won’t even notice.”

9. “By running Chaos Monkey in the middle of a
business day, in a carefully monitored environment
with engineers standing by to address any problems,
we can still learn the lessons about the weaknesses
of our system, and build automatic recovery
mechanisms to deal with them.
So next time an instance fails at 3 am on a Sunday,
we won’t even notice.”
–Netflix Technology Blog, 2011
http://bit.ly/netflix-chaos

10. “By running Chaos Monkey in the middle of a
business day, in a carefully monitored environment
with engineers standing by to address any problems,
we can still learn the lessons about the weaknesses
of our system, and build automatic recovery
mechanisms to deal with them.
So next time an instance fails at 3 am on a Sunday,
we won’t even notice.”
–Netflix Technology Blog, 2011
http://bit.ly/netflix-chaos

11. “By running Chaos Monkey in the middle of a
business day, in a carefully monitored environment
with engineers standing by to address any problems,
we can still learn the lessons about the weaknesses
of our system, and build automatic recovery
mechanisms to deal with them.
So next time an instance fails at 3 am on a Sunday,
we won’t even notice.”
–Netflix Technology Blog, 2011
http://bit.ly/netflix-chaos

12. “By running Chaos Monkey in the middle of a
business day, in a carefully monitored environment
with engineers standing by to address any problems,
we can still learn the lessons about the weaknesses
of our system, and build automatic recovery
mechanisms to deal with them.
So next time an instance fails at 3 am on a Sunday,
we won’t even notice.”
–Netflix Technology Blog, 2011
http://bit.ly/netflix-chaos

13. Don’t be a jerk!

14. Game Days

15. 90 Minutes

16. 90 Minutes
30 minutes planning.

17. 90 Minutes
30 minutes planning.
50 minutes playing.

18. 90 Minutes
30 minutes planning.
50 minutes playing.
10 minutes reporting.

19. The Team

20. The Team
1 subject matter expert

21. The Team
1 subject matter expert
1 SRE

22. The Team
1 subject matter expert
1 SRE
Sometimes, 1 junior
engineer/developer

23. Before
• Schedule it.

24. Before
• Schedule it.
• Pick tests. Start easy.

25. Before
• Schedule it.
• Pick tests. Start easy.
• Write down what you expect to happen.

26. Before
• Schedule it.
• Pick tests. Start easy.
• Write down what you expect to happen.
• Write down your plan if things go wrong.

27. Before
• Schedule it.
• Pick tests. Start easy.
• Write down what you expect to happen.
• Write down your plan if things go wrong.
• Share your document!

28. During
• Staging -> Production (off-peak) -> Production (primetime)

29. During
• Staging -> Production (off-peak) -> Production (primetime)
• Announce start in group chat.

30. During
• Staging -> Production (off-peak) -> Production (primetime)
• Announce start in group chat.
• Maintain discussion in group chat.

31. During
• Staging -> Production (off-peak) -> Production (primetime)
• Announce start in group chat.
• Maintain discussion in group chat.
• Monitor for outages.

32. During
• Staging -> Production (off-peak) -> Production (primetime)
• Announce start in group chat.
• Maintain discussion in group chat.
• Monitor for outages.
• Run your test and take notes!

33. After
• Create cards/tickets to track issues that need work.

34. After
• Create cards/tickets to track issues that need work.
• Write a summary & key lessons.

35. After
• Create cards/tickets to track issues that need work.
• Write a summary & key lessons.
• Email to all of engineering.

36. After
• Create cards/tickets to track issues that need work.
• Write a summary & key lessons.
• Email to all of engineering.
• Celebrate!

37. 🍾🎉🐶

38. Game Levels
0. Terminate the service. Block access to 1 dependency.

39. Game Levels
0. Terminate the service. Block access to 1 dependency.
1. Block access to all dependencies.

40. Game Levels
0. Terminate the service. Block access to 1 dependency.
1. Block access to all dependencies.
2. Terminate the host.

41. Game Levels
0. Terminate the service. Block access to 1 dependency.
1. Block access to all dependencies.
2. Terminate the host.
3. Degrade the environment.
Monitor & alert on Latency, Errors, Traffic, & Saturation

42. Game Levels
0. Terminate the service. Block access to 1 dependency.
1. Block access to all dependencies.
2. Terminate the host.
3. Degrade the environment.
4. Spike traffic.

43. Game Levels
0. Terminate the service. Block access to 1 dependency.
1. Block access to all dependencies.
2. Terminate the host.
3. Degrade the environment.
4. Spike traffic.
5. Terminate the region/cloud.

44. Experiment time!

45. Review
Share!
Plan!
Have fun!

46. Additional Resources
• systemctl, kill, iptables

47. Additional Resources
• systemctl, kill, iptables
• Comcast - https://github.com/tylertreat/comcast

48. Additional Resources
• systemctl, kill, iptables
• Comcast - https://github.com/tylertreat/comcast
• Vegeta - https://github.com/tsenart/vegeta

49. Additional Resources
• systemctl, kill, iptables
• Comcast - https://github.com/tylertreat/comcast
• Vegeta - https://github.com/tsenart/vegeta
• Kubernetes & Istio -
https://istio.io/docs/tasks/traffic-management/fault-injection/

53. apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: wordpress-virtualservice
spec:
hosts:
- "*"
gateways:
- wordpress-gateway
http:
- route:
- destination:
host: wordpress
fault:
abort:
percentage:
value: 50.0
httpStatus: 500
delay:
percentage:
value: 50.0
fixedDelay: 30s

54. Additional Resources
• Gremlin - https://www.gremlin.com/
• ChaosToolkit - https://chaostoolkit.org/

55. Questions?
jason.yee@datadoghq.com
@gitbisect

56. Slides: bit.ly/cmr-chaos
Documents: bit.ly/cmr-chaos-docs
jason.yee@datadoghq.com
@gitbisect