My slides for the #STSI2017 videoconference about Chaos Engineering.

1. Chaos Engineering
Willian Paixao
September 26, 2017
DevOps Engineer at Ellmount

2. Essential

3. Monitoring
What gets measured gets managed.
1

4. Monitoring levels
• Server
2

5. Monitoring levels
• Server
• Network
2

6. Monitoring levels
• Server
• Network
• Application
• Lint
• Coverage
• Unit
• API
2

7. Monitoring levels
• Server
• Network
• Application
• Lint
• Coverage
• Unit
• API
• UI
2

8. Monitoring levels
• Server
• Network
• Application
• Lint
• Coverage
• Unit
• API
• UI
• Performance
2

9. Testing
3

10. Chaos Engineering

11. Definition
Chaos Engineering is the discipline of experimenting on a
distributed system in order to build confidence in the system’s
capability to withstand turbulent conditions in production.
4

12. Reasons
5

13. Failure injection
• Killing daemons
6

14. Failure injection
• Killing daemons
• Partially deleting database, cache and queues
6

15. Failure injection
• Killing daemons
• Partially deleting database, cache and queues
• Simulating network failure
• Injecting latency between services
• Dropping packages
• Blocking ports
6

16. Failure injection
• Killing daemons
• Partially deleting database, cache and queues
• Simulating network failure
• Injecting latency between services
• Dropping packages
• Blocking ports
• Changing the system’s clock
6

17. Failure injection
• Killing daemons
• Partially deleting database, cache and queues
• Simulating network failure
• Injecting latency between services
• Dropping packages
• Blocking ports
• Changing the system’s clock
• Making the server run out of resources (CPU, memory or
disk)
6

18. Failure injection
• Killing daemons
• Partially deleting database, cache and queues
• Simulating network failure
• Injecting latency between services
• Dropping packages
• Blocking ports
• Changing the system’s clock
• Making the server run out of resources (CPU, memory or
disk)
• Stressing the application
6

19. Failure injection
• Killing daemons
• Partially deleting database, cache and queues
• Simulating network failure
• Injecting latency between services
• Dropping packages
• Blocking ports
• Changing the system’s clock
• Making the server run out of resources (CPU, memory or
disk)
• Stressing the application
Use your imagination!
6

20. Log
7

21. Happy path
• Reproduce past events or break random services?
8

22. Happy path
• Reproduce past events or break random services?
• Warn people or just break it?
8

23. Happy path
• Reproduce past events or break random services?
• Warn people or just break it?
• Graceful restart services
8

24. Happy path
• Reproduce past events or break random services?
• Warn people or just break it?
• Graceful restart services
• Break it!
8

25. Happy path
• Reproduce past events or break random services?
• Warn people or just break it?
• Graceful restart services
• Break it!
• Monitor, analyze and document
8

26. Happy path
• Reproduce past events or break random services?
• Warn people or just break it?
• Graceful restart services
• Break it!
• Monitor, analyze and document
• Repeat
8

27. Chaos Monkey
github.com/Netflix/chaosmonkey
9

28. References
Chaos Engineering
Site Reliability Engineering 10

29. References
A Netflix Guide to Microservices
Engineering Netflix Global Operations in the Cloud
11

30. References
Chaos & Intuition Engineering at Netflix
Choose Your Own Adventure
12

31. Questions?
13

32. Contact me
@willianpaixaoo // willian@ufpa.br
14