Chaos Drills can contribute a lot to your services resilience, and it’s actually quite a fun activity. We’ve (Outbrain) built a tool called GomJabbar to help you run those drills. GomJabbar is similar to Netflix's Chaos Monkey, but was built for a more relaxed environment / platform requirements, and allow it to run on your private cloud infrastructure. In this talk I'll explain why we built it, and how you can utilize it to improve your infrastructure and services resilience to failures. The video can be found here: https://youtu.be/Jecwwd_xoiI

1. Chaos Drills FTW
Introducing GomJabbar
Eran Harel
@eran_ha

2. On Call Madness

3. Can we make our on-call shift less stressful?

4. Chaos Drills FTW!!!!!!!11

5. The problem space

6. Practice Handling Failures During Work Time

7. Drill Plan

8. Select a target
$ curl "http://localhost:8080/gj/api/selectTarget"
{
"-2898334020417800152" : {
"instanceCount" : 16,
"module" : "MyModule",
"host" : "some-host.mycompany.com",
"tags" : [
"all",
"cluster-dc1",
"environment-prod",
]
}
}

9. Activate Failure
$ curl http://localhost:8080/gj/api/trigger
?targetToken=-2898334020417800152
&faultId=graceless_shutdown

10. Now we wait...

11. Post Mortem
● The scenario
● Expected behavior
● Actual behavior
● Action Items

12. Priceless Data!

13. "I must not fear.
Fear is the mind-killer.
Fear is the little-death that brings total obliteration.
I will face my fear.
I will permit it to pass over me and through me.
And when it has gone past I will turn the inner eye to
see its path.
Where the fear has gone there will be nothing.
Only I will remain."
(Litany Against Fear - Frank Herbert - Dune)
Getting Started at your Organization

14. So... What is GomJabbar?
● Discovery
● Fault Execution Scheduling
● Configurable Fault Commands
● Configurable Target Filters
● Audit Log
● Revert Automation

15. Ad-Hock Fault Commands
commands:
graceless_shutdown:
description: "Force kill services"
fail: "sudo pkill -9 -f ${module}"
revert: "sudo service ${module} start"

16. Fault Scripts
scripts:
My_Fault_Script:
description: "Fails stuff"
fail:
URL: "http://my.script.com/script.sh"
args: "-f foo -b bar"
revert:
URL: "http://my.script.com/script.sh"
args: "-revert"

17. What did we learn?
● Broken Monitoring
● Broken Alerts
● Unattended Ownerless Services
● SPOF Modules
● Cascading Failures caused by high latency and
packet loss

18. Too Long; Didn’t Listen

19. Links
https://github.com/outbrain/GomJabbar
https://github.com/outbrain/GomJabbar/issues
https://github.com/outbrain/GomJabbar/blob/master/docs/user-guide.md
http://principlesofchaos.org/

20. Can GomJabbar be used on key
Engineers?

21. Questions?