The document discusses the importance of using automated testing to prove that software is fault tolerant. It emphasizes that features without automated tests are just rumors and that you don't know if a service is truly fault tolerant unless you test for faults. It promotes using protocol level test doubles like Wiremock and stubbed databases like Cassandra to deterministically create any scenario, including unhappy paths and faults. This allows software to be tested as if it were surviving chaos monkey attacks and real world failures before being deployed to production. Examples of fault scenarios that can be tested include delays, distributed failures, timeouts and errors.

1. THINK YOUR SOFTWARE IS FAULT
TOLERANT? PROVE IT
CHRISTOPHER BATEY
@CHBATEY

2. TESTING

3. FEATURES WITHOUT AUTOMATED TESTS
ARE MERELY RUMOURS

4. HAPPY PATH

5. UNHAPPY PATH

6. YOU DON’T KNOW A
SERVICE IS FAULT
TOLERANT IF YOU
DON’T TEST FAULTS

7. PROTOCOL LEVEL TEST DOUBLE

8. YOUR SERVICE
PROTOCOL
LEVEL
TEST
DOUBLE
TEST
PRIMING
Real HTTP/
TCP

9. Examples
REST Driver
Wiremock
MOCO
Stubbed Cassandra

10. DETERMINISTICALLY CREATE ANY
SCENARIO

13. TO SURVIVE THE CHAOS MONKEY

14. TO SURVIVE REALITY

15. ENOUGH FLUFF

16. Pesky User
(that’s us)
using cURL
Enterprise hello
world
A different team’s
software (probably
rubbish)
Crazy distributed
database
HTTP
/name
TCP
surname

17. Dependency VM
Wiremock
Stubbed
Cassandra
192.168.2.2

18. Wiremock
192.168.2.2
Enterprise hello
world
localhost
192.168.2.1

19. PRIMING
HITTING THE
SERVICE

21. DEMO TIME

23. Wiremock
192.168.2.2
Enterprise hello
world
localhost
192.168.2.1
Delay is here

24. Wiremock
192.168.2.2
Enterprise hello
world
localhost
192.168.2.1
Delay is here

25. Wiremock
192.168.2.2
Enterprise hello
world
localhost
192.168.2.1
Delay is here

26. Wiremock
192.168.2.2
Enterprise hello
world
localhost
192.168.2.1
Delay is here

27. NOT JUST FOR HTTP

28. PRODUCTION?
Application
Issues here

29. PRODUCTION?
ApplicationApplicationApplicationApplicationApplication

30. ApplicationApplicationApplicationApplicationApplication
ApplicationApplicationApplicationApplicationApplication
DC1
DC2

31. Distributed failures
success
read_request_timeout
unavailable
write_request_timeout
server_error
protocol_error
bad_credentials
overloaded
is_bootstrapping
truncate_error

32. READ TIMEOUT
Application C
R1
R2
R3C=QUROUM
Replication factor: 3
timeout
timeout
Read timeout

33. DEMO TIME

36. Resources
Wiremock: http://wiremock.org/
Awesome TC guide: http://blog.edseek.com/~jasonb/
articles/traffic_shaping/
Stubbed Cassandra: http://scassandra.org
Wireshark: https://www.wireshark.org/
Saboteur: https://github.com/tomakehurst/saboteur
Example app + VM: https://github.com/chbatey/vagrant-
wiremock-saboteur

37. THANKS FOR
LISTENING
@CHBATEY