Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2pqx1Jn. Heather Nakama tells the story of implementing chaos testing on a small product, and how several small and targeted early investments in chaos engineering saved time and effort. Filmed at qconsf.com. Heather Nakama is a Senior Software Engineer at Azure Search, a managed search-as-a-service offering from Microsoft's Azure cloud service platform. She works on the backend infrastructure that deploys, monitors, elastically scales, and automatically heals clusters hosting customer services. She has a passion for distributed systems and ensuring that they are scalable, fault-tolerant, and reliable.

1. Heather Nakama, QCon
11/14/17
CHAOS
ENGINEERING
ON A BUDGET
Heather Nakama
Senior Software Engineer
Microsoft Azure Search

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
chaos-engineering-budget

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. CHAOS
ENGINEERING
ON A BUDGET IS…

5. CHAOS
ENGINEERING
ON A BUDGET IS… POSSIBLE

6. CHAOS
ENGINEERING
ON A BUDGET IS… ESSENTIAL
IT WILL SAVE YOU TIME AND EFFORT

7. CHAOS
ENGINEERING
5
Systematic approach to building confidence in the
resiliency of distributed systems through triggering
chaotic operations

8. CHAOS
ENGINEERING
6
It’s not causing faults for the love of mayhem or to
freak out your team members

9. CHAOS
ENGINEERING
7
(although that’s
FUN, too)

10. CHAOS
ENGINEERING
8
Inject faults to learn about your system
Either you find bugs or you build confidence in
your system’s behavior

11. THE PROBLEM
WITH
DISTRIBUTED
SYSTEMS

12. 10
DISTRIBUTED
SYSTEMS

13. 11
DISTRIBUTED
SYSTEMS
Concretely, what does this mean?

14. Set of possible system states grows
EXPONENTIALLY
as sub-component count increases
PROBLEM
#1

15. Failure is NORMAL at scale
PROBLEM
#2

16. CASE STUDY #1
TOO MANY
SYSTEM STATES

17. AZURE SEARCH
MICROSOFT CLOUD AI PLATFORM

18. 16
BEFORE
CHAOS ENGINEERING
PARANOIA
ABOUT
SYSTEM
RELIABILITY

19. “
-Martin Fowler
[Y]ou are doing enough testing if the following is true:
YOU RARELY GET BUGS THAT ESCAPE INTO
PRODUCTION
YOU ARE RARELY HESITANT TO CHANGE SOME
CODE FOR FEAR IT WILL CAUSE PRODUCTION BUGS
17

20. “
-Martin Fowler
[Y]ou are doing enough testing if the following is true:
YOU RARELY GET BUGS THAT ESCAPE INTO
PRODUCTION
YOU ARE RARELY HESITANT TO CHANGE SOME
CODE FOR FEAR IT WILL CAUSE PRODUCTION BUGS
18

21. 19
BEFORE
CHAOS ENGINEERING
1 HOUR OF
CODE
=
3 HOURS OF
VALIDATION

22. 20
RIVER OF SCARY
SYSTEM BUGS
GAP IN
TEST
COVERAGE

23. 21
RIVER OF SCARY
SYSTEM BUGS
GAP IN
TEST
COVERAGE
UNSUSTAINABLE
HUMAN
MITIGATION

24. 22
RIVER OF SCARY
SYSTEM BUGS
GAP IN
TEST
COVERAGE
TEMPORARILY
SAFE CUSTOMERS
UNSUSTAINABLE
HUMAN
MITIGATION

25. Systems states to test had grown
EXPONENTIALLY
PROBLEM
#1
I was attempting (and failing) to scale my effort
EXPONENTIALLY

26. Test cluster in
production

27. Random Mutator
Test cluster in
production
Random
operations

28. Random Mutator
Test cluster in
production
Diagnostics
Random
operations
Regular
health
checks

29. 27
INCREDIBLY EFFECTIVE

30. 28
Runs MORE TESTS than I can
INCREDIBLY EFFECTIVE

31. 29
Runs MORE TESTS than I can
Finds MORE BUGS than I do
INCREDIBLY EFFECTIVE

32. 30
Runs MORE TESTS than I can
Finds MORE BUGS than I do
DOESN’T NEED TO SLEEP LIKE I DO
INCREDIBLY EFFECTIVE

33. 31
AFTER
CHAOS ENGINEERING
HIGH
CONFIDENCE
IN SYSTEM
QUALITY

34. 32
3 WEEKS OF WORK PRODUCED
3 YEARS OF TESTS

35. “
-Martin Fowler
[Y]ou are doing enough testing if the following is true:
YOU RARELY GET BUGS THAT ESCAPE INTO
PRODUCTION
YOU ARE RARELY HESITANT TO CHANGE SOME
CODE FOR FEAR IT WILL CAUSE PRODUCTION BUGS
33

36. “
-Martin Fowler
[Y]ou are doing enough testing if the following is true:
YOU RARELY GET BUGS THAT ESCAPE INTO
PRODUCTION
YOU ARE RARELY HESITANT TO CHANGE SOME
CODE FOR FEAR IT WILL CAUSE PRODUCTION BUGS
34

37. WHY IS
RANDOMNESS SO
EFFECTIVE?

38. CONFIDENCE
IN SYSTEM
HEALTH
36
Testing explores the n dimensional space
described by the set of all possible system
states and operations
x
y

39. CONFIDENCE
IN SYSTEM
HEALTH
37
Testing explores the n dimensional space
described by the set of all possible system
states and operations
x
y

40. CONFIDENCE
IN SYSTEM
HEALTH
38
Testing explores the n dimensional space
described by the set of all possible system
states and operations
x
y

41. CONFIDENCE
IN SYSTEM
HEALTH
39
Testing explores the n dimensional space
described by the set of all possible system
states and operations
x
y

42. CONFIDENCE
IN SYSTEM
HEALTH
40
As system complexity increases, the states
and operations that must potentially be
tested increase exponentially
• Curse of dimensionality
x
y
z

43. 41
TOO MANY
system states to test
(And correctly reducing the test set is too expensive)

44. 42
TOO MANY
system states to test
CONFIDENCE
(And correctly reducing the test set is too expensive)

45. 43
Reduces test set in an evenly distributed way
that avoids engineer bias and blind spots
Generating test cases is very CHEAP,
enabling more tests to be run
x
y
z
RANDOMNESS

46. MONTE CARLO
APPROACH TO
SYSTEM
CONFIDENCE
44
Question What’s the probability that my system
will encounter a bug?
Prohibitively expensive
approach
Pre-defining system states and operations
through engineer effort and validating them
Cheaper approximation Validating system health at regular intervals
Sample set Randomly generated system states +
transient failures from running in production

47. 45
Question What’s the probability that my system
will encounter a bug?
Prohibitively expensive
approach
Pre-defining system states and operations
through engineer effort and validating them
Cheaper approximation Validating system health at regular intervals
Customer-generated system states
from running in production
Sample set
“TRADITIONAL”
CHAOS ENGINEERING
MONTE CARLO
APPROACH TO
SYSTEM
CONFIDENCE

48. 46
TELEMETRY
is essential
x
y
z

49. DIGRESSION:
MACHINE
LEARNING AND
AUTOMATED
MITIGATION

50. CASE STUDY #2
FAILURE IS
INEVITABLE AT
SCALE

51. 49
THE 2 AM PHONE CALL
LEARNING LESSONS THE
HARD WAY

52. 50
Secondary failures caused by a bug in
ERROR RECOVERY LOGIC
LEARNING LESSONS THE
HARD WAY

53. 51
Resulted in multiple engineers fixing
clusters by hand ALL NIGHT
LEARNING LESSONS THE
HARD WAY

54. 52
Resulted in multiple engineers fixing
clusters by hand ALL NIGHT
LEARNING LESSONS THE
HARD WAY
(including my boss )

55. 53
WHAT HAPPENED!?

56. 54
HOW CAN YOU TRUST
ERROR RECOVERY CODE
IF IT NEVER RUNS?

57. 55
POSSIBLE
SYSTEM
STATES

58. 56
STATES
TESTED BY
CHAOS
COMPONENT
STATES
NOT
TESTED

59. 57
STATES
TESTED BY
CHAOS
COMPONENT
FAILURE
STATES

60. Failure is INEVITABLE at scale
PROBLEM
#2

61. Failure is INEVITABLE at scale
But not DETERMINISTICALLY
PROBLEM
#2

62. Failure is INEVITABLE at scale
But not DETERMINISTICALLY
PROBLEM
#2
(And not reliably within a sprint cycle)

63. 61
Promote FAILURE to a
FIRST-CLASSmember of the system

64. Random Mutator
Test cluster in
production
Diagnostics
Random
operations
Regular
health
checks

65. Random Mutator
Test cluster in
production
Diagnostics
Random
operations
Regular
health
checks
Chaos Monkey
Fault
injection

66. I don’t want to
get woken up at
2am again 

67. I don’t want to
get woken up at
2am again 
I validated 10
times this week
that you won’t!
Security blanket of
high confidence in
system health

69. 67
A/B TESTING

70. 68
INADVERTENT
A/B TESTING

71. 69
INADVERTENT
A/B TESTING
(I turned off the Chaos Monkey
and 2 SEVERE BUGS escaped
into production)

72. 70
INADVERTENT
A/B TESTING
(I turned off the Chaos Monkey
and 2 SEVERE BUGS escaped
into production)
(causing my boss to get called at 4am )

73. 71
(RCA of shame)

74. SUMMARY

75. Heather Nakama, QCon
11/14/17
CHAOS ENGINEERING HAS INCREASED MY
CONFIDENCE IN THE RELIABILITY OF MY
CODE WHILE COSTING LESS IN TIME AND
RESOURCES

76. Heather Nakama, QCon
11/14/17
CHAOS
ENGINEERING
PROVIDES HIGH CONFIDENCE
AT LOW COST

77. Heather Nakama, QCon
11/14/17
DISTRIBUTED SYSTEMS ARE CHAOTIC AND
REQUIRE CHAOTIC TESTING METHODS
MONTE CARLO-STYLE RANDOM SAMPLING IS
A CHEAP WAY TO GAIN CONFIDENCE
ABOUT SYSTEM HEALTH
…AS LONG AS YOU INCLUDE FAILURE IN
YOUR SAMPLE SET

78. Heather Nakama, QCon
11/14/17
AZURE SEARCH
• Inside Azure Search: Chaos Engineering
• A Startup At Microsoft
HEATHER NAKAMA
• Distributed systems
• Chaos engineering
• Self-healing systems through machine learning
• Biologically inspired systems
Art by Designed by Vexels.com

79. Heather Nakama, QCon
11/14/17
QUESTIONS?

80. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
chaos-engineering-budget