Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1RMrvo0. The authors present their experience in collaboration between industry and academia, describing how a “big idea” -- lineage-driven fault injection -- evolved from a theoretical model into an automated failure testing system that leverages Netflix’s state-of-the-art fault injection and tracing infrastructures. Filmed at qconlondon.com. Kolton Andrus is the founder of Gremlin Inc. He is passionate about building resilient systems, primarily as it lets him break things for fun and profit. Peter Alvaro is an Assistant Professor of Computer Science at the University of California Santa Cruz. He is the creator of the Dedalus language and co-creator of the Bloom language.

1. Monkeys in Lab Coats
Applied Failure Testing Research at

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/failure-test-research-netflix

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. The whole is greater than the sum of its parts.
- Aristotle
[Metaphysics]

5. The Professor vs The Practitioner
Peter Alvaro
Ex-Berkeley, Ex-Industry
Assistant Prof @ Santa Cruz
Misses the calm of PhD life
Likes prototyping stuff
Kolton Andrus
Ex-Netflix, Ex-Amazon
‘Chaos’ Engineer
Misses his actual pager
Likes breaking stuff

6. Measures of Success
Academic
H-Index
Grant warchest
Department ranking
Industry
Availability (i.e. 99.99% uptime)
Number of Incidents
Reduce Operational Burden

7. An Unlikely Team?

9. but ... it’s manual
Works Great!

10. Surely there is a better way ...

12. Free lunch?

13. The End?
(Academia + Industry)

14. Let’s build it
“Can we, pretty please?”

15. Freedom and Responsibility
Core Value

16. Responsibility
Academic Industry
Prove that it works
Show that it scales
Find real bugs

17. The Big Idea Lineage Driven
Fault Injection

18. What could possibly go wrong?
Consider computation
involving 100 services
Search Space:
2100
executions

19. “Depth” of bugs
Single Faults Search Space:
100 executions

20. “Depth” of bugs
Combination of 4 faults Search Space:
3M executions

21. “Depth” of bugs
Combination of 7 faults Search Space:
16B executions

22. Random Search
Search Space:
2100
executions

23. Engineer-guided Search
Search Space:
???

24. Fault-tolerance “is just” redundancy

25. But how do we know redundancy when we see it?
Hard question: “Could a bad thing ever happen?”
Easier: “Exactly why did a good thing happen?”
“What could have gone wrong?”

26. Lineage-driven fault injection
Why did a good thing happen?
Consider its lineage.
What could have gone wrong?
Faults are cuts in the lineage graph.
Is there a cut that breaks all supports?
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1 Bcast2
Client Client

27. Lineage-driven fault injection
Why did a good thing happen?
Consider its lineage.
What could have gone wrong?
Faults are cuts in the lineage graph.
Is there a cut that breaks all supports?
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1 Bcast2
Client Client

28. What would have to go wrong?
(RepA OR Bcast1)
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast2
Client Client
Bcast1

29. What would have to go wrong?
(RepA OR Bcast1)
AND (RepA OR Bcast2)
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1 Bcast2
Client Client

30. What would have to go wrong?
(RepA OR Bcast1)
AND (RepA OR Bcast2)
AND (RepB OR Bcast2)
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1
Client Client
Bcast2

31. What would have to go wrong?
(RepA OR Bcast1)
AND (RepA OR Bcast2)
AND (RepB OR Bcast2)
AND (RepB OR Bcast1)
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1 Bcast2
Client Client

32. Search Space Reduction
Each Experiment finds
a bug, OR
Reduces the
Search space

33. The prototype system “Molly”
Recipe:
1. Start with a successful
outcome. Work backwards.
2. Ask why it happened: Lineage
3. Convert lineage to a boolean
formula and solve
4. Lather, rinse, repeat
2. Lineage 3. CNF
Fail1. Success
Why?
Encode
Solve
4. REPEAT

34. The Big Idea Meets Production

35. 1. Start with a successful outcome
2.
Lineage
3.
CNF
Fail1. Success
Why?
Encode
Solve
4. REPEAT

36. What is success?

38. “Start with the customer and work
backwards”
Leadership Principle

41. “Streaming” Data

42. Joining the Streams

43. Missing Data?

44. Lesson 1
Work backwards from what you know

45. 2. Ask why it happened
2.
Lineage
3.
CNF
Fail1. Success
Why?
Encode
Solve
4. REPEAT

46. Request Tracing

48. Request Tracing

49. Alternate Execution

50. Evolution over time

51. Redundancy through History

52. Lesson 2
Meet in the middle

53. 3. Solve
2.
Lineage
3.
CNF
Fail1. Success
Why?
Encode
Solve
4. REPEAT

54. A “small” matter of code

55. 4. Lather, Rinse, Repeat
2.
Lineage
3.
CNF
Fail1. Success
Why?
Encode
Solve
4. REPEAT

56. Turn the crank, right?

57. Idempotence

60. Bins and Balls
Request
Class 1
Class 2
Class 3
Class n
[...]
r’ r

61. Class n
Predicting Request Graphs
Request

62. Class n
Predicting Request Graphs
Request
Some function f:
Requests → Classes

63. F( ) =
Class n
Request
Predicting Request Graphs

65. Solve the Machine Learning problem?
or the Failure Testing one?

66. Simplest thing that will work?

67. ["bookmarks”, “recent”]
["playlist", 0, “name”]
["ratings"]
Falcor Path Mapping
=>
“bookmarks,playlist,ratings”

68. Lesson 3
Adapt the theory to the reality

69. Many moons passed...

70. Does it work? YES!

71. Case study: “Netflix AppBoot”
Services ~100
Search space (executions) 2100
(1,000,000,000,000,000,000,000,000,000,000)
Experiments performed 200
Critical bugs found 6

72. Future Work
Richer device metrics
Request class creation
Better experiment selection
Search prioritization
Richer lineage collection
Exploring temporal
interleavings

73. Lessons
Work backwards from what you know
Meet in the middle
Adapt the theory to the reality

74. Academia + Industry

75. Academia + Industry
Academia Industry

76. Thank You!
Peter Alvaro
@palvaro
palvaro@ucsc.edu
Kolton Andrus
@KoltonAndrus
kolton@gremlininc.com

77. References
● Netflix Blog on ‘Automated Failure Testing’ http://techblog.netflix.
com/2016/01/automated-failure-testing.html
● Netflix Blog on ‘Failure Injection Testing’ techblog.netflix.com/2014/10/fit-
failure-injection-testing.html
● ‘Lineage Driven Fault Injection’
http://people.ucsc.edu/~palvaro/molly.pdf

78. Photo Credits
● http://etc.usf.edu/clipart/4000/4048/children_7_lg.gif
● http://cdn.c.photoshelter.com/img-
get2/I0000MIN8fL0q8AA/fit=1000x750/taiwan-hiking-river-tracing-walking.jpg
● http://i.imgur.com/iWKad22.jpg
● https://blogs.endjin.com/2014/05/event-stream-manipulation-using-rx-part-2/
● http://youpivot.com/category/features/
● https://www.cloudave.com/33427/boards-need-evolve-time/
● https://www.linkedin.com/pulse/amelia-packager-missing-data-imputation-
ramprakash-veluchamy

79. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/failure-
test-research-netflix