Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2AwAlb9. Lavanya Kanchanapalli shares her experience in maintaining a great Netflix customer experience while enabling fast and safe data propagation. She also talks about detecting and preventing bad data that is essential to high availability, ways to make circuit breakers, data canaries and staggered rollout effective, and efficient validations via sharing data and isolating change. Filmed at qconsf.com. Lavanya Kanchanapalli works as a Senior Software Engineer at Netflix.

1. Crisis to Calm: A Story of Data
Validation @ Netflix
Lavanya Kanchanapalli

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
data-validation-netflix

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

5. ● Rollback data
● Increase capacity
● Netflix starts to work

6. ● Duplicate objects
Root Cause
● Apps failed
● Cascading failures
● Netflix goes down!

10. Data Change
Change in Behavior
Netflix Microservices
App1
Cloud
App2 Appn

14. Metadata Architecture
Video
Metadata
Service
Amazon S3
Source
System
Source
System
Netflix
Services
Netflix
Services
Netflix
Services
Netflix
ServicesNetflix
Service
Traffic

15. ● Single publisher
● Multiple consumers
● Hollow (hollow.how)
● Versioned data
● Fast propagation

16. Bad data happens!

17. Leaked Content

18. Disables Features

19. Deletes data

21. Data change = Code Push

22. 1
Detection Staggering
2
Rollback
3
Staggering
2
Rollback
3

23. 1
Detection Staggering
2
Rollback
3
Rollback
3

24. 1
Detection Staggering
2
Rollback
3

25. Many Unknowns

26. Would it be too slow?

27. Would there be too many failures?

28. Would it cost too much?

29. 1
Detection Staggering
2
Rollback
3

30. 1
Detection Staggering
2
Rollback
3
Staggering
2
Rollback
3

31. Chapter 1: Circuit Breakers

32. Metadata Architecture
Video
Metadata
Service
Amazon S3
Source
System
Source
System
Netflix
Services
Netflix
Services
Netflix
Services
Netflix
ServicesNetflix
Service
Traffic

33. ● Integrity checks
● Duplicate detection
● Object counts
● Semantic checks
Circuit Breakers

34. Know your data change

35. Knobs are key to sanity
● On/ off
● Threshold
● Exclusions

38. Business value is the key

40. Efficiency
● Change Isolation
● Sampling

41. Chapter 2: Canaries

42. Traditional Canaries
Canary
(New Code)
Baseline
(Old Code)
Shadow
Traffic
Shadow
Traffic
Video
Metadata
Service
Amazon S3
Netflix
Services
Netflix
Services
Netflix
Services
Netflix
ServicesNetflix
Service
Source
System
Source
System
Traffic

43. Data Canaries
Netflix
Services
Netflix
Services
Netflix
Services
Netflix
ServicesVideo
Metadata
Service
Amazon S3
Source
System
Source
System
Netflix
Service
Netflix Data
Canary
Service
Netflix
Service
Traffic

44. Netflix Data Canary Service
● Pick key use case(s)
● Pick data to test
● Test with latest data
Netflix ServiceNetflix Data
Canary
Service

45. 1
Detection Staggering
2
RollbackRollback
3

46. Amazon WS Global Infrastructure
STAGGERED ROLLOUT

47. 1
Detection Staggering
2
Rollback
3

48. Keep calm & rollback
● Pin back
● Root cause
● Unpin

49. Rollback
● Visibility
● Traversing

50. Data Diff UI

51. Circuit Breaker UI

52. Pinning UI

55. Data validation is key to high availability
● Data change = Code push
● Circuit breakers & canaries
● Staggering and rollback

56. Thank You

57. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
data-validation-netflix