Netflix talk at the Kaizen Data Conference

1. Data Over Matter
|Eli Bressert
Manager, Data Engineering & Analytics
Innovating the next
generation of products

2. today's talk
Evolution of Netflix
Data at Netflix
Data over matter

3. evolution of Netflix

4. 1999 DVD Service
2007 Streaming Service
2012 Netflix Originals
2016 Netflix Global

5. 1999 DVD Service
2007 Streaming Service
2012 Netflix Originals
2016 Netflix Global

7. 1999 DVD Service
2007 Streaming Service
2012 Netflix Originals
2016 Netflix Global
2012 2013 2014 2015 2016

8. 1999 DVD Service
2007 Streaming Service
2012 Netflix Originals
2016 Netflix Global

9. 1999 DVD Service
2007 Streaming Service
2012 Netflix Originals
2016 Netflix Global

10. data @ Netflix

12. data
over
matter

13. experimentation
not opinion

14. experimentation
not opinion

15. user experience
in the UI

16. 2011

17. 2013

18. image optimization

20. a/b testing
@ Netflix

21. Discovery Algorithms
Design/UI
Product Functionality
Messaging
Streaming qualityM
NM
Sign-up flow
Messaging
Marketing

22. Other
Netflix
Services
architecture overview

23. Ignite org structure
A/B test data from multiple teams
common UI layer & shared resources
A/B test visualization is decoupled from
A/B test data processing layer

24. additional support
BYOD (Bring Your Own Data)
Internal Engineers
3rd party data
All tests
Key: Decoupling A/B test UI from Data
processing layer(s)

25. Analyst manually creates table with list
of users ahead of time
A/B system simply allocates from this
pool of usersB
RT
User participation criteria are
defined in UI
Allocate on the fly if a user
meets those criteria

28. Ignite: the winner is ...
Statistical significance + simple highlighting
gives “at a glance” read for test cells
Statistical significance values are calculated
by the UI so switching the control cell does
not require a new query
Druid backend allows interactive query
speeds on massive datasets with high
cardinality dimensions
UI “falls back” to Presto when the
complexity of the query grows beyond
simple OLAP
Full stack javascript: Node.js + React

30. nearly done

31. start banking some good
questions :)
nearly done

32. a/b testing to ∞?

33. a/b testing to ∞?

34. enabling partners

35. analytics
self-service
tools
a/b testing
maximizing value
of data for product

36. ?
ways to reach me/us
@astrobiased
ebressert at netflix.com
@netflixdata