Netflix was a trailblazing innovator in machine learning as applied to personalization and recommendation systems but there are many other applications of machine learning at Netflix, especially as we further evolve into a global entertainment company. This talk will give an overview of how machine learning is leveraged before content launches on Netflix and how machine learning can support the creative process and serve as a tool for decision makers in our content and marketing organization. The process of creating content is a high-touch, creative endeavor so we need to be similarly creative in the machine learning innovations we develop. From neural nets that predict audience size for content that doesn't exist yet, to NLP and deep learning techniques that mine scripts to highlight properties we need legal clearance for ... we are building unprecedented innovations. The talk will also broadly cover the challenges we face in this space, including data scarcity and making ML interpretable for non-technical stakeholders.
2. Today’s talk - key theme
ML to support decisions and
decision-makers, not ML to automate
decisions
3. ● Not comprehensive - highlight examples
● Not technical - highlight application
4. The team …
Machine learning research scientists, mostly LA-based
Build and own a diverse ecosystem of “ML products”
● Including “demand (audience size) modeling as a service”
In addition, we heavily utilize causal inference, analytics, etc.
5.
6.
7. (Netflix Original) Title Lifecycle
LAUNCHLOCALIZATION
& QC
BUSINESS
NEGOTIATIONS
CREATIVE
PITCH
POST
PRODUCTIONPRODUCTION
PRE-
PRODUCTION
PRE-LAUNCH
PROMOTION
PRE-GREENLIGHT POST-GREENLIGHT PRE-LAUNCH POST-LAUNCH
8. ML to support decisions and
decision-makers, not ML to automate
decisions
… why?
13. Verticals are a building block
How do we use data & data science to identify
strengths and weaknesses in our entire catalog?
Today and in 2 years
Multiple objectives and constraints
ML + Operations Research
19. (Netflix Original) Title Lifecycle
LAUNCHLOCALIZATION
& QC
BUSINESS
NEGOTIATIONS
CREATIVE
PITCH
POST
PRODUCTIONPRODUCTION
PRE-
PRODUCTION
PRE-LAUNCH
PROMOTION
PRE-GREENLIGHT POST-GREENLIGHT PRE-LAUNCH POST-LAUNCH
Marketing
Budgets
20. Incrementality: The Causal Effect of an Ad
Example from “Ghost Ads”:
Sporting goods retailer who
ran an experiment:
● Retargeting
● 570k users
● 2 weeks
● 9 million impressions
● Ad spend: $30,500
● Avg. CPM = $3.40
Incrementality: The
difference in the outcome
because the ad was shown;
the causal effect of the ad.
Per impression:
$100k/9M=$0.011 ⇒ $11 RPM
Johnson, Garrett A. and Lewis, Randall A. and Nubbemeyer, Elmar I, Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness (January 12, 2017). Simon Business
School Working Paper No. FR 15-21. Available at SSRN: https://ssrn.com/abstract=2620078
$1.1M
$1.0M
$100k
Other
Advertisers’
Ads
Control Treatment
Revenue
Random Assignment
21. Causal inference for marketing more broadly
When we can’t run A/B Tests, we run
quasi-experiments
22. The vision is causal inference
“layers” on top of the predictions
Can we adapt concepts we’ve
pioneered in the advertising space?
23. 1) Opportunity detection
2) Resource allocation
3) Optimization
○ Giving our titles the best chance to
succeed
37. Conclusion …
● We are pursuing any and all data science tools if they help us find
opportunities, provide insights to decision-makers, and improve
the chance our titles will succeed
● Art and Science coming together in unprecedented ways
● “Democratizing” ML / data science as stakeholder set expands
38. Shameless plug …
● We are looking for
○ ML experts
○ OR experts
○ PMs
○ Leaders for senior roles (managing a research team)
kuphoff@netflix.com
Thank you!
