A keynote at RecSys 2014: The Value of Better Recommendations - For Business, Consumer, Producer, and Society. A story, told from the Netflix perspective, of Internet TV and how recommendations systems enable the long tail, improve economics, and spread a global culture, with thoughts on objective metrics, measurement techniques, AB testing.

1. Quantifying the Value of
Better Recommendations

2. The Value of Better Recommendations
Stakeholders:
■ Business value
■ Consumer value
■ Producer value
■ Cultural and Societal value
Recommenders have power for great benefit,
but also for harm
Use your power wisely!

3. Neil Hunt
1999: Started work on recommendations at Netflix
Goal was to improve satisfaction, while solving the %New problem
2006-2009: Netflix Prize
Public recognition of the importance of recommenders
2007-2011: Transition to streaming
Complete catalog, short supply → curated catalog, unlimited supply
2014: 300 people working on “content discovery”
$150M investment

4. Context

5. Technology Eliminates
Constraints on Personal Choice
Constraints on Personal Choice Falling Away
■ Geographic “trade radius”
■ Production / manufacturing
■ Shelf-space
■ Channel (TV, radio)
Recommenders allow access to the long-tail of choices:
■ Discovery
■ Evaluation

6. Recommenders Enable Long Tail Media
■ There are no bad shows,
just shows with small audiences
■ It’s our job to find and motivate
exactly the right audience

7. Linear TV Channel - one choice available
■ Only watch what’s being broadcast
■ 21 hours/week of prime time - nothing else matters
On-demand - unlimited catalog accessible instantly
■ Paradox of choice:
1000s of possibilities, most not interesting
Need a custom channel for each user (50M channels):
■ 20-50 personalized choices
Netflix - TV of the Future - 50M Channels

8. No More Commercials
… or only relevant and interesting ads
(chosen by recommenders…)

9. Richer Storytelling
Freed of the constraints of linear TV,
not all shows must be 42 minutes with a cliffhanger end
Discovery from outside a channel grid liberates the format
The same was true for novels in Dicken’s time:
Pickwick Papers was published in 20 weekly magazines
with 32 pages of text (a 4-fold broadsheet)
and 16 pages of advertising support
They too, were liberated by advances in technology - making books possible

10. Business Objectives

11. Why Do Businesses Invest in Recommenders?
Better Economics…
■ Makes a traditional business better, or
(Netflix, Amazon, Spotify, Pandora, ...)
■ Enables new businesses not possible before
(LinkedIn, GoogleNews, Instagram, Waze, Pinterest, any free service with
ads, …)

12. Why Do Businesses Invest in Recommenders?
The Tension:
■ Enhancing customer satisfaction
■ Better choices
■ Shorter time to choose
■ Suggesting more profitable products
■ Choices with better margins
■ Advertising
■ Long-term vs. Short-term tradeoff?

13. Netflix Choices
All our content is licensed to a fixed fee:
Each possible choice has same cost impact
We don’t sell advertising on our service. Never will.
We don’t sell our recs or data to third parties in any form.
For Netflix, it’s all about customer satisfaction

14. Quantifying Netflix Benefits
7B hours per quarter
50M subscribers worldwide
90 minutes/day average
150M choices/day

15. Quantifying Netflix Benefits
A good choice leads to a complete viewing
A poor choice leads to abandonment, and risk of cancel
10% “better” choices → +500M/month good outcomes
If 1% of those avoids a cancellation → $500M/year
Our measurement thresholds:
0.1% retention improvement ($5..50M/year)
0.1% more viewing per time period

16. Our Business Metrics
Business Value
New Trials Retention
Hours of ViewingWord of Mouth

17. Retention is a Blunt Measuring Instrument

18. Retention is a Blunt Measuring Instrument

19. Measuring Users-at-Threshold
Hours of Viewing
Frequency
Medians Averages
Baseline

20. Measuring Users-at-Threshold
Hours of Viewing
Frequency
Medians Averages
Baseline Higher Avg

21. Measuring Users-at-Threshold
Hours of Viewing
Frequency
Medians Averages
BaselineHigher Median Higher Avg

22. But We’re Still Measuring the Wrong Thing...
We optimize hours of viewing…
But all hours are not created equal
Implication:
■ We machine-learn addictive over compelling
■ Partly innoculated by also measuring retention
What signal can we find for valued hours?

23. What if the Retention Driver is Something
Else?
Avoiding Failed Sessions (user found nothing to watch)
Reducing Time-to-Play
Maximizing fraction-of-content-viewed
Maximizing velocity of episode consumption

24. Consumer Objectives

25. What Consumers Say...
“I don’t need suggestions, just show me the good stuff”
“Don’t hide anything - I want to evaluate it all”

26. Winning The Moment of Truth
■ Moment of truth
■ 1-2 minutes to find something
■ 20-50 chances to connect
■ Or the user has moved on...

27. The Intrinsic/Social Spectrum

28. Oracle vs. Advisor
✗ ✓

29. Content Producer Implications

30. The Cliff of Conventional Media
Producers must aim for broad
audience or be irrelevant
Target Audience
Consumption
Too small
No-one knows
No-one cares
Consum
ption
m
atches
target audience

31. Recommender Systems Level The Cliff
Economics of high-end
producers is less exponential
Producers can target the
audience of their choice
New producers with niche
product can emerge
Target Audience
Consumption
Even small audiences
can be engaged
Consum
ption
m
atches
target audience

32. Recommender Systems Level The Cliff
Long-tail producers aren’t excluded
Much greater cultural diversity is enabled

33. Does Data Drive the Product?
■
■
■
■
■
■

34. Netflix Use of Data for Content
✓ Predict reach & hours for a project given what we know
? Give insight to choice of cast, location, etc. if requested
✗ DO NOT dictate “she has to die at the end of S2-E1”
The director’s choices matter!

35. Cultural and Societal Implications

36. Democratization of Media
The cultural implication of the media cliff is lack of access
to less prominent voices, channels, products
Recommendations systems can provide the market.
Producers are stepping in to fill that niche

37. The Cultural Exception
■ Marketing economics drives large commercial culture
to displace local, regional, niche culture
■ France: l’exception culturelle under GATT
■ Canada: cultural exemption under NAFTA
■ Recommendation systems can reduce the swamping
effect of large commercial culture
■ More to gain by exporting French culture to the world
than by limiting import of global culture to France
Protectionism can yield to multiculturalism

38. Filter Bubbles and Echo Chambers
Proposition:
■ Recommendation systems reinforce existing taste,
don’t expose users to the new, unexpected or different
If this keeps users happy, it’s likely to be true
Our experience is that diversity and serendipity play a
large role in delivering recommendations that win
?

39. Final Thoughts

40. We are just scratching the surface of what’s possible

41. We are just scratching the surface of what’s possible
We depend upon our users trusting us with their data
-- they might lose that trust

42. We are just scratching the surface of what’s possible
We depend upon our users trusting us with their data
-- they might lose that trust
We have the ability to do amazing things for culture
or distort it horribly by following a false north-star

43. We are just scratching the surface of what’s possible
We depend upon our users trusting us with their data
-- they might lose that trust
We have the ability to do amazing things for culture
or distort it horribly by following a false north-star
Be creative, but humble, and amaze the world!