The Netflix experience is driven by a number of Machine Learning algorithms: personalized ranking, page generation, search, similarity, ratings, etc. On the 6th of January, we simultaneously launched Netflix in 130 new countries around the world, which brings the total to over 190 countries. Preparing for such a rapid expansion while ensuring each algorithm was ready to work seamlessly created new challenges for our recommendation and search teams. In this post, we highlight the four most interesting challenges we’ve encountered in making our algorithms operate globally and, most importantly, how this improved our ability to connect members worldwide with stories they'll love.

2. Recommending for the World
Yves Raimond (@moustaki)
03/16
Research/Engineering Manager
Search & Recommendations Algorithm Engineering
Netflix

3. Some background

7. ● > 75M members
● > 190 countries
● > 3.7B hours of content streamed every
month
● > 1000 device types
● 36% of peak US downstream traffic
Netflix scale

8. Recommendations @ Netflix

9. Goal
Help members find content to watch and enjoy
to maximize satisfaction and retention

19. ▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪ …
Models & Algorithms

21. Going global
● How do we make sure all these
algorithms are ready to work on a
global scale?
● Led us to investigate many
challenges, leading to many rollouts
of new algorithms, over the last year
○ Tech blog post
○ Company blog post

22. Challenge 1: Uneven Video Availability

23. US
FR

24. US
FR
1,000 users
100 users

25. 0
...
...Co-occurrences

26. ! =
?????

27. R ≈ UM

28. ! =

29. What would have happened if the two
videos were available to the same
members?

30. US
FR
1,000 users
100 users
100,000 users
10 users

31. ≈

32. US
FR
2016-01-01 2016-01-02
Newly available

33. What would have happened if the two
videos were available to the same
members for the same amount of time?

34. Challenge 2: Cultural Awareness

35. Two questions

36. 1) Similar users, in two different countries. Should they get similar
recommendations?

37. 2) Overall, should recommendations be different for users in Japan vs users
in Argentina? What about new users?

38. Regional models
Group countries into regions, and train
individual models on each region.
Pros
● Easy!
● Catalog can be constrained to be
relatively uniform
● Solves question 2
Cons
● Doesn’t solve question 1
● How to define groupings?
● Algorithms x A/B model variants x
regions
● Biggest country in the region will
dominate
● Sparsity

39. Sparsity and global models
Only a small fraction of users from all countries
would be interested in these titles. Models trained
locally perform poorly -- lack of data.
Pooling data from all countries discovers a
worldwide community of interest, making
recommendations better for these users.

40. Global communities - Anime

41. Global communities - Bollywood

42. Local taste vs personal taste
● Personal taste benefits from global algorithms
○ Taste patterns travel globally
● Local taste still needs to be taken into account in order to solve 2)
● Incorporate signals and priors capturing local taste patterns (e.g. country and
language)

44. Challenge 3: Language

45. Instant search
● Ranking entities for partial queries
● Optimizing for the minimum number of interactions needed to find something
● Different languages involve very different interaction patterns
● How to automatically detect and adapt to such patterns in newly introduced
languages?

46. Hangul alphabet, 3 syllables but
requires 7 (2 + 3 + 2) interactions

47. One interaction

48. Language & Recommendations
≈+
US US/AU FR
?

49. Challenge 4: Does it even work?

50. Tracking quality
● Objective: build algorithms that work equally well for all our members
● Looking at global metrics might hide issues with small subsets of members
● How to identify sub-optimality for a subset of our members?
○ Language, country, device, …
○ Slicing on all dimensions lead to sparsity and noisiness
○ Automatically grouping observations for the purpose of automatically detecting outliers
● Metrics, instrumentation and monitoring
○ Detect problems
○ Highlight areas of improvement

51. Conclusion

52. ● Catalog differences, cultural awareness, language and metrics
● Worldwide communities of interest for better recommendations
○ Thinking about global actually led us to test and release better algorithms
○ But also need to capture signals and priors related to cultural preferences
● Quickly finding entities in any language
● Detecting issues at a finer grain
● … Still a lot of work to do!
○ Better global algorithms… (Now that we have data)
○ Better cultural/language awareness
○ Better user and item cold start
○ Reactiveness
○ Better algorithms for anomaly detection
Conclusion

53. Questions?