As a data driven company, we use Machine learning based algos and A/B tests to drive all of the content recommendations for our members. Traditionally, these recommendations are precomputed in a batch processing fashion, but such a model cannot react quickly based on member interactions, title interests, popularity etc. With an ever-growing Netflix catalog, finding the right content for our audience in near real-time would provide the best personalized experience. We’ll take a deep dive into our realtime Spark Streaming ecosystem at Netflix. Both it’s infrastructure and business use cases. On the infrastructure front, we will delve into scale challenges, state management, data persistence, resiliency considerations, metrics, operations and auto-remediation. We will talk about a few use cases that leverage real-time data for model training, such as providing the right personalized videos in a member’s Billboard and choosing the right personalized image soon after the launch of the show. We will also reflect on the lessons learnt while building such high volume infrastructure.

1. #ML7SAIS
Elliot Chow, Netflix
Nitin Sharma, Netflix
Near Real-Time Recommendations - Spark
Streaming
#ML7SAIS

2. #ML7SAIS
Agenda
● Recommendations @ Netflix
● The Need for Near Real Time
● Use Cases
● Common Infrastructure
● Scaling Challenges

3. #ML7SAIS
Recommendations at Netflix
● Personalize the Netflix experience for each member
○ Goal: Quickly help members find content they’d like to
watch
○ Risk: Member may lose interest and abandon the service
○ Challenge: Recommending at scale

4. #ML7SAIS
Scale @ Netflix
● 125M+ active members
● 190 countries
● 450B+ unique events/day
● 700+ Kafka topics

5. #ML7SAIS
● Data stored in Hive/S3
● Batch ETLs using Spark/Hive
● Table partitioning by day or hour
● Job scheduling by both CRON or data availability
● Latency often is on the order of days
Typical Data Pipelines @ Netflix

6. #ML7SAIS
● Dynamic catalog
● Growing member base
● Time sensitivity
○ Content popularity changes
○ Member interests evolve
The Need for Near Real Time (NRT)

7. #ML7SAIS
● Increasing amount of data
○ Process data as soon as possible to keep latencies low
○ Minimize amount of data to reprocess in case of failure
● Multi-Armed Bandits Adoption
The Need for Near Real Time (NRT)

8. #ML7SAIS
Use Cases
● Video Insights
● ML Pipelines for Recommendations

9. #ML7SAIS
NRT for Video
Insights

10. #ML7SAIS
Video Insights
● New title launches
● Early reads on title performance
● Slice by various dimensions

11. #ML7SAIS
Video Insights
Service

12. #ML7SAIS
Video Insights
Service

13. #ML7SAIS
Video Insights
Service

14. #ML7SAIS
Video Insights
Service

15. #ML7SAIS
Video Insights
Service
Client

16. #ML7SAIS
Video Insights - State
● Counts maintained in Spark
● Custom state management
○ Based on mapWithState implementation
○ Easier to re-use the function f in batch mode
○ Lower-level control over state management
○ scanByKey alternative for keyed state
input.scan(initRDD)((currentRDD, batchRDD) => f(currentRDD, batchRDD))

17. #ML7SAIS
NRT for Recommendations

18. #ML7SAIS
Billboard Recommendations
● Recommend a relevant title to each
member
● Right time
● Respond quickly to member feedback

19. #ML7SAIS
Artwork Personalization
19
● Personalized Image
● Visual Evidence
● Quickly adapting - Title
launches, member tastes
● Rapid learning - Cold start

20. #ML7SAIS
Traditional Recommendations
Millions of
Play related
Signals
Training
pipelines
Models
Precompute/Live
Compute
Rankings
DataStore/
Online
caches
ETL
Layers
Few days

21. #ML7SAIS
NRT Recommendations
Millions of
Play related
Signals
Models
Precompute/Live
Compute
Rankings
DataStore/Online
caches
Training
pipelines
Streaming
layer
NRT

22. #ML7SAIS
Required Data
● Impressions, Plays, etc.
● Attribution
● Explore/Exploit Metadata

23. #ML7SAIS
Attribution Infrastructure
Stream Processing Batch Processing
Query API

24. #ML7SAIS
Stream Processing - Zoomed in
Impression
Cache
Video
Metadata
Play
Enrich

25. #ML7SAIS
Batch Processing - Zoomed in
Impression
Play
Experimentation
Data
Windowed
Impression
Windowed
Play
Dedupe
Join
Video
Metadata

26. #ML7SAIS
Common Infrastructure

27. #ML7SAIS
Netflix Spark Stack
● Jenkins
● Spinnaker
● ASG
● Runners: Marathon, Meson
● Resource Manager: Mesos
● Storage: HDFS, S3, EFS
● Multi-Region

28. #ML7SAIS
Multi Region Challenges
● Geo routing
● Impression in one region;
play in another
● Streaming - Multi Region
● Batch Reduce/Merge - One
region

29. #ML7SAIS
Can it withstand Chaos?
• Chaos is a design principle
• Instance Failovers => Region Failovers
• Transparent to the consumers
• Over provisioned
• Long term - Autoscale

30. #ML7SAIS
When Things Go South
● What if something doesn’t look right?
○ Stream Processing is stuck
○ Driver/Executor failures
○ Intermittent issues with external dependencies
● Metrics - Spark metrics to Atlas (similar to RRDTool + Graphite)
● Getting paged at 2 am - Not fun :) !
● Need for auto-remediation - less operational overhead

31. #ML7SAIS
Auto Remediation Infrastructure
SQS Auto Remediation
De-queue
Triggers
Metadata

32. #ML7SAIS
• Scalability Performance
tuning
– Micro batch interval
– Memory Tuning
– Parallelism/Shuffle
tradeoff
• Data quality issues
– Low latency vs data
quality
Streaming Challenges