RealTime Recommendations @Netflix - Spark Streaming

1. #ML7SAIS
Elliot Chow, Netflix
Nitin Sharma, Netflix
Near Real-Time Recommendations - Spark
Streaming
#ML7SAIS

2. #ML7SAIS
Agenda
● Recommendations @ Netflix
● The Need for Near Real Time
● Use Cases
● Common Infrastructure
● Scaling Challenges

3. #ML7SAIS
Recommendations at Netflix
● Personalize the Netflix experience for each member
○ Goal: Quickly help members find content they’d like to
watch
○ Risk: Member may lose interest and abandon the service
○ Challenge: Recommending at scale

4. #ML7SAIS
Scale @ Netflix
● 125M+ active members
● 190 countries
● 450B+ unique events/day
● 700+ Kafka topics

5. #ML7SAIS
● Data stored in Hive/S3
● Batch ETLs using Spark/Hive
● Table partitioning by day or hour
● Job scheduling by both CRON or data availability
● Latency often is on the order of days
Typical Data Pipelines @ Netflix

6. #ML7SAIS
● Dynamic catalog
● Growing member base
● Time sensitivity
○ Content popularity changes
○ Member interests evolve
The Need for Near Real Time (NRT)

7. #ML7SAIS
● Increasing amount of data
○ Process data as soon as possible to keep latencies low
○ Minimize amount of data to reprocess in case of failure
● Multi-Armed Bandits Adoption
The Need for Near Real Time (NRT)

8. #ML7SAIS
Use Cases
● Video Insights
● ML Pipelines for Recommendations

9. #ML7SAIS
NRT for Video
Insights

10. #ML7SAIS
Video Insights
● New title launches
● Early reads on title performance
● Slice by various dimensions

11. #ML7SAIS
Video Insights
Service

12. #ML7SAIS
Video Insights
Service

13. #ML7SAIS
Video Insights
Service

14. #ML7SAIS
Video Insights
Service

15. #ML7SAIS
Video Insights
Service
Client

16. #ML7SAIS
Video Insights - State
● Counts maintained in Spark
● Custom state management
○ Based on mapWithState implementation
○ Easier to re-use the function f in batch mode
○ Lower-level control over state management
○ scanByKey alternative for keyed state
input.scan(initRDD)((currentRDD, batchRDD) => f(currentRDD, batchRDD))

17. #ML7SAIS
NRT for Recommendations

18. #ML7SAIS
Billboard Recommendations
● Recommend a relevant title to each
member
● Right time
● Respond quickly to member feedback

19. #ML7SAIS
Artwork Personalization
19
● Personalized Image
● Visual Evidence
● Quickly adapting - Title
launches, member tastes
● Rapid learning - Cold start

20. #ML7SAIS
Traditional Recommendations
Millions of
Play related
Signals
Training
pipelines
Models
Precompute/Live
Compute
Rankings
DataStore/
Online
caches
ETL
Layers
Few days

21. #ML7SAIS
NRT Recommendations
Millions of
Play related
Signals
Models
Precompute/Live
Compute
Rankings
DataStore/Online
caches
Training
pipelines
Streaming
layer
NRT

22. #ML7SAIS
Required Data
● Impressions, Plays, etc.
● Attribution
● Explore/Exploit Metadata

23. #ML7SAIS
Attribution Infrastructure
Stream Processing Batch Processing
Query API

24. #ML7SAIS
Stream Processing - Zoomed in
Impression
Cache
Video
Metadata
Play
Enrich

25. #ML7SAIS
Batch Processing - Zoomed in
Impression
Play
Experimentation
Data
Windowed
Impression
Windowed
Play
Dedupe
Join
Video
Metadata

26. #ML7SAIS
Common Infrastructure

27. #ML7SAIS
Netflix Spark Stack
● Jenkins
● Spinnaker
● ASG
● Runners: Marathon, Meson
● Resource Manager: Mesos
● Storage: HDFS, S3, EFS
● Multi-Region

28. #ML7SAIS
Multi Region Challenges
● Geo routing
● Impression in one region;
play in another
● Streaming - Multi Region
● Batch Reduce/Merge - One
region

29. #ML7SAIS
Can it withstand Chaos?
• Chaos is a design principle
• Instance Failovers => Region Failovers
• Transparent to the consumers
• Over provisioned
• Long term - Autoscale

30. #ML7SAIS
When Things Go South
● What if something doesn’t look right?
○ Stream Processing is stuck
○ Driver/Executor failures
○ Intermittent issues with external dependencies
● Metrics - Spark metrics to Atlas (similar to RRDTool + Graphite)
● Getting paged at 2 am - Not fun :) !
● Need for auto-remediation - less operational overhead

31. #ML7SAIS
Auto Remediation Infrastructure
SQS Auto Remediation
De-queue
Triggers
Metadata

32. #ML7SAIS
• Scalability Performance
tuning
– Micro batch interval
– Memory Tuning
– Parallelism/Shuffle
tradeoff
• Data quality issues
– Low latency vs data
quality
Streaming Challenges