1. From Idea to
Execution: Spotify’s
Discover Weekly
Chris Johnson :: @MrChrisJohnson
Edward Newett :: @scaladaze
DataEngConf • NYC • Nov 2015
Or: 5 lessons in building
recommendation products at scale

2. Who are We??
Chris Johnson Edward Newett

3. Spotify in Numbers
• Started in 2006, now available in 58 markets
• 75+ Million active users, 20 Million paying subscribers
• 30+ Million songs, 20,000 new songs added per day
• 1.5 Billion user generated playlists
• 1 TB user data logged per day
• 1,700 node Hadoop cluster
• 10,000+ Hadoop jobs run daily

4. Challenge: 30M songs… how do we recommend
music to users?

5. Discover

6. Radio

7. Related Artists

8. Discover Weekly
• Started in 2006, now available in 58 markets
• 75+ Million active users, 20 Million paying subscribers
• 30+ Million songs, 20,000 new songs added per day
• 1.5 Billion user generated playlists
• 1 TB user data logged per day
• 1,700 node Hadoop cluster
• 10,000+ Hadoop jobs run daily

9. The Road to
Discover Weekly

10. 2013 :: Discover Page v1.0
• Personalized News Feed of
recommendations
• Artists, Album Reviews, News
Articles, New Releases, Upcoming
Concerts, Social
Recommendations, Playlists…
• Required a lot of attention and
digging to engage with
recommendations
• No organization of content

11. 2014 :: Discover Page v2.0
• Recommendations grouped into
strips (a la Netflix)
• Limited to Albums and New
Releases
• More organized than News-Feed
but still requires active
interaction

12. Insight: users spending more time on
editorial Browse playlists than Discover.

13. Idea: combine the
personalized experience
of Discover with the lean-
back ease of Browse

14. Meanwhile… 2014 Year In Music

15. Play it forward: Same content as the
Discover Page but.. a playlist

16. Lesson 1:
Be data driven from
start to finish

17. Slide from Dan McKinley - Etsy
2008 2012 2015

18. • Reach: How many users are you reaching
• Depth: For the users you reach, what is the
depth of reach.
• Retention: For the users you reach, how many
do you retain?
Define success metrics BEFORE you
release your test

19. • Reach: DW WAU / Spotify WAU
• Depth: DW Time Spent / Spotify WAU
• Retention: DW week-over-week retention
Discover Weekly Key Success Metrics

20. 2008 2012 2015
Slide from Dan McKinley - Etsy

21. Step 1: Prototype (employee test)

22. Step 1: Prototype (employee test)

23. Results of Employee Test were very positive!

24. 2008 2012 2015
Slide from Dan McKinley - Etsy

25. Step 2: Release AB Test to 1% of Users

26. Google Form 1% Results

27. Personalized image resulted in 10% lift in WAU
• Initial 0.5% user test
• 1% Spaceman image
• 1% Personalized
image

28. Lesson 2:
Reuse existing
infrastructure in creative
ways

29. Discover Weekly Data Flow

30. Recommendation
Models

31. 1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
•Aggregate all (user, track) streams into a large matrix
•Goal: Approximate binary preference matrix by inner product of 2 smaller matrices by minimizing the
weighted RMSE (root mean squared error) using a function of plays, context, and recency as weight
X YUsers
Songs
• = bias for user
• = bias for item
• = regularization parameter
• = 1 if user streamed track else 0
•
• = user latent factor vector
• = item latent factor vector
[1] Hu Y. & Koren Y. & Volinsky C. (2008) Collaborative Filtering for Implicit Feedback Datasets 8th IEEE International Conference on Data Mining
Implicit Matrix Factorization

32. 1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
•Aggregate all (user, track) streams into a large matrix
•Goal: Model probability of user playing a song as logistic, then maximize log likelihood of binary
preference matrix, weighting positive observations by a function of plays, context, and recency
X YUsers
Songs
• = bias for user
• = bias for item
• = regularization parameter
• = user latent factor vector
• = item latent factor vector
[2] Johnson C. (2014) Logistic Matrix Factorization for Implicit Feedback Data NIPS Workshop on Distributed Matrix Computations
Can also use Logistic Loss!

33. NLP Models on News and Blogs

34. Playlist itself is a
document
Songs in
playlist are
words
NLP Models work great on Playlists!

35. [3] http://benanne.github.io/2014/08/05/spotify-cnns.html
Deep Learning on Audio

36. •normalized item-vectors
Songs in a Latent Space representation

37. •user-vector in same space
Songs in a Latent Space representation

38. Lesson 3:
Don’t scale until
you need to

39. Scaling to 100%: Rollout Challenges
‣Create and publish 75M playlists every week
‣Downloading and processing Facebook images
‣Language translations

40. Scaling to 100%: Weekly refresh
‣Time sensitive updates
‣Refresh 75M playlists every Sunday night
‣Take timezones into account

41. Discover Weekly publishing flow

45. What’s next?
Iterating on content
quality and interface
enhancements

46. Iterating on quality and adding a feedback loop.

47. DW feedback comes at the expense of presentation bias.

48. Lesson 4:
Users know best. In the
end, AB Test everything!

49. Lesson 5 (final lesson!):
Empower bottom-up
innovation in your org and
amazing things will happen.

50. Thank You!
(btw, we’re hiring Machine Learning and
Data Engineers, come chat with us!)