This document discusses Spotify's approach to real-time personalization using ad tech. It provides details on Spotify's monetization strategy, including using ads to fund free music streaming. The system analyzes user behavior data in real-time and batches to improve ad relevance and deliver the right message for every moment. Key parts of the system use Storm for real-time stream processing to power ad targeting based on a user's current music session. The overall design includes collecting logs, processing them in Storm topologies, storing user profiles, and enabling real-time lookups to power ad serving.

1. May 6th, 2014
Real-Time Personalization
with Ad Tech @Spotify
Kinshuk Mishra
kinshuk@spotify.com
@_kinshukmishra
1

2. About Me
• @Spotify since 2011
• Data and Backend engineering
• With the Monetization team

3. May 12, 2014
Spotify in numbers
Started in 2006, available in 56 markets
20+ million songs, 20,000 added daily
24+ million active users, 6+ million subscribers
1.5 billion playlists

4. May 12, 2014
Monetization at Spotify
• $1 billion paid to rights holders since launch
• 1 in every 4 Spotify users is a paying subscriber
• Ad revenue pays for free tier music streaming
• Build platform to power artist and music label promotions

5. What is our mission?
• Power the free tier with ads customers love.
– Delight users, brands and artists with standout
promotional experiences.
– Deliver the right message for every moment.
5

6. Delight users, brands and artists with standout
promotional experience
6
• Ad formats need to be
- Native
- Actionable
- Scalable
- Engaging

7. 7
Spotlight

8. 8
Album promotion

9. 9
Sponsored Genre

10. 10
Sponsored Genre

11. 11
!(Right message for every moment)

12. Things we had to do
• Make ads context aware.
• Create measurable ad formats.
• Build tools to help brands understand their audience.
• Build flexible targeting infrastructure.
• Analyze user behavior to improve ad relevancy.
12

13. Why is real time interesting?
• True assessment of the context
- current view, last track, etc.
• Immediate feedback
- ad clicked, followed a playlist, etc.
• Increased ad opportunities
- hyperlocal, current mood, etc.
13
source : http://nativemobile.com/tag/real-time-bidding

14. How to serve right message for every moment ?
• Fix the WTFs for ads in the music context
- Realtime short window activity analysis
• Improve ad relevancy based on user behavior
- (Realtime + Batch) long window activity analysis
14

15. Some quick wins
• Fix the WTFs for ads in the music context
- Fix the “Not Safe For Kids” ads problem.
- Fix the relevancy of ads in your music session.
• Improve ad relevancy based on user behavior
- Fix ad relevancy based on user’s musical taste.
15

16. Targeting Architecture
16

17. Desired overall design
17

18. Our requirements
• Scalable log collection
• Capability to process logs in batch and
realtime mode and aggregate user activity
• Capability to store user profiles and
enable serve time lookups
• Ability to update and add new features to
existing user profiles

19. Bigdata @Spotify and choices
• 700 node hadoop cluster
• 400 GB service logs daily
• 4.5 TB user data daily
• 7500 hadoop jobs daily
• 64 TB data generated daily
• YARN Map-Reduce, Giraph,
Storm, Spark, etc.
!
19

20. Overall design and tech choices
20

21. Source : http://tfosuccess.com/day-106-107-the-storm-arrived/
What is Storm?
• Real time stream processing
• Like Hadoop without HDFS
• Like Map/Reduce with many reducer steps
• Fault tolerant and guaranteed message processing
21

22. Storm @Spotify
• storm-0.8.0
• 22 node cluster
• 15+ topologies
• 200k tuples/second
• ads, recommendation,
analytics, monitoring, etc.
22
source: http://storm.incubator.apache.org/

23. Use-case
Use real-time session genre information to control
ad serving in real-time
23

24. Realtime activity analysis
• Kafka -> Storm -> Memcached -> Targeting Backend
• Soft realtime
• Easy to scale
24

25. Getting data
25

26. Getting data across the globe
26

27. Genre Topology
27

28. May 12, 2014
Measuring effectiveness
• Build ad quality scores from lower-level metrics
• Positive response, avoidance, etc.
• A/B test hypotheses, evaluate against quality scores
• Build quality score optimization into real-time system

29. May 6th, 2014
Want to join the band?
https://www.spotify.com/us/jobs/ or
https://twitter.com/Spotifyjobs
!
Kinshuk Mishra
kinshuk@spotify.com @_kinshukmishra