Spotify has many time series to be forecasted such as streams forecast, monthly active users, ads inventory and consumption, etc. Prophet library has been very effective in capturing most of the time series requirements, however tuning is necessary as the default set of parameters don't perform well on our more noisy datasets with many confounding factors.

1. Prophet at Scale:
Using Prophet at scale to tune and
forecast time series at Spotify
Mahan Hosseinzadeh
Data Engineer
PyData London 2019, July 13th

2. Background
2

3. Who am I
● Data & Software Engineer
● 3 years on Spotify’s Forecasting team
● Spark, Scio, Google Cloud & Hadoop user
3

4. Spotify’s Fast Facts
Spotify is a digital music, podcast, and video streaming freemium service
● 100M+ Subscribers
● 217M+ Monthly Active Users
● 50M+ Songs
● 3B+ Playlists
● 79 Markets
(retrieved 2019-08)
https://newsroom.spotify.com/company-info/
4

5. Forecasting at Spotify
● We forecast streams, monthly active users, ads inventory and etc.
● 10K+ time series to forecast
● Usually forecasting 2 years ahead
● Happens at the end of each quarter
5

6. Time Series Data
“a series of data points indexed (or listed or graphed) in time order.” Wikipedia
6

7. What is Prophet
● An open source time series forecasting tool by Facebook
● Available in Python and R
● Forecasting with default settings is often accurate as other complex models
7

8. Spark on GCP*
● Dataproc is a Google cloud service running Spark
● Fast
● Easy-to-use
● Fully managed
● Cost-effective
* Google Cloud Platform
8

9. Python is a new King!
● Majority of data science libs are available in Python
● Prophet is also in Python
● Python+Spark=PySpark helps to scale easily
● Installing Python libs on Dataproc is a piece of cake
9

10. Scaling in the cloud
10

11. One Forecast
● Runtime varies from couple of seconds to few minutes
● Runtime is based on input size and Prophet settings
11

12. Many Forecasts
10K forecasts takes ~2 days when running sequentially
12

13. Many Forecasts on Spark
10K forecasts takes ~30 mins when running distributed
13
Task 1
Task 2
Executor 1
Task 1
Task 2
Executor 2
Worker Node 1
Task 1
Task 2
Executor 1
Task 1
Task 2
Executor 2
Worker Node 2
SparkContext
Driver

14. In-house Model Distributor
14
● A tool to distribute/scale a Python model
● Plug & play, no Spark knowledge needed
● Runs a model with permutation of parameter values
● Model outputs can be aggregated with Spark for further analysis
● Full integration with Bigquery, Cloud Storage and Dataproc

15. In-house Model Distributor
15
training data
1 set of parameters
1 forecast
training data
many sets of parameters
many forecasts
Model Distributor

16. Parallel Tuning
Occasionally Prophet default settings are not good enough and tuning is needed
16
Task 1
Task 2
Executor 1
Task 1
Task 2
Executor 2
Worker Node 1
Task 1
Task 2
Executor 1
Task 1
Task 2
Executor 2
Worker Node 2
SparkContext
Driver
Forecast 1
with 5 sets of settings
Forecast 2
with 3 sets of settings

17. Parallel Tuning
Defined parameters
17
country
UK
SE
IT
US
prior_scale
0.5
0.6
0.7
0.8
n_changepoints
45
50
55
60

18. Task 1
Executor 1
Worker Node 2
Task 1
Executor 1
Worker Node 1
Sequential Tuning
18
SparkContext
Driver
Forecast 1
with setting ranges
Forecast 2
with setting ranges

19. Sequential Tuning
Range parameters
19
country
UK
SE
IT
US
prior_scale
[0.5 to 0.8]
n_changepoints
[45 to 60]

20. Interesting stats
● 1 node: 16 vCPUs, 60GB memory
● 50 nodes: 800 vCPUs, 3TB memory
● Cluster creation: ~4 mins
● 10K forecasts: ~30 mins
● Cost: ~5$
20

21. Challenges
21
● Get the most out of Dataproc cluster
● It’s not efficient to tune all 10K+ forecasts
● Job finishes with no results
● Testing locally with local Spark is not enough to find some of the errors
● Big output with so many small files

22. Thank you!
mahanhosseinzadeh@spotify.com
https://www.linkedin.com/in/mahanhoss
22

23. Container
Registry
BigqueryScio
(Dataflow)
Styx
BigquerySpark
(Dataproc)
Tech stack with data ingestion
23