Deploying Data Science for Distribution of The New York Times - Anne Bauer

1. Deploying Data Science for Distribution of Anne Bauer anne.bauer@nytimes.com Lead Data Scientist, NYTimes PyData 20181017

2. Single copy newspaper distribution 1.  people still buy physical newspapers? 2.  algorithms 3.  experiments to test the algorithms 4.  ...we need to modify the algorithms 5.  app architecture

4. YES! at ~ 47,000 stores! (And you should too.)

5. How many papers should we deliver to each store each day? Too many or too few: a waste of $$, or missed sales! “Single copy” optimization

6. Single copy: the process Weekly process •  Stores report sales for 1-2 weeks ago (depending on the distributor) •  We pick up the data via FTP, ingest them into our systems •  Our models are retrained, predictions run •  Predictions are handed off via FTP to the circulation department Turnaround time ~ few hours

7. Single copy: the existing algorithm Heuristics with many if/then statements •  Highest sale over recent weeks × A + B •  A, B are extremely hand-tuned by store type, location, ... •  Interspersed amid 4600 lines of COBOL

8. Single copy: the existing algorithm Heuristics with many if/then statements •  Highest sale over recent weeks × A + B •  A, B are extremely hand-tuned by store type, location, ... •  Interspersed amid 4600 lines of COBOL

9. Single copy: the existing algorithm Heuristics with many if/then statements •  Highest sale over recent weeks × A + B •  A, B are extremely hand-tuned by store type, location, ... •  Interspersed amid 4600 lines of COBOL •  Difficult to modify to include, e.g. print site cost differences •  Quintessential time series modeling problem. Perfect for data science!

11. Algorithm components The problem is separable into two parts: Prediction: Given previous sales, how many papers will sell next Thursday? Policy: We think N papers will sell, with a known uncertainty distribution. How many should we send (draw)?

12. First pass: AR(1) Xt = c + φ Xt-1 + εt Daeil Kim

13. AR(1) Prediction •  Xt = c + φ Xt-1 + εt •  Today’s sale is a linear function of last week(s) •  One model per store per day of week •  Use the past year’s data to fit for c, φ •  AR(1) vs. AR(N) and training window chosen via cross-validation Policy •  Draw = ceil(demand) •  Bump: if there have been recent sell-outs, send an extra

14. AR(1) Implementation •  Python 2, with statsmodels AR model. Single script. •  Plots (matplotlib pngs) hosted using Flask to monitor draws & sales •  Run by cron on a local server •  No separate dev/prd environments; code “deployed” via scp

15. Second pass: Poisson Regression Dorian Goldman

16. Poisson Regression Prediction •  Today’s sale is a linear function of the previous week(s) and the previous year •  One model per store per day of week •  Use the past year’s data to fit model parameters •  Feature time scales chosen via cross-validation •  Assume the sales are drawn from a Poisson distribution rather than Gaussian •  Sell-outs considered in the likelihood function

17. Poisson Regression b: # papers bought d: # papers delivered (the draw) z: demand (Poisson distributed latent variable) λ: Poisson parameter for the demand distribution Each store has a different λ each day. z for that store & day is drawn from a Poisson distribution with that λ. Parameterize Poisson parameter λ as log-linear function of features X. θ are the parameters fitted in the problem via ML.

18. Poisson Regression b: # papers bought d: # papers delivered (the draw) z: demand (Poisson distributed latent variable) λ: Poisson parameter for the demand distribution Probability of the # bought given the demand depends on if the demand > papers delivered (i.e. if there was a sell-out) Use this probability for a maximum likelihood estimation of the parameters θ that describe λ

19. Poisson Regression Policy: Newsvendor Algorithm •  Profit = price × min(d, z) – cost × d •  Take derivative of the profit, set it equal to zero, implies: Probability(z <= d) = (price-cost)/price •  Optimal draw: smallest integer such that Probability(z <= d) >= (price-cost)/price •  Probability given by the CDF of the Poisson distribution, z = the demand prediction, brute force find best d. z = demand d = draw = # delivered

20. Poisson Regression Implementation: refactored code! •  Models abstracted to sklearn-like classes to allow for easy future expansion with plug & play model integration •  Common library of functions to: •  get data from the DB •  calculate costs •  check data quality •  ... •  __init__() •  query() •  transform() •  fit() •  predict() •  policy()

22. Treatment & Control groups: match sales Simple approach •  Take a random sample that approximates the total sales distribution •  For each member of this “treatment” sample, find closest match in mean sales Trial & error checks! •  Exclude cases with any large differences in sales during the training period •  Only consider matches with the same production costs (~print site) •  Make sure treatment & control sell the paper on the same weekdays •  Better no match than a distant match

23. Reporting D3 Dashboard Optimize for profit: ✔ Make stakeholders happy: ✗ Our profit comes at the expense of sales! Sales matter beyond sales profit. Circulation numbers matter. Hard to quantify that value!

24. Goal: Optimize for profit ... but don’t decrease sales “too much” ∴ Constrained optimization

26. Constrained newsvendor algorithm Policy: Newsvendor Algorithm •  Profit = price × min(d, z) – cost × d Maximize profit – λ × sales (negative λ to boost sales) Effectively modifies the sales price of the paper •  (price – λ) × min(d, z) – cost × d •  Optimal draw: smallest integer such that Probability(z <= d) >= (price-λ-cost)/(price-λ) Negative λ → increase effective sales price → worth sending extra papers z = demand d = draw = # delivered

27. The stakeholders choose λ To our surprise, they chose λ such that sales loss ~0 and profit was suboptimal. But still much better than the original algorithm!! This tuneable knob is very handy; we run experiments with different λs and the stakeholders can make the final decisions on which results are best. Δ | |

28. Reporting: model comparison Look at both profit and sales differences between treatment & control Leave trade-off decisions to the stakeholders: better for everyone.

30. Current architecture: Google Cloud App Engine: Web front end App Engine Flex: Back ends for reporting and predictions BigQuery, Cloud Storage, Cloud SQL: for hosting data and configuration Deployed via Drone (github.com/NYTimes/drone-gae) Github → Docker → GCR → AE Flex Github → AE Standard

31. Architecture: Process Data transfer •  Weekly cron job per distributor, on AE instance •  Taskqueue task: copy data from FTP to BQ, using config info in GCS •  Task fails if the data are not there •  The task queue retries every N minutes until the data shows up Logging •  Logs sent to Stackdriver, emails sent upon errors •  Quality checks and progress messages sent to Slack

32. Architecture: Process Reporting •  Reads data from BQ •  Calculates aggregations & stats about algorithm experiments, using config info from CloudSQL (BQ & pandas) •  Saves aggregated data back to BQ •  Runs statistical tests on data quality (e.g. last week’s total sales within 3σ of previous mean), aborts if failure •  Syncs the aggregated BQ tables with CloudSQL, for use in filtering the front end UI

33. Architecture: Process Predictions •  Reads data from BQ •  Retrains and predicts next week’s sales & how many papers to deliver to each store each day (sklearn, scipy), using config info from CloudSQL •  Saves results to GCS •  Runs tests for unexpected changes in predictions, aborts if failure Upload •  The front end copies the results from GCS back to the FTP site

34. A well-distributed project experiments A/B testing algorithms with $ directly as a KPI communication Fold qualitative business concerns into the math engineering Google Cloud Platform improves our process algorithms Sell-outs, costs directly incorporated