Abstract:- In this talk, we propose a generalized machine learning framework for e-commerce businesses. The framework is responsible for over 30 different user-level predictions including lifetime value, recommendations, churn predictions, engagement and lead scoring. These predictions provide a vital layer of intelligence for a digital marketer. Kinesis is used to capture browsing information from over 120M users across 100 companies (both in-app and web). A data processing and feature engineering layer is build on Apache Spark. These features provide inputs to predictive models for business applications. Different models each for Churn, Lifetime value, Product recommendation and search are written on Spark. These models can be plugged into any marketing campaign for any integrated e-commerce company leading to a generalized system. We finally present a monitoring system for machine learning called RS Sauron. This system provides more than 200 objective metrics measuring the health of predictive models, and depicts KPIs for model accuracy in a continual setting.

1. Generalized B2B Machine Learning
at Reten4on Science
Andrew Waage (andrew@reten*onscience.com)
Co-founder / CTO

2. 2012
Santa Monica
~50 (and hiring!)

3. AI Powered Marketing Automation Platform
Step 1
Collect Data
Step 2
Generate Predictions
Step 3
Automation Powers Intelligent
Campaigns Across Channels
Ecom / Retail Behavioral Custom Demographic
Email
Campaigns
On-Site
Display
Mobile Call
Center

4. What kind of Scale?
100’s Clients
210M+ Customers Tracked
1000+ Client-specific Models
2 Billion+
Predictions Daily10K+ Actions / second

5. “Generalized ML Platform”
What’s the
challenge?

6. The Challenge
• Many Clients
• Dirty Data
• Sparse Datasets
• Custom Attributes
• Various Industries
Clean
PredictionsModel Layer!
C1 C2 C3 C4

7. What Kind of Predictions?
Purchase Probability: High-likelihood
Lifecycle Stage: Ready to Buy
Churn Time: 300 days
Customer Future Value: $925
Contact Frequency: Every 3 days
Optimal Time to Engage: Thursday 7-9PM
Optimal Incentive/Discount: Dollars Off
Product Recommendations: Based on interest
Optimal Subject Line: Individual preference
Optimal Template: Individual preference

8. Our Approach & Learnings
1. Robust Ingestion Pipeline
2. Common Feature Engineering Layer
3. “Plug-in” Architecture for Models
4. Evaluation / AB Testing
5. Robust Monitoring & Visualization

9. 1. Robust Ingestion Pipeline
10K+ Actions Per Second
auto-scaling!
auto-scaling
lambdas!
• Abstraction Layer: Data
Ingestion
• Do not compromise for
clean data
• Auto-scaling everywhere
• High confidence in
upstream data
Flume
Kinesis

10. 2. Common Feature Engineering Layer
• Abstraction: Feature Layer
• Allow custom features
• Handle feature selection
• Modelers know what to
expect
Raw Data
User Behavior
Features
Product
Features
User Sta4c
Features
Timing Model CLV Model Recommender

11. 3. Model Plug-in Architecture
C6
C3
C4
C5
C7
• Plug-in Architecture
• Tune model hyper-parameters
• A/B test models per client
C1
C8 C2
Client’s Model Execution Plan

12. Recommender System
Multi-Layer Personalization:
• Layer 1: ML / Algorithmic
• ALS CF, Content-based, Item-Item
• Layer 2: User-level Domain Logic
• User-level predictions
(gender, location, shoe sizes)
• Layer 3: Client-tuned Domain Logic / Controls
• Rank by profit-margin
• Increase discovery rate influxer
1. Algorithmic
2. User-Level
Domain Logic
3. Client-Level
Controls

13. 4. Model Evaluation / Fast Feedback
A/B Framework
M1 M2 M3
• Start Simple
• Collect feedback data
• Skip long production cycle
• Unbiased policy
generation is important
M1
Campaign Predic4ons

14. 5. Robust Model Monitoring and Visualization
“Sauron”
(LOTR)!
Monitor,
monitor,
monitor!!

15. Monitor Recs: Distributions, Coverage, Diversity
REF: hQp://sauron.rsci.co/advanced/sites/181/data_types/31

16. Monitoring Subject Line Bandit Models

17. Churn Rates, ROC Curve, Reliability Curve

18. Our Data Science Stack
Persistence! Pipeline / Process! Viz / Monitor! Code!

19. Takeaways
1. Use abstraction layers
- Clean / common interfaces
2. Monitor, monitor, monitor
- Fast feedback
3. Start simple and keep iterating

20. Thank You!
andrew@retentionscience.com