The concept of talk is as follows: - to give a general idea about user segmentation task in DMP project and how solving this problem helps our business - to tell how we use autoML to solve this task and to explain its components - to give insights about techniques we apply to make our pipeline fast and stable on huge datasets

1. AutoML for user segmentation
Ilya Boytsov
Rambler&Co

2. Rambler&Co - largest media holding in Russia

3. About Rambler&Co AdTech
Projects:
• Data management platform (user
segmentation)
• Recommender systems
• “Lumiere” (forecasting offline cinema traffic)
• Computer vision

4. About Rambler&Co AdTech
Projects:
• Data management platform (user
segmentation)
• Recommender systems
• “Lumiere” (forecasting offline cinema traffic)
• Computer vision

5. In this talk:
• DMP and user segmentation tasks explained
• Key structures of AutoML pipeline for user segmentation
• Problems we faced while maintaining pipeline
• Feature engineering for machine learning at scale
• Optimization of pipeline tasks

6. Data management platform (DMP): a powerful AdTech
solution
-Collect user behavior data from various sources
-Integrate data to create a complete customer view
-Store and manage audience segments
-Target audience segments in online ad companies

7. Types of Data Sources of Data
1st party data – raw events logs (visited
websites)
2nd party data – customer journey data
3rd party data – data collected from partners
Media resources
Products and services
Data from ad campaigns, behavioral factors
Other sources
Образец слайда

8. DMP AutoML pipeline: solution for any user
segmentation task
About 1000 models fitted on daily basis
Every model is being applied on 300 million of test samples daily
ML problems:
• binary/multiclass classification
• Look alike –> binary classification(segment vs random)

9. Retargeting_973
Look-alike 0.0-1%
Look-alike 1.0-5.0%
Look-alike 5.0-10.0%
Retargeting_1069
Look-alike 0.0-1%
Look-alike 1.0-5.0%
Look-alike 5.0-10.0%
Examples: Look-alike modelling boosts ctr

10. General principles of DMP AutoML
All models have similar structure of fit and apply stages
Adding models and exploitation options have to be implemented with
web interface
No need for ML developers to support a scope of key operations

11. Felix
Backoffice and web
interface for AutoML
pipeline
Create new models, add new segments,
visualize model performance and many more

13. AutoML pipeline daily workflow
Felix
Compute features
Create train table
Train models
Compute pivots load
pivots
Apply and slice
predictions
Compute
metrics
Load
models

14. Workflow manager: Apache Airflow
• Run a series of tasks as DAG (directed acyclic graph)
• Express task dependencies
• Handle failures

15. Train and apply DAG`s
Train DAG interval: every 4 hours
Apply DAG interval: every hour

16. Train and apply DAG`s
Train DAG interval: every 4 hours
Apply DAG interval: every hour
Problem:
Some target segments(labels) finish computing
slower than others.
Solution:
While some models wait for target segments, other
models keep training

17. Train and apply DAG`s
Train DAG interval: every 4 hours
Apply DAG interval: every hour

18. Key problems we faced
• Data collection delay
• Out of memory issues
• High cardinality feature matrices
• Too much time to map predictions with label thresholds
• Some models are being applied more often than others

19. Data collection delay
• Use Airflow sensor to wait for MAX_ FEATURE_DELAY

20. Data collection delay: do not wait too much
• Use Airflow sensor to wait for MAX_ FEATURE_DELAY
• If exceeded fill the missing parts of features table with last computed
day

21. Feature Engineering(FE): overcoming high cardinality
feature matrices
Main rule:
New Features must be
applicable for a majority of
models
Key techniques
• Counting based FE
• Distance based FE

22. Feature matrix of shape (N, 10000)
id Feature_1 ... Feature_10000
1 42 ... 542
.... ... ... ...
N 89 ... 0

23. Distance based FE: Cluster distance
Algorithm:
1) Reduce dimension of feature matrix if needed (we use SVD
decomposition)
2) Fit KMeans clustering algorithm with K clusters on given data
3) Calculate distance from sample point to centroid of Kth cluster
4) Use distances as feature representation for sample row

24. Feature matrix of shape (N, K)
id dist_to_1st_cluster ... dist_to_Kth_cluster
1 0.6757 ... 0.0942
.... ... ... ...
N 0.342 ... 0.6113

25. Problem:
It may take much time for KMeans to converge
and compute distances for every model…

26. Solution: “Global” Cluster distance Feature
• Fit KMeans only once on representative
unlabeled sample to extract general information
and use for all models

27. Experimental results:
• Replacing individually fitted by model distance features with ”Global” feature
doesn’t harm model quality
• Combining both feature representations improve roc auc score about 1%

28. Counting based FE
Traditional approaches:
1) Feature Hashing
2) One hot encoding
User Domain Count
Bob news.rambler.ru 5
Bob auto.ru 11
Bob mercedes-benz.ru 15

29. Counting based FE
Traditional approaches:
1) Feature Hashing
2) One hot encoding
General problems:
1) High cardinality
2) Efficient with only linear models

30. Counting based FE: DRACULA
Domain Robust Algorithm for Counting Based Learning
Source: http://www.slideshare.net/SessionsEvents/misha-bilenko-
principal-researcher-microsoft

31. Algorithm
Compute counts table from all train
data
Compute P(label | feature) for
every unique feature
Aggregate list of probabilities to get
low cardinality data representation
01
02
03

32. Counts of visited domains for single
user
User Domain Count
Bob news.rambler.ru 5
Bob auto.ru 11
Bob mercedes-benz.ru 15
Domain Count
news.rambler.ru 95859
auto.ru 31040
mercedes-benz.ru 1386
Total counts in train data

33. Counts table
Domain Total count Count(label=0) Count(label=1)
news.rambler.ru 95859 41268 54591
auto.ru 31040 26809 4231
mercedes-benz.ru 1386 1120 266
Total 128285 69131 59154

34. 𝑁label = domain frequency with given label
𝑁 𝑝𝑜𝑠 𝑎𝑝𝑟𝑖𝑜𝑟 = smoothing constant * aprior class
probability
𝑁 𝑛𝑒𝑔 𝑎𝑝𝑟𝑖𝑜𝑟 =1- 𝑁 𝑝𝑜𝑠 𝑎𝑝𝑟𝑖𝑜𝑟
N = 𝑁 𝑝𝑜𝑠 + 𝑁 𝑛𝑒𝑔
P(label | domain) =
𝑁 𝑙𝑎𝑏𝑒𝑙 + 𝑁 𝑝𝑜𝑠 𝑎𝑝𝑟𝑖𝑜𝑟
𝑁+𝑁 𝑝𝑜𝑠 𝑎𝑝𝑟𝑖𝑜𝑟 + 𝑁 𝑛𝑒𝑔 𝑎𝑝𝑟𝑖𝑜𝑟
General formula: probability of label given domain

35. Compute data representation for single user
P(label=0|domain = ”news.rambler.ru”) = 0.43
P(label=0|domain = ”auto.ru”) = 0.86
P(label=0|domain = ”mercedes-benz.ru”) =
0.81

36. Compute data representation for single user
P(label=0|domain = ”news.rambler.ru”) = 0.43
P(label=0|domain = ”auto.ru”) = 0.86
P(label=0|domain = ”mercedes-benz.ru”) = 0.81
N = 10
Bins = [0.0 , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9,
1.0 ]
Histogram = [0, 0, 0, 0, 1, 0, 0, 0, 2, 0]

37. Compute data representation for single user
P(label=0|domain = ”news.rambler.ru”) = 0.43
P(label=0|domain = ”auto.ru”) = 0.86
P(label=0|domain = ”mercedes-benz.ru”) = 0.81
N = 10
Bins = [0.0 , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9,
1.0 ]
Histogram = [0, 0, 0, 0, 1, 0, 0, 0, 2, 0]
Interpretation: the concentration of
nonzero elements in histogram
represents estimation of P(label | user)

38. Advances of algorithm
• Scalable (add new features, recompute probabilities)
• Adaptive (fits for binary and multiclass classification and regression as
well)
• Efficient for gradient boosting decision trees due to low cardinality
• Ability to compute features in distributed manner (mapreduce)
• Ability to store counts table with count min sketch

39. Compute models pivots task
Approach to approximate label thresholds
Problems:
• how to select thresholds for
labels?
• how to do it computationally
fast?
Desired solution:
• Heuristics to approximate label
thresholds

40. Compute models pivots task
Approach to approximate label thresholds
Precision-threshold for binary classification.
What probability threshold optimizes given metrics quality?

41. Compute models pivots task
Approach to approximate label thresholds
Algorithm:
• Take sample of apply data ( we use 5%, about 15 million samples)
• Compute probabilities histogram for this sample
• Use Nth percentile as estimation for label threshold

42. Apply model task
Task interval: every hour
Number of models per run: 200
General problem:
• Some models are being applied more often then others

43. Priority schema of apply models
1) Request all models
2) Filter out not yet trained models
3) Sort by date of adding a model (descending)
4) Sort by date of last apply (ascending)
5) Take N top priority models

44. Key notes:

45. Key notes:
1) Think of scalable approach

46. Key notes:
1) Think of scalable approach
2) Implement monitoring of pipeline performance

47. Key notes:
1) Think of scalable approach
2) Implement monitoring of pipeline performance
3) Make experiments

48. In lieu of conclusion...
Pipeline automation...is full of fun

49. Questions? Contact me!
https://www.facebook.com/ieboytsov
i.boytsov@rambler-co.ru