This is my talk given at eXelate's "big data eXposed 2015" event in Tel-Aviv.
It touches the common problem of trying to optimize websites for better conversion rates, and covers the range of solutions - starting at classic A/B testing (unsuitable for rapidly changing content), going through Multi-Arm Bandits and finally arriving at Contextual Bandits - online machine learning algorithms allowing for true personalization in real-time.
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
Taking Machine Learning from Batch to Real-Time (big data eXposed 2015)
1.
2. Learn Like a Human – Taking Machine
Learning from Batch to Real-Time
Elad Rosenheim
3. Who am I
Architect at Dynamic Yield,
“Predictors” Team Lead
Previously:
AlphaCSP
SAP
Performance & Scale, DevOps
Measure All the Things!
East-Asia & Japan
4. Who’s Dynamic Yield?
We’re optimizing & personalizing websites since 2011
Start-up in Tel-Aviv
headed by Liad Agmon
I Joined as 5th employee, we’re 50 now and growing fast
5. On the Agenda
Our clients’ problem
Old School Solutions
Meet the ML Bandits
6. Our clients’ problem
Publishers, retailers, SaaS
all share a common problem
They know their domain
but not how to optimize for each user
Screen real-estate is limited
yet everyone sees the same thing
7. What top videos to show on
NBC News’ site?
What user segments should see
this element at this location?
What’s the best layout for this
element?
8. Both the layout of this page and
each element in it deserve testing
What’s the best layout?
What types of products to show
whom?
9. What articles to show on
ynet’s homepage?
What titles and images?
In what order?
10. What is the best default sort order for products on Adika?
Does is significantly differ between user segments?
11. The Beginning
First, there was the educated guess
Then, there was the A/B test
"Data Beats Opinion“
Freedom to experiment (with nice tools)
Hopefully: less fear of change, less politics
How does it work?
Split traffic between baseline and alternative variations
In theory: sit & wait for significant results
In practice: peek at the numbers till the nice “95% confidence”
12. A/B Tests: Already Old School?
While you wait, you're bleeding clicks
clicks == money
What about the really dynamic stuff?
Campaigns, Current Headlines, Products on Sale
13. Enter the Multi-Arm Bandits
A Single-Arm Bandit
Suppose I have multiple arms in front of me,
each with its unknown mean reward…
How do I optimize income from multiple machines?
Caution or Haste?
Explore vs. Exploit
In our context:
How do I optimize multiple variations?
14. Bandits - A Classic Problem
(Very) Simple Solutions
ε-greedy, ε–decreasing
First 100% random explore, then ~90% exploit?
Magic numbers, built-in revenue loss
Bayesian-based approaches
Smoother curve from explore to exploit
“Winner” is now a less relevant term
15. Bandits work well when…
We want to find the variation “best on average“
…but we’re not improving the conversion rate of any single variation
2.4% 1.7% 0.4%
16. Enter Personalization
Each of us is a beautiful and unique feature vector!
By showing the right variation to the right people,
we can improve conversions per variation
and beat the best variation
ML Challenge Accepted
17. The Usual Suspects
Collaborative Filtering?
Very big, very sparse matrix
Cold Start
Batch
Not suitable in this case
Classifiers?
Logistic Regression, Random Forest et al.
Periodically learn over all converters so far
More data == more time, bigger model
Not the classic question
18. What We Need
Like a bandit, we need to learn as we go (not in batch),
but this time with “context” - the user’s data
Incremental Learning over the stream of impressions & rewards
(“Partial Fit”)
We’re looking to…
Start learning from the first impression
Handle the explore-exploit curve
Run fast (enough)
In the worst case: converge on the best variation, like a bandit
19. Meet the Contextual Bandits
They “eat” the data stream
They demand fast access to user data
Historical or immediate
Their model is always ready for action
In the Papers
Linear Bayes, LinUCB
What we do: Per-Variation Logistic Regression
A variant supporting updates in “mini-batches”
Exploration-on-top
Worst case: “Garbage In Multi Arm Bandit Out”
Light on memory, compact output
20. Online should be fast & scale
Offline: a testbed for iteratively testing new ideas
New algorithms
Tweaked parameters
Feature transformations
How We Do It: Online & Offline
21. The Online Flow
DY Web Servers
a. get our script
b. log impressions,
conversions
Queue
Per Test
Learn
Workers
User
DB
Persist
ModelLoad to
Predict Server
Queue
Per Test
A B C
A B C
A B C
Predictions
22. The Offline Evaluator
Test, Improve, Iterate
Using real-world data
Using generated data
From easy to hard
23. Going Global
Learn in the center site, fast predict in each geo. How?
Push models via local Redis slaves
Compressed SSH tunnel
User data - daily aggregation
Storage into LMDB (simple, fast memory-mapped K/V DB)
Sync via S3 (LZ4 compressed), read from SSD
Learn & Predict services
Python as ML lingua franca: NumPy, SciPy, scikit-learn
24. Elad & Idan Say Goodbye
Better data beats better algorithms
Reduce aggressively
Keep It Simple, Smart!
Elad Rosenheim
Idan Michaeli
Read our blog
Hiring? but of course! What’s with the Groundhog?
Hinweis der Redaktion
איך ניסו עד היום לתקוף את הבעיות האלה?
בואו ניקח את הפיתרון שלנו שלב אחד קדימה
בואו ניקח את הפיתרון שלנו שלב אחד קדימה
בואו ניקח את הפיתרון שלנו שלב אחד קדימה
בואו ניקח את הפיתרון שלנו שלב אחד קדימה
בקיצור, העולם לא כ"כ מופלא כפי שמוכרים לנו
בואו ניקח את הפיתרון שלנו שלב אחד קדימה
לבעיה הזו אין פיתרון אופטימלי, אבל יש בהחלט גישות שונות ברמות שונות של מורכבות
עכשיו, בנדיטים הם לא כ"כ רעים למעשה...
והמגבלה הזו מביאה אותנו לשלב הבא בחיפוש, והוא: פרסונליזציה
יפה, אז קיבלנו את האתגר. על איזה אלגוריתמים אנחנו חושבים?
אז בעצם, אנחנו מחפשים משהו אחר – משפחה חדשה של אלגוריתמים
אז בואו ונכיר את הגיבורים החדשים שלנו...
אלגוריתם כשלעצמו זה דבר נחמד, אבל איך בונים את כל המעטפת לפרודקשן?