How we learned to rank search results big data meetup

How we learned to rank
search results
Mouloud Lounaci & Andres Pipicello
Argentina Big Data Meetup
Meetup #6: Long time, no see OLX
October, 2018

2
mouloud.lounaci@olx.com
@mlounaci
https://www.linkedin.com/in/mlounaci
andres.pipicello@olx.com
https://www.linkedin.com/in/andrespipicello

Plan
OLX Group
Personalization and Relevance (PnR)
Learning To Rank (LTR)
The ranking journey
Building Dataset
Modeling
Serving the model
Results
3

4
Preface What do you need to know
about OLX Group ?
OLX Group

Scale of Data at OLX Group
5
35B
Monthly
Page Views
350M
Monthly
Users
60M
Monthly
Listings
4B
Daily
Events
Every minute...
2.5M events captured
500 houses listed
500 cars listed
1000 phones listed

Classifieds ?
6
Two-sided Marketplace
● Buyers looking for goods or services
● Sellers offering goods or services
OLX’s Mission
● Match buyers with sellers using
○ large scale data
○ state-of-the-art technology

Today it’s ALL about “search”
7
▪ Retrieval of relevant listings
– Query understanding
– Query-listing matching
▪ Ranking of relevant listings
– Learning to rank (LTR) using
query, user and listing
features
Query
Ranked relevant
Items

Actually it’s about “Ranking”
8
▪ Retrieval of relevant listings
– Query understanding
– Query-listing matching
▪ Ranking of relevant listings
– Learning to rank (LTR) using
query, user and listing
features
Query
Ranked Items

9
Chapter 1
Who are “we” ?
What do we do ?
How do we do it ?Personalization and
Relevance

PnR - Architecture
12
Data Sources
Indexing
Retrieval +
Ranking

Ad Retrieval
13
Phoenix
Constructs the feed to send to the user
Manages all the different spells (algorithms)
used in the feed
Splitter for A|B Testing

Ad Retrieval
14
Loki
Executes the spell (algorithm) from Phoenix
Interacts with all the different data sources
Caches items for fast Page 2 retrieval

15
Chapter 2
Ragnarok ?
Why LTR ?
What is LTR for usLearning To Rank

What we want to do ?
“Learn from the data how to rank a resultset
for a search query”
aka.
RAGNAROK
AD RERANKER

Manual models become hard to tune with a very large number of
features.
1
Leverages large volume of user behaviour (Clicks/replies) data in
an automated way
Create a personalized ranking by including user features (social
search)
2
3
Why Learning To Rank ?

Top r e
do n
Top Rer d
do n
User
query
(Re)ranked
results
Spell
Returned
Documents
User
behaviour
If I click/reply,
then it’s
relevant for me
RAGNAROK
AD RERANKER
Overview

19
Chapter 3
Before we start, we need
Tools
The search for gold
Mining the gold (Spark),
Funnel ?
Modeling, or transforming
the gold
Serving the model
The ranking journey

The ranking journey
Step 1
Building Infra
● Access large
history Data
(Reservoir)
● Build Infra to
process it
(EMR)
Building Dataset
● Process Label
(Judgement
score proxy
with
Clicks/Replies).
● Process
Features.
Step 2
Analysing Dataset
● Analyse Click
and reply
behaviour
● Build “Gold
standard”
dataset for
ranking
Step 3
Building Model
● Iterating on
models
● Evaluating
models
● Selecting a
model
Step 4
Serving Model
● Design Service
Architecture
● Define Service
Requirement
● Create Ranking
Endpoint
Step 5
Integration with PnR
Architecture
● Integrate the
ranking in the
ad retrieval flow
● Define the
interaction with
the PnR
components .
Step 6

Step 1
User browsing
(parquet - 1h delay)
RELEVANCE
RESERVOIR
Ads
(json - 5 min delay)
Labeled Dataset
Features
Building Infra
STORAGEPROCESSING
Building Infra

Big Data ?
23
1 year android history for South Africa
data...
Step 1
Building Infra
5B
User events
800M
Search
Impressions
40M
Individual
searches

Scalability is key
24
1 year history for South Africa data...
Step 1
Building Infra
5B
User events
800M
Search
Impressions
40M
Individual
searches

Step 2
Gold looks like this for us...
query_id query
features
Item
position
item_id Item
features
Label (Relevance
Judgement)
1 ... 1 item1 ... 0
1 ... 2 item2 ... 3
1 ... 3 item3 ... 1
1 ... 4 item4 ... 2
Building Dataset
The search for the “gold” standard dataset

Step 2
We used spark (EMR) to build the dataset from user browsing data.
Building Dataset
Hydra
(Trackings)
Labeled
searches
(funnel)
Labeling
(apply funnel)
Let’s “spark” it off

Proxy label, the “funnel”
Step 2
Building Dataset
?

Step 3
● Considering searches with at least one reply for training
(to improve quality)
● Include searches with more than 3(4) impressions (user
behaviour affected by smaller resultset)
● Inside each search consider impression up to 30-50-60th
position
● Metric that gives more importance to top position(NDCG with
customized decay)
Analyzing Dataset
Analysing Dataset

Q1
D1,1
D1,2
…
D1,m
Q2 Qn
...
D2,1
D2,2
...
D2,m
Dn,1
Dn,2
...
Dn,m
Pointwise Pairwise Listwise
f(Qi, Di,j) = s o F(Qi, Di,j > Di,k) = s o n {o,1} f(Qi, {Di,j,...,Di,m}) = {Di,j,...,Di,m})
ra d
Baseline
Q1,D1
Q1,D2
Q1,D3
Q1,D1 0.85
Q1,D3 0.65
Q1,D2 0.30
Q1,D1>D2
Q1,D2>D3
Q1,D3>D4
Q1,D1>D2 1
Q1,D2>D3
0
Q1,D3>D4
1
Q1,D1
Q1,D2
Q1,D3
D1
D3
D2
Step 4
Building Model
Start with a simple approach

McRank from classification to ranking
● Pointwise approach
● Train a classifier to predict the
relevance judgment k i {0, 1, 2}
● Use the class probabilities P(Y=k)
Ran g re =∑ P(Y=k) ∗T(k), w e n o se T(k)=k
Inspired by :
https://papers.nips.cc/paper/3270-mcrank-learning-to-rank-using-multiple-classification-and-gradient-boosting.pdf
Step 4
Building Model

35
Combined Model
Item Features Buyer Features Seller Features
Static Features
Interaction
Features
(Browsing)
Three Classes of Features
Step 4
Building Model

Search String
Search Location
Search Time
Ad Title
Ad Description
Ad Location
Ad Creation Time
Ad Price
Ad Private or Business
Ad Image Count
Ad Category
Textual Similarity (BM25)
Length of the Title
Length of the Description
Freshness
Proximity
Price
Is the Seller a Private Business
Image Count
Category
Raw attributes Features
Static Item/query features
Step 4
Building Model

Item Interaction Features - Example
Interactions
Impressions
Ad Views(Clicks)
Replies
Data source: ods.fact_listing_activity
Time Interval
30 days
7 days
Last day
Item Interaction Features
num_impressions_30days
num_adviews_30days
num_replies_30days
num_impressions_7days
num_adviews_7days
num_replies_7days
num_impressions_lastday
num_adviews_lastday
num_replies_lastday
Step 4
Building Model

We met “Mleap“ on the way
39
Step 5
Serving Model

40
Step 5
Serving Model
The Service
Aws Data Pipeline For
training
Scala Akka Htttp with
mleap service on
Openshift for
prediction
Training every 7 days Serving
RAGNAROK
AD RERANKER
Ranked Items
ReRanked Items

41
Chapter 4
Does this work ?
Offline ?
Online ?Results

Preliminary results (Offline)
Feature Weight
Proximity 13
bm25 8.7
Freshness 4.6
Price 0
Title Length -4.3
Description Length -7.3
+14% nDCG

Preliminary results (Online)
Feature Weight
Proximity 13
bm25 8.7
Freshness 4.6
Price 0
Title Length -4.3
Description Length -7.3
+14% nDCG
+8%
Replies/DAU

44
Final results (Offline)
Feature Weight
Item Replies Received - 30 days 21.6
Preference for Cars - 30 days 15
Proximity 9.1
bm25 8.2
Preference for Car Parts - 30 days 8.2
Freshness 5.7
+71% nDCG
Item Performance
Buyer preference
Basic features

45
Final results (Online)
Feature Weight
Item Replies Received - 30 days 21.6
Preference for Cars - 30 days 15
Proximity 9.1
bm25 8.2
Preference for Car Parts - 30 days 8.2
Freshness 5.7
+71% nDCG
Coming soon...
Item Performance
Buyer preference
Basic features

46
The end Thank you
Any questions ?

How we learned to rank search results big data meetup

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie How we learned to rank search results big data meetup

Ähnlich wie How we learned to rank search results big data meetup (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

How we learned to rank search results big data meetup