33. Relevance Labelling for Contextual Search
• For learning we need labels.
• Relevance labelling for contextual (personalized) search
(auto-completion) is not trivial.
• Previous work on personalized search [Fox et al., 2005]
• Samples search impressions from the logs
• Documents with SAT clicks are annotated with relevant labels
• The goal is to learn a re-ranking model that improves the
ranking of those relevant documents given the context.
35. Experimental Settings
• Ranker: Lambda-Mart [Burges et al., 2011]
• AOL testbed
• 657K users (Mar-May 2006)
• 128,620 queries in the prefix-tree
• Userid, query, timestamp
• Bing testbed
• 196K logged in users with Microsoft LiveID (Jan-2013)
• 699,862 queries in the prefix-tree
• Userid, query, timestamp, age, gender, zip code
• Training & testing on different sets of users
36. Personalized Ranking Features
• Demographics
• Age (5 groups)
• Gender (2 groups)
• Zip-code (10 groups)
• Search history
• Short (session)
• Long (all past queries)
37. Personalization by Age
The effectiveness of auto-completion personalization according to
the user’s age in terms of MRR. All differences are statistically
significant (P < 0.01)
Testbed
Baseline
Personalized
MRR (Gain/Loss)
Bing (age)
-
-
+3.80%
Frequently promoted suggestions for different age groups
Below 20
21-30
31-40
41-50
Above 50