1. Query Log Mining
Yandex Challenge 2011
Nikita Spirin, Shih-Wen Huang, Shuo Yang, Anirudh Ravula

2. Search logs are used to improve search
• Learn a ranking functions
– Users click on meaningful results
• Personalize search based on users history
– Previous user searches unveil users interests
• Identify spammers
– Bots click on suspicious websites more often
• Tune contextual advertizing models
• Recommend and disambiguate queries
– See also “java programming” Vs. “java coffee”

3. Yandex QLM Challenge 2011 goals
• Learn a ranking function
– For a given query provide a list of ordered URLs using
the information from the log
• Plan for today
– Task description
– General framework: learning to rank (L2R)
– Features for L2R
– Preferences extraction for L2R
– Ranking algorithms
– Collaborative Filtering and graph-based approaches
– Experiments
– Future Plans to improve

4. Task description: Input to the challenge
• Query log
– Query action
SessionID TimePassed QUERY QueryID RegionID ListOfURLs
– Click action
SessionID TimePassed CLICK URLID
• Training relevance labels from {0,1} set
QueryID RegionID URLID RelevanceLabel
• Testing query/region pairs
– The goal is to provide relevant URLs for these new
query/region pairs

5. Some real input data
• Snapshot of the real Yandex query log
SessionID Time Action QueryId RegionId URL URL URL
• Training relevance labels from {0,1} set
QueryId RegionId URL Relevance

6. Some statistics about the query log
• Unique queries: 30,717,251
• Unique URLs: 117,093,258
• Sessions: 43,977,859
• Total records in the log: 340,796,067
• Assessed query-region-url triples for the total
query set (training + test): 71,930
• Log size: 17 Gb (doesn’t’t fit into memory)

7. General Framework: Learning to Rank (L2R)
• Training formalization:
– Given an ordered set of ranks Y = {0,1} (0 < 1)
– Given a set of queries Q = {q1, . . . , qn}
– A list of documents is associated with each query
Dq = {dq1, . . . , dq,n(q)}
– Factor ranking model:
• Xqd = ( f1(q, d), . . . , fm(q, d) ), feature vector for q-d pair
• Goal of L2R:
– Learn a Ranker: X Y

8. Subtasks of L2R from query logs
• Extract preferences (absolute, pairwise)
form a query log using click-through
statistics
• Generate features (factors) to make a
problem structured
• Learn a ranking algorithm

9. SVM for L2R = RankSVM
• Extract preferences from a query log based on
some heuristics

10. Boosting for L2R = RankBoost
• Uses each feature as a decision stump
• Builds a linear weighted ensemble model

11. Ensemble Approach
• Generate multiple models by varying…
– Feature subsets
– Algorithms parameters
– Ranking models
– Model Subsets
– Averaging strategies (weighted, quality-absed,
etc.)
• Finally average [similar to CombMNZ]

12. Best result so far
0.642436

13. Future work
• Add more models
– SVMpref (reduction on L2R to classification)
– Direct optimization of AUC
– Experiment with more sophisticated ensemble
models (MonoRank, etc.)