4. 4
“Relevance” vs “Ranking”
Conceptually “relevance” determination and “ranking” can be thought of a two
different steps (even if they are implemented as one in a search engine)
5. 5
“Relevance” vs “Ranking”
Conceptually “relevance” determination and “ranking” can be thought of a two
different steps (even if they are implemented as one in a search engine)
Relevance
6. 6
“Relevance” vs “Ranking”
Conceptually “relevance” determination and “ranking” can be thought of a two
different steps (even if they are implemented as one in a search engine)
Relevance
Ranking
1
2
11. 11
Measuring query-document similarity
See “Introduction to Information Retrieval” by Manning et al:
http://nlp.stanford.edu/IR-book/
> 700
papers
Goal: given query + document string, compute “similarity”
13. 13
Measuring query-document similarity
“philadelphia phillies”
Query Model
tokenization
normalization (stemming)
query expansion
intent
In this context “document” can also refer to title tag, meta description, H1, etc.
0.74
14. 14
Measuring query-document similarity
“philadelphia phillies”
Query Model
tokenization
normalization (stemming)
query expansion
intent
Document Model
tokenization
normalization (stemming)
vector space representation
language model
In this context “document” can also refer to title tag, meta description, H1, etc.
0.74
15. 15
Measuring query-document similarity
“philadelphia phillies”
Query Model
tokenization
normalization (stemming)
query expansion
intent
Document Model
tokenization
normalization (stemming)
vector space representation
language model
In this context “document” can also refer to title tag, meta description, H1, etc.
Scoring function
0.74
21. Document representation
Probability Ranking Principle
P(R = 1 | d, q) or P(R = 0 |
d, q)
TF-IDF Language Model
P(optimization | search, engine)
>>
P(walking | search, engine)
22. Which method performs best?
What are the characteristics of sites that rank highly?
14,000+ keywords
Top 50 results
600,000 URLs
Google-US, no personalization
March 2013
Mean Spearman Correlation
Remember: “correlation is not causation”
23. Which method performs best?
We tried a few different types of smoothing for the language model,
Dirichlet worked best (Zhai and Lafferty SIGIR 2001)
28. 50 results
450
random
pages
movie reviews For each
query:500 pages
10% relevant
90% irrelevant
URL ID PA In SERP?
86 92 1
355 90 0
… … …
27 18 0
URL ID Language
Model
In SERP?
213 0.97 1
156 0.95 1
… … …
355 0.06 0
29. 50 results
450
random
pages
movie reviews For each
query:500 pages
10% relevant
90% irrelevant
URL ID PA In SERP?
86 92 1
355 90 0
… … …
27 18 0
URL ID Language
Model
In SERP?
213 0.97 1
156 0.95 1
… … …
355 0.06 0
P@50 is the “Precision of the top 50 results”. It is the percentage of top 50
results by PA/Language Model that are actually in the SERP.
Top 50
ranked
30. 50 results
450
random
pages
movie reviews For each
query:500 pages
10% relevant
90% irrelevant
URL ID PA In SERP?
86 92 1
355 90 0
… … …
27 18 0
URL ID Language
Model
In SERP?
213 0.97 1
156 0.95 1
… … …
355 0.06 0
P@50 is the “Precision of the top 50 results”. It is the percentage of top 50
results by PA/Language Model that are actually in the SERP.
Top 50
ranked
32. Takeaways
Implication: Query-document similarity is based on decades of
research. It’s immune to algorithm change.
Action item: With sophisticated query and document models, no
need to optimize separately for similar words, e.g. “movie
reviews” vs “movie review”.
33. Takeaways
Implication: Query-document similarity is based on decades of
research. It’s immune to algorithm change.
Action item: With sophisticated query and document models, no
need to optimize separately for similar words, e.g. “movie
reviews” vs “movie review”.
Action item: Each page is relevant to many different keywords,
so optimize each page for a broad set of related keywords,
instead of a single keyword.
34. Takeaways
Implication: Query-document similarity is based on decades of
research. It’s immune to algorithm change.
Action item: With sophisticated query and document models, no
need to optimize separately for similar words, e.g. “movie
reviews” vs “movie review”.
Action item: Each page is relevant to many different keywords,
so optimize each page for a broad set of related keywords,
instead of a single keyword.
Use case: Content creation. What keywords will this new blog
post target? Is it relevant to a set of queries?
