The talk explores following topics: - What is the search relevance and why is it important? - Relevance scoring in Elasticsearch - Manipulating relevance with Query DSL structure - Pros and cons in using Machine Learning for improving search relevance - Using Learning to Rank (aka Machine Learning for better relevance) in Elasticsearch

1. IMPROVING SEARCH
RELEVANCE IN ELASTICSEARCH
USING MACHINE LEARNING
LEARNING TO RANK IN ELASTICSEARCH

2. MILORAD IMBRA
SOFTWARE ARCHITECT

3. SEARCH RELEVANCE
• Content, content, more content - more products, more logs, more emails, more articles
• Search box – magical solution
• Relevance – matching / ranking results to user’s needs

4. DEMO

5. MEASURING RELEVANCE
• User engagement
• nDCG (Normalized Discounted Cumulative Gain)
• Precision@k, Recall@k, MRR/MAP
• A/B Testing (checking for improvements)

6. RELEVANCE SCORING IN ELASTICSEARCH
• query -> _score (higher the score, more relevant the document)
• TF/IDF (term frequency/inverse document frequency) standard similarity algorithm in ES
• Term frequency : How often does the term appear in this document? The more often, the higher the weight.
tf(t in d) = √frequency
• Inverse document frequency: How often does the term appear in all documents in the collection? The more
often, the lower the weight
idf(t) = 1 + log ( numDocs / (docFreq + 1))
• Field-length norm: How long is the field? The shorter the field, the higher the weight.
norm(d) = 1 / √numTerms

7. DEMO

8. PRACTICAL SCORING FUNCTION
score(q,d) =
queryNorm(q)
· coord(q,d)
· ∑ (
tf(t in d)
· idf(t)²
· t.getBoost()
· norm(t,d)
) (t in q)

9. HOW TO MANIPULATE RELEVANCE IN ES
• Query-time boosting
• Ignoring TF/IDF
• function_score query
• Boosting by popularity
• Random scoring
• script_score function
• Changing similarity algorithms (pluggable)

10. DEMO

11. MACHINE-LEARNED RANKING
• LTR – Learning To Rank
• Creating ranking models for information retrieval systems
• Re-ranking – doesn't replace the search engine

12. ELASTICSEARCH LEARNING TO RANK
• Based on RankLib (8 algorithms: MART, RankNet, RankBoost, AdaRank, CoordinateAscent,
LambdaMART, ListNet, RandomForests)
• Judgment lists (“golden sets”)
• Features – selecting and experimenting with features is the main task in LTR

13. IMPLEMENTING LEARNING TO RANK
• Creating Judgment Lists
• Selecting and experimenting with features
• Incorporating features into the judgment lists (training set)
• Running RankLib to train models (and testing models)
• Deploying the model into Elasticsearch to use it at search time

14. DEMO

15. TAKE INTO CONSIDERATION
• How to produce good judgment lists that match users sense of search quality?
• What metrics best measure whether search results are useful to users?
• Is Your Infrastructure Ready For Learning To Rank (collecting and logging user behavior and features)?
• How will you detect when/whether your model needs to be retrained?
• How will you A/B test your model vs your current solution?

16. CONCLUSION
• No silver bullet for Search Relevance Engineering
• ES LTR can be useful tool that enables scaling
• Still a lot of hard work! Make your infrastructure ready.

17. REFERENCES
• Relevant Search With applications for Solr and Elasticsearch, Doug Turnbull and John Berryman,
Manning 2016
• Learning to Rank in Solr, https://www.youtube.com/watch?v=M7BKwJoh96s
• https://elasticsearch-learning-to-rank.readthedocs.io/
• https://opensourceconnections.com/blog/2017/02/14/elasticsearch-learning-to-rank/
• RankLib - https://sourceforge.net/p/lemur/wiki/RankLib/

18. Q & A

19. THANKS!