Haystack 2019 - Establishing a relevance focused culture in a large organization - Tom Burgmans
•
1 like•115 views
For a relevance engineer one of the most difficult tasks in the tuning process is to convince others in the organization that this is a joint effort. Even the brightest search guru doesn't get very far when working in isolation, so establishing cross-collaboration through the organization is essential. But how to get there? On top of that, in a large organization a relevance engineer often works on multiple seemingly unrelated search projects. The challenge is not to get drowned in building custom solutions for each project, but to design generic and re-usable strategies which solve many problems at once. In this session we'll discuss how to build a widely supported basis for search quality improvements in an organization. It is full of practical tips and examples which could help you in establishing a cross-functional culture that is optimal for relevance tuning. It also zooms in on an holistic approach of solving multiple equivalent search issues at once.
Recommended
More Related Content
Similar to Haystack 2019 - Establishing a relevance focused culture in a large organization - Tom Burgmans
Similar to Haystack 2019 - Establishing a relevance focused culture in a large organization - Tom Burgmans (20)
More from OpenSource Connections
More from OpenSource Connections (20)
Recently uploaded
Recently uploaded (16)
Haystack 2019 - Establishing a relevance focused culture in a large organization - Tom Burgmans
- 1. Establishing a relevance focused culture in a large organization Tom Burgmans Search Scientist April 25th 2019 1
- 2. 2 Once upon a time
- 3. 3 • Matching all- and only the correct documents • Tuning the scoring algorithm • Accurate highlighting • Attractive KWIC summaries • Variety of facets / filters • Predictive autocompleting • Auto-correcting misspellings • Suggesting spelling corrections • Providing intuitive feedback • Support grouping / collapsing • Pro-active recommendations • Direct answers Improving Search Quality means… =
- 4. 4 When tuning relevance… creating/maintaining judgment sets feature development query processing (vocabularies) curating/enriching content search engine configuration tuning scoring algorithm strategy meetings analysis & research bug fixing What we ‘tune’: What we’re actually doing: Search Engine Content Vocabularies Product features
- 5. 5 “A goal without a plan is just a wish.” ― Antoine de Saint-Exupéry
- 6. 6 Search Quality team Boss / Product owner Content architect / curator Search specialist Domain expert(s) owns the tuning program understands the content understands the search engine understands the user’s need Project manager Business Analyst Solution architect DevOps engineer Controlled Vocabulary specialist manages the tuning exercise understands the business’ needs architects the overall design manages infrastructure manages dictionaries designs the user interface UX developer / Front-end designer Bill of materials Documentation space Issue tracker Meeting schedule Project essentials Tuning environment Full content set Customer feedback Usage logsAnalysis tools Judgement queries https://youtu.be/4tcizH8b5pY Mature version product & content
- 7. Let’s go! Search Engine Content Vocabularies Product features Categorize issues Strategize improvements Study user feedback and logs Analyze and experiment Tune search Domain expert(s) Content architect Search specialist Update tuning environment Update production Meetings Search Quality team Boss Solution architect Project manager DevOps engineer Controlled Vocabulary specialist UX developer Business Analyst
- 8. Maintenance Categorize issues Strategize improvements Study user feedback and logs Analyze and experiment Domain expert(s) Content architect Search specialist Update tuning environment Update production Meetings Search Quality team Boss Solution architect Search Engine Content Vocabularies Product features Tune search
- 9. “The whole is greater than the sum of its parts.” Aristotle
- 10. 10
- 11. 11
- 12. Relevance metrics Search Quality dashboard DWH Judgment set tests Environments Environments & software statistics User behavior statistics A/B tests Change metrics Search Quality Team
- 13. query autocomplete semantic suggestions index spelling SaaS Designing search in a large company Search Engine Extremely flexible similarity class https://youtu.be/BsyVsmuS50c Highlighter Query cooking Content transformation & enrichment Spellchecker Autocompleter Semantic query suggester Generic, re-usable, highly customizable, loosely coupled components Plugins
- 14. Search engine query User query to AST Query analysis Query planning Tree to Solr syntax ? term q=… Regular expressions Java classes Vocabularies Expand synonyms Remove stop words Quote concepts Normalize dates Apply auto filters Detect concepts & synonyms Detect auto-filters Detect known queries Detect dates & other patterns Detect stop words Flexible query cooking pipeline
- 15. 15 Lessons learned • Quality is value • Have a plan • Get involved early • Work in a team with multiple disciplines • Create a community • Have decent tools • Design generic, re-usable, customizable • Don’t stop
- 16. The End