Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Relevancy and Search Quality Analysis - Search Technologies

694 Aufrufe

Veröffentlicht am

Slides from Enterprise Search & Analytics Meetup @ Cisco Systems - http://www.meetup.com/Enterprise-Search-and-Analytics-Meetup/events/220742081/

Relevancy and Search Quality Analysis - By Mark David and Avi Rappoport

The Manifold Path to Search Quality

To achieve accurate search results, we must come to an understanding of the three pillars involved.

1. Understand your data

2. Understand your customers’ intent

3. Understand your search engine

The first path passes through Data Analysis and Text Processing.

The second passes through Query Processing, Log Analysis, and Result Presentation.

Everything learned from those explorations feeds into the final path of Relevancy Ranking.

Search quality is focused on end users finding what they want -- technical relevance is sometimes irrelevant! Working with the short head (very frequent queries) has the most return on investment for improving the search experience, tuning the results, for example, to emphasize recent documents or de-emphasize archive documents, near-duplicate detection, exposing diverse results in ambiguous situations, using synonyms, and guiding search via best bets and auto-suggest. Long-tail analysis can reveal user intent by detecting patterns, discovering related terms, and identifying the most fruitful results by aggregated behavior. all this feeds back into the regression testing, which provides reliable metrics to evaluate the changes.

By merging these insights, you can improve the quality of the search overall, in a scalable and maintainable fashion.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Relevancy and Search Quality Analysis - Search Technologies

  1. 1. 1 The Manifold Path to Search Quality Enterprise Search & Analytics Meetup Mark David – Architect, Data Scientist Avi Rappaport – Senior Search Quality Analyst 19 March 2015
  2. 2. 2 “manifold” • adjective – having numerous different parts, elements, features, forms, etc. - dictionary.com
  3. 3. 3 Search Technologies: Who We Are The leading independent IT services firm specializing in the design, implementation, and management of enterprise search and big data search solutions.
  4. 4. 4 Solutions Corporate Wide Search – “Google for the Enterprise.” A single, secure point of search for all users and all content. Strategic initiative for corporate wide information distribution and search. Data Warehouse Search – A Big Data search solution that enables interactive query and analytics with extremely large data sets for business intelligence and fraud detection. E-Commerce Search – Leverages machine learning and accuracy metrics to deliver a better online user experience and maximize revenues from visitor search activity. Search & Match – Increase recruiter productivity and fill rates in the staffing industry. Provides a better search experience followed by automated candidate-to-job matching. Search for Media & Publishing – Improve user search experience for publishers of large amounts of content such as government organizations, research firms, and media publications. Government Search – A solution focused on design and development search for government information portals or archiving systems.
  5. 5. 5 Search Technologies: Background San Diego London UK San Jose, CR Cincinnati Prague, CZ Washington (HQ) Frankfurt DE • Founded 2005 • 150+ employees • 600+ customers worldwide • Deep enterprise search expertise • Consistent revenue growth • Consistent profitability
  6. 6. 6 600+ Customers
  7. 7. 7 Search Technologies: What We Do • All aspects of search application implementation – Content access and processing, search system architecture, configuration, deployment – Accuracy analysis, metrics, engine scoring, relevancy ranking, query enhancement – User interface, analytics, visualization • Technology assets to support implementation – Aspire high performance content processing – Content Connectors (Document, Jive, SharePoint, Salesforce, Box.com, etc.) • Engagement models – Most projects start with an “assessment” – Fully project-managed solutions, designed, delivered, and supported – Experts for hire, supporting in-house teams or as a subcontractor
  8. 8. 8 Search Engine and Big Data Expertise Our Technology and Integration Partners
  9. 9. 9 Content sources Connectors Aspire Content Processing Pipelines Indexes Search Engine Web Browser Staging Repository Publishers Technology Assets 1. Aspire Framework – High Performance Content Processing – Ingests and processes content and publishes to a variety of indexes for commercial and open source search engines 2. Aspire Data Connectors – API level access to content repositories 3. Query Processing Language (QPL) – Advanced query processing Complements to commercial and open source search technologies 1 2 3 QPL
  10. 10. 10 Engine Data Users
  11. 11. 11 Understand Your Data • Data Analysis – Access patterns & rates, sources, schemas, field typing, duplicates, near-duplicates, term frequencies, etc. • Content Processing – Source connection, format conversion, sub-document separation, field boundaries, multiple-source assembly, etc. • Text Processing – Character decoding, tag stripping, tokenization, sentence boundaries, normalization, entity extraction, pattern recognition, disambiguation, filtering, etc.
  12. 12. 12 Understand Your Users • Search Scope – Interviews – Log Analysis – Scenarios – Wireframes & mockups • Search Quality • Improvements – Relevance – Coverage – UX
  13. 13. 13 Understand Your Search Engine • How does it score results? • How accurate is it for the short head? • How accurate is it for the long tail? • When you change it to improve a particular type of query, how do you know that the overall accuracy improved?
  14. 14. 14 Regression Testing of Search • Step 1: Gather a Set of Judgments • If you already have lots of user data: – Use click log analysis to gather sets of clearly good and clearly bad results – Ignore unclear tracks • If user data not yet available: – Manual judgments • End up with a set of queries with associated “good” and “bad” documents
  15. 15. 15 Regression Testing of Search • Step 2: Instrument the Search Results • Periodically execute all those queries, and score the results • How to score: – Every good document adds a position-based amount – Every bad document subtracts the same amount – Unknown documents don’t affect the score (except by occupying a position)
  16. 16. 16 Understand the Data
  17. 17. 17 Relevancy Improvements from Data • Text Processing – Typos – Entity Extraction – Breaks – Parts of Speech • Data Analysis – TF-IDF – Phrase Dictionary – Boilerplate
  18. 18. 18 To Correct or Not To Correct • Should typos be “fixed”? • This goes back to knowing your audience • Example: Haircutz • In document-to-document situations, generally yes.
  19. 19. 19 Bigger Needles in the Haystack • Entity Extraction: How big a chunk? • Example: mdavid@searchtechnologies.com – Is that 1, 2, 3, 4, or 5 tokens? • Multi-indexing is a key component of accuracy – Different people think differently, so the indexes need to have different ways of representing the data.
  20. 20. 20 Breaker, Breaker • Don’t match across boundaries – Paragraph – Sentence – Phrase • Whitespace does have meaning! • Punctuation does have meaning!
  21. 21. 21 Parts is Parts • Figuring out the part of speech (noun vs. verb vs. adjective) would seem to clearly help – We avoid matching on the incorrect version • Study after study shows that it does not! • Why not? – Closely related (in English) • Example: to go on a run – Prevalence of noun phrases in the group of “important” terms
  22. 22. 22 How Common are Tokens Terms? • Term Frequency (not “Token Frequency”) – Example: The West, West London, The Wild West • Do your full text processing when you’re gathering statistics – And adjust it and re-run it when the data changes • Inverse Document Frequency – In how many docs does this term occur? – NOT: How many times does this term occur across all docs?
  23. 23. 23 Let Me Re-Phrase That • Some general dictionaries are freely available – Example: locations (geonames.org) • Others can be derived – Example: Company names from stock markets, business registries, Wikipedia, etc. • More useful are terms from your industry – Can you think of lists that are available internally? – Example: Job titles in a recruiting company • Most useful are terms from your data – Statistical generation of common 2-shingles and 3-shingles – Query log analysis
  24. 24. 24 Lorem ipsum… • Boilerplate text recognition • Pre-process: – Simple text processing this time – Split by paragraphs – Calculating hash signatures for paragraphs – Count occurrences • Find the cliff • Filter out early in the main pipeline – Early steps must match the entire pre-processing pipeline
  25. 25. 25 Understand the Users
  26. 26. 26 Search Quality • Best possible results – Given the searchable data – For the primary users and their primary tasks • Simple query term matching - relevance • And beyond – Enriched content – Query enhancement • Results presentation – Clarity – Context
  27. 27. 27 Short Head & Long Tail • Query Frequency – Short Head • A few frequent queries – Short Middle • Often to 50% by traffic – Long Tail • Rare to unique queries • Can be to 75% distinct
  28. 28. 28 But What Do They Really Want? • Query log reports show what users think they’re looking for – Domain research for more about why • Behavior shows more about whether they’re finding it – Session ending • Frequent for zero matches – CTR - click-through rate • Results (with bounce rates) – Query refinement • Typing, facets • Navigation via search
  29. 29. 29 You say “tomay-toe” • Users vocabulary is not content vocabulary – Consistent problems from small to web-scale search • Create synonyms • Scalable automated disambiguation – Data analysis • Using dictionaries and co-occurrence – Search log behavior analysis • Query refinement and reformulation, click tracks – Language ambiguity - even Netflix has a hard time past 85% – Human domain expertise, editorial oversight
  30. 30. 30 Scope (aka, this is not Google) • User confusion – Is this a location box? – Is it Google? • Design for clarity – UI and graphic design – Watch out for default to subscope searches • Improve content coverage • Add Best Bets for internal and external locations • Link to other search engines • Federate search
  31. 31. 31 Fix The Short Head Issues
  32. 32. 32 Best Bang for the Buck • Concentrate on the short head – Top 10% by traffic • Simple relevance test – Perform query – Evaluate results • Are there any results? • Are they the most useful available? (Domain expertise) • Validate against user behavior – Store judgments – Easy fixes – Re-test (easy to miss this)
  33. 33. 33 Choosing, Not Typing • Auto-suggest – Curated search suggestions • Best Bets • Did you mean?
  34. 34. 34 Context and Navigation • Facets • Results grouping / diversity – options for ambiguous queries • Integrate with collaboration tools – Allow user comments, reviews
  35. 35. 35 Relevance and Ranking • Best results patterns – Part or serial number queries • Tuned boosting – Feedback on clicks and other signals – Freshness • de-duplication!
  36. 36. 36 Relevance and Regression Testing
  37. 37. 37 Search Technologies Co
  38. 38. 38 Reference Architecture Content sources Connectors Indexes Semantics Text Mining Quality Metrics Aspire Content Processing Pipelines Aspire Aspire Aspire Aspire Aspire Aspire Aspire Aspire Big Data Framework Big Data Array Indexes QPL Search Engine Web Browser Staging Repository