Better search engine testing for solr

•

0 gefällt mir•1,787 views

A refactoring of my earlier presentation targeted more towards Solr, and less on testing. I'm basing my presentation for Lucene EuroCon on this. Would love some more feedback!

Technologie Design

BETTER SEARCH
ENGINE TESTING
FOR SOLR

CVILLE MEETUP | EPUGH@O19S.COM | @DEP4B

1

WHY AM I QUALIFIED TO BE
UP HERE?
• Principal of OpenSource
Connections - Solr Search
Consultancy

• Member of Apache Software
Foundation

• Presenter at conferences (OSCON,
ApacheCON, UberConf, ExpoQA,
STPcon 2009/2011!)

• Fascinated by the art of software
development
2

AGENDA

Why Test Search?

Two General Approaches

Test Tools for Solr

Wrap Up

5

photo credit to Jay Grieves @ www.bigswingingdeveloper.com
7

TWO APPROACHES TO TEST

Humans

Automated
Tests

8

HUMAN TESTING

• Better deals with ambiguity

9

I KNOW IT WHEN I SEE IT!

http://en.wikipedia.org/wiki/Les_Amants

11

SWEET SPOT

• Repeat Testing for Relevancy Tuning

• Performance Testing: hitting SLA?

• Regression Testing (Typical use case for automated testing)

13

“ONE TRUE ANSWER”

• Absolute Truth / Matrix / Grid / TREC / Relevancy Assertions

• The correct answers for each search are known ahead of
time

• Humans judges often decide these correct answers, stored
as Relevancy Assertions

• Can be labor intensive to setup

• A “Numerical Grade” is produced for comparision

14

COMPARABLE RESULTS
Original AjaxSolr WebSolr

gata 4 40.0 80.0 80.0
HIF 1 alpha 90.0 100.0 30

HIF-1 alpha 100.0 100.0 30

HIF1 alpha 70.0 100.0 100.0

HIF1A 100.0 100.0 100.0
iNOS 60.0 60.0 60.0
IRE-1 alpha 30 90.0 90.0
17

Search
Stack

User
Interface

Search
Engine

Informa3on

19

A HYBRID APPROACH?

Humans

Automated
Tests

22

A HYBRID APPROACH?

Tool Supported
Testing Humans
Automated
Tests
Judgements
23

TESTING IS A HUMAN ACTIVITY

James Bach,
satisfice.com

24

THANK YOU!

• twitter: dep4b

• speakerrate: http://www.speakerrate.com/epugh/

• email: epugh@opensourceconnections.com

25
25

Empfohlen

Better Search Engine TestingOpenSource Connections

A Dickens of A KeynoteJames Gray

DevOps Army of One - DevOps Days Silicon Valley 2015funjon

Building a Successful Internal Adversarial Simulation Team - Chris Gates & Ch...Chris Gates

Going Purple : From full time breaker to part time fixer: 1 year later Chris Gates

SEERS - Standardised Bug Reportingbooshtukka

Adversarial Simulation Nickerson/Gates Wild West Hacking Fest Oct 2017Chris Gates

Your Automated Execution Does Not Have to be FlakyAlan Richardson

Empfohlen

Better Search Engine TestingOpenSource Connections

A Dickens of A KeynoteJames Gray

DevOps Army of One - DevOps Days Silicon Valley 2015funjon

Building a Successful Internal Adversarial Simulation Team - Chris Gates & Ch...Chris Gates

Going Purple : From full time breaker to part time fixer: 1 year later Chris Gates

SEERS - Standardised Bug Reportingbooshtukka

Adversarial Simulation Nickerson/Gates Wild West Hacking Fest Oct 2017Chris Gates

Your Automated Execution Does Not Have to be FlakyAlan Richardson

Better Search Engine Testing - Eric Pughlucenerevolution

So we're going no-QA - how do we get the devs to do enough testing?Steve Wells

Practical Security Testing for Developers using OWASP ZAP at Dot Net Bangalor...gmaran23

10 Useful Testing Tools for Open Source Projects @ TuxCon 2015Peter Sabev

Shawn Wallace - Test automation in brownfield applicationsQA or the Highway

Sandbox vs manual malware analysis v1.1Michael Gough

5 cro tools that i can't live withoutCraig Sullivan

Open source-secret-sauce-rit-2010Ted Husted

Evaluative ResearchNatalie Verter

Cues for Better Scent in DebuggingRui Maranhao Abreu

Web Application Software TestingAndrew Kandels

Lessons Learned in Test Automation From ZombielandMatt Barbour

Testing in a Continuous Delivery World - LondonCD Meetup - May 2014Amy Phillips

Jesse Hull - LOGIN Conference 2012Login Conference LLC

Chaining the Beast - Testing Wonder Applications in the Real WorldWO Community

Introduction to atddShawn Wallace

Agile Testing Days 2018 USA - API Testing FundamentalsJoEllen Carter

My Little Webap - DevOpsSec is MagicApollo Clark

JavaOne 2014 Security Testing for Developers using OWASP ZAPSimon Bennetts

Natural born conversion killers - Conversion JamCraig Sullivan

EncoresOpenSource Connections

Test driven relevancyOpenSource Connections

Weitere ähnliche Inhalte

Ähnlich wie Better search engine testing for solr

Better Search Engine Testing - Eric Pughlucenerevolution

So we're going no-QA - how do we get the devs to do enough testing?Steve Wells

Practical Security Testing for Developers using OWASP ZAP at Dot Net Bangalor...gmaran23

10 Useful Testing Tools for Open Source Projects @ TuxCon 2015Peter Sabev

Shawn Wallace - Test automation in brownfield applicationsQA or the Highway

Sandbox vs manual malware analysis v1.1Michael Gough

5 cro tools that i can't live withoutCraig Sullivan

Open source-secret-sauce-rit-2010Ted Husted

Evaluative ResearchNatalie Verter

Cues for Better Scent in DebuggingRui Maranhao Abreu

Web Application Software TestingAndrew Kandels

Lessons Learned in Test Automation From ZombielandMatt Barbour

Testing in a Continuous Delivery World - LondonCD Meetup - May 2014Amy Phillips

Jesse Hull - LOGIN Conference 2012Login Conference LLC

Chaining the Beast - Testing Wonder Applications in the Real WorldWO Community

Introduction to atddShawn Wallace

Agile Testing Days 2018 USA - API Testing FundamentalsJoEllen Carter

My Little Webap - DevOpsSec is MagicApollo Clark

JavaOne 2014 Security Testing for Developers using OWASP ZAPSimon Bennetts

Natural born conversion killers - Conversion JamCraig Sullivan

Ähnlich wie Better search engine testing for solr (20)

Better Search Engine Testing - Eric Pugh

So we're going no-QA - how do we get the devs to do enough testing?

Practical Security Testing for Developers using OWASP ZAP at Dot Net Bangalor...

10 Useful Testing Tools for Open Source Projects @ TuxCon 2015

Shawn Wallace - Test automation in brownfield applications

Sandbox vs manual malware analysis v1.1

5 cro tools that i can't live without

Open source-secret-sauce-rit-2010

Evaluative Research

Cues for Better Scent in Debugging

Web Application Software Testing

Lessons Learned in Test Automation From Zombieland

Testing in a Continuous Delivery World - LondonCD Meetup - May 2014

Jesse Hull - LOGIN Conference 2012

Chaining the Beast - Testing Wonder Applications in the Real World

Introduction to atdd

Agile Testing Days 2018 USA - API Testing Fundamentals

My Little Webap - DevOpsSec is Magic

JavaOne 2014 Security Testing for Developers using OWASP ZAP

Natural born conversion killers - Conversion Jam

Mehr von OpenSource Connections

EncoresOpenSource Connections

Test driven relevancyOpenSource Connections

How To Structure Your Search Team for SuccessOpenSource Connections

The right path to making search relevant - Taxonomy Bootcamp London 2019OpenSource Connections

Payloads and OCR with SolrOpenSource Connections

Haystack 2019 Lightning Talk - The Future of Quepid - Charlie HullOpenSource Connections

Haystack 2019 Lightning Talk - State of Apache Tika - Tim AllisonOpenSource Connections

Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...OpenSource Connections

Haystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj BharadwajOpenSource Connections

Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...OpenSource Connections

Haystack 2019 - Search-based recommendations at Politico - Ryan KohlOpenSource Connections

Haystack 2019 - Search with Vectors - Simon HughesOpenSource Connections

Haystack 2019 - Natural Language Search with Knowledge Graphs - Trey GraingerOpenSource Connections

Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...OpenSource Connections

Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...OpenSource Connections

Haystack 2019 - Architectural considerations on search relevancy in the conte...OpenSource Connections

Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...OpenSource Connections

Haystack 2019 - Establishing a relevance focused culture in a large organizat...OpenSource Connections

Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...OpenSource Connections

2019 Haystack - How The New York Times Tackles Relevance - Jeremiah ViaOpenSource Connections

Mehr von OpenSource Connections (20)

Encores

Test driven relevancy

How To Structure Your Search Team for Success

The right path to making search relevant - Taxonomy Bootcamp London 2019

Payloads and OCR with Solr

Haystack 2019 Lightning Talk - The Future of Quepid - Charlie Hull

Haystack 2019 Lightning Talk - State of Apache Tika - Tim Allison

Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...

Haystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj Bharadwaj

Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...

Haystack 2019 - Search-based recommendations at Politico - Ryan Kohl

Haystack 2019 - Search with Vectors - Simon Hughes

Haystack 2019 - Natural Language Search with Knowledge Graphs - Trey Grainger

Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...

Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...

Haystack 2019 - Architectural considerations on search relevancy in the conte...

Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...

Haystack 2019 - Establishing a relevance focused culture in a large organizat...

Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...

2019 Haystack - How The New York Times Tackles Relevance - Jeremiah Via

Kürzlich hochgeladen

Understanding the Laravel MVC ArchitecturePixlogix Infotech

#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard

A Domino Admins Adventures (Engage 2024)Gabriella Davis

Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik

04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG

Key Features Of Token Development (1).pptxLBM Solutions

Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies

The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los

Install Stable Diffusion in windows machinePadma Pradeep

Scaling API-first – The story of a global engineering organizationRadu Cotescu

How to Remove Document Management Hurdles with X-Docs?XfilesPro

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

Kürzlich hochgeladen (20)

Understanding the Laravel MVC Architecture

#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Maximizing Board Effectiveness 2024 Webinar.pptx

A Domino Admins Adventures (Engage 2024)

Integration and Automation in Practice: CI/CD in Mule Integration and Automat...

08448380779 Call Girls In Civil Lines Women Seeking Men

[2024]Digital Global Overview Report 2024 Meltwater.pdf

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Injustice - Developers Among Us (SciFiDevCon 2024)

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

Key Features Of Token Development (1).pptx

Benefits Of Flutter Compared To Other Frameworks

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Install Stable Diffusion in windows machine

Scaling API-first – The story of a global engineering organization

How to Remove Document Management Hurdles with X-Docs?

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi

Handwritten Text Recognition for manuscripts and early printed texts

Better search engine testing for solr

1. BETTER SEARCH ENGINE TESTING FOR SOLR CVILLE MEETUP | EPUGH@O19S.COM | @DEP4B 1

2. WHY AM I QUALIFIED TO BE UP HERE? • Principal of OpenSource Connections - Solr Search Consultancy • Member of Apache Software Foundation • Presenter at conferences (OSCON, ApacheCON, UberConf, ExpoQA, STPcon 2009/2011!) • Fascinated by the art of software development 2

3. 3. X Co So m on ing AUTHOR 3

4. FATHER 4

5. AGENDA Why Test Search? Two General Approaches Test Tools for Solr Wrap Up 5

6. REDUCE HUMAN ERROR 6

7. photo credit to Jay Grieves @ www.bigswingingdeveloper.com 7

8. TWO APPROACHES TO TEST Humans Automated Tests 8

9. HUMAN TESTING • Better deals with ambiguity 9

10. RELEVANCY IS LIKE PORNOGRAPHY.... 10

11. I KNOW IT WHEN I SEE IT! http://en.wikipedia.org/wiki/Les_Amants 11

12. AUTOMATED TESTING 12

13. SWEET SPOT • Repeat Testing for Relevancy Tuning • Performance Testing: hitting SLA? • Regression Testing (Typical use case for automated testing) 13

14. “ONE TRUE ANSWER” • Absolute Truth / Matrix / Grid / TREC / Relevancy Assertions • The correct answers for each search are known ahead of time • Humans judges often decide these correct answers, stored as Relevancy Assertions • Can be labor intensive to setup • A “Numerical Grade” is produced for comparision 14

15. SO WHAT IS A JUDGEMENT LIST? 15

16. 16

17. COMPARABLE RESULTS Original AjaxSolr WebSolr gata 4 40.0 80.0 80.0 HIF 1 alpha 90.0 100.0 30 HIF-1 alpha 100.0 100.0 30 HIF1 alpha 70.0 100.0 100.0 HIF1A 100.0 100.0 100.0 iNOS 60.0 60.0 60.0 IRE-1 alpha 30 90.0 90.0 17

18. WRAP UP 18

19. Search Stack User Interface Search Engine Informa3on 19

20. 20

21. 21

22. A HYBRID APPROACH? Humans Automated Tests 22

23. A HYBRID APPROACH? Tool Supported Testing Humans Automated Tests Judgements 23

24. TESTING IS A HUMAN ACTIVITY James Bach, satisfice.com 24

25. THANK YOU! • twitter: dep4b • speakerrate: http://www.speakerrate.com/epugh/ • email: epugh@opensourceconnections.com 25 25