Better search engine testing for solr

•

0 gefällt mir•1,787 views

A refactoring of my earlier presentation targeted more towards Solr, and less on testing. I'm basing my presentation for Lucene EuroCon on this. Would love some more feedback!

Technologie Design

BETTER SEARCH
ENGINE TESTING
FOR SOLR

CVILLE MEETUP | EPUGH@O19S.COM | @DEP4B

1

WHY AM I QUALIFIED TO BE
UP HERE?
• Principal of OpenSource
Connections - Solr Search
Consultancy

• Member of Apache Software
Foundation

• Presenter at conferences (OSCON,
ApacheCON, UberConf, ExpoQA,
STPcon 2009/2011!)

• Fascinated by the art of software
development
2

AGENDA

Why Test Search?

Two General Approaches

Test Tools for Solr

Wrap Up

5

photo credit to Jay Grieves @ www.bigswingingdeveloper.com
7

TWO APPROACHES TO TEST

Humans

Automated
Tests

8

HUMAN TESTING

• Better deals with ambiguity

9

I KNOW IT WHEN I SEE IT!

http://en.wikipedia.org/wiki/Les_Amants

11

SWEET SPOT

• Repeat Testing for Relevancy Tuning

• Performance Testing: hitting SLA?

• Regression Testing (Typical use case for automated testing)

13

“ONE TRUE ANSWER”

• Absolute Truth / Matrix / Grid / TREC / Relevancy Assertions

• The correct answers for each search are known ahead of
time

• Humans judges often decide these correct answers, stored
as Relevancy Assertions

• Can be labor intensive to setup

• A “Numerical Grade” is produced for comparision

14

COMPARABLE RESULTS
Original AjaxSolr WebSolr

gata 4 40.0 80.0 80.0
HIF 1 alpha 90.0 100.0 30

HIF-1 alpha 100.0 100.0 30

HIF1 alpha 70.0 100.0 100.0

HIF1A 100.0 100.0 100.0
iNOS 60.0 60.0 60.0
IRE-1 alpha 30 90.0 90.0
17

Search
Stack

User
Interface

Search
Engine

Informa3on

19

A HYBRID APPROACH?

Humans

Automated
Tests

22

A HYBRID APPROACH?

Tool Supported
Testing Humans
Automated
Tests
Judgements
23

TESTING IS A HUMAN ACTIVITY

James Bach,
satisfice.com

24

THANK YOU!

• twitter: dep4b

• speakerrate: http://www.speakerrate.com/epugh/

• email: epugh@opensourceconnections.com

25
25

Empfohlen

Better Search Engine TestingOpenSource Connections

A Dickens of A KeynoteJames Gray

DevOps Army of One - DevOps Days Silicon Valley 2015funjon

Building a Successful Internal Adversarial Simulation Team - Chris Gates & Ch...Chris Gates

Going Purple : From full time breaker to part time fixer: 1 year later Chris Gates

SEERS - Standardised Bug Reportingbooshtukka

Adversarial Simulation Nickerson/Gates Wild West Hacking Fest Oct 2017Chris Gates

Your Automated Execution Does Not Have to be FlakyAlan Richardson

Empfohlen

Better Search Engine TestingOpenSource Connections

A Dickens of A KeynoteJames Gray

DevOps Army of One - DevOps Days Silicon Valley 2015funjon

Building a Successful Internal Adversarial Simulation Team - Chris Gates & Ch...Chris Gates

Going Purple : From full time breaker to part time fixer: 1 year later Chris Gates

SEERS - Standardised Bug Reportingbooshtukka

Adversarial Simulation Nickerson/Gates Wild West Hacking Fest Oct 2017Chris Gates

Your Automated Execution Does Not Have to be FlakyAlan Richardson

Better Search Engine Testing - Eric Pughlucenerevolution

So we're going no-QA - how do we get the devs to do enough testing?Steve Wells

Practical Security Testing for Developers using OWASP ZAP at Dot Net Bangalor...gmaran23

10 Useful Testing Tools for Open Source Projects @ TuxCon 2015Peter Sabev

Shawn Wallace - Test automation in brownfield applicationsQA or the Highway

Sandbox vs manual malware analysis v1.1Michael Gough

5 cro tools that i can't live withoutCraig Sullivan

Open source-secret-sauce-rit-2010Ted Husted

Evaluative ResearchNatalie Verter

Cues for Better Scent in DebuggingRui Maranhao Abreu

Web Application Software TestingAndrew Kandels

Lessons Learned in Test Automation From ZombielandMatt Barbour

Testing in a Continuous Delivery World - LondonCD Meetup - May 2014Amy Phillips

Jesse Hull - LOGIN Conference 2012Login Conference LLC

Chaining the Beast - Testing Wonder Applications in the Real WorldWO Community

Introduction to atddShawn Wallace

Agile Testing Days 2018 USA - API Testing FundamentalsJoEllen Carter

My Little Webap - DevOpsSec is MagicApollo Clark

JavaOne 2014 Security Testing for Developers using OWASP ZAPSimon Bennetts

Natural born conversion killers - Conversion JamCraig Sullivan

EncoresOpenSource Connections

Test driven relevancyOpenSource Connections

Weitere ähnliche Inhalte

Ähnlich wie Better search engine testing for solr

Better Search Engine Testing - Eric Pughlucenerevolution

So we're going no-QA - how do we get the devs to do enough testing?Steve Wells

Practical Security Testing for Developers using OWASP ZAP at Dot Net Bangalor...gmaran23

10 Useful Testing Tools for Open Source Projects @ TuxCon 2015Peter Sabev

Shawn Wallace - Test automation in brownfield applicationsQA or the Highway

Sandbox vs manual malware analysis v1.1Michael Gough

5 cro tools that i can't live withoutCraig Sullivan

Open source-secret-sauce-rit-2010Ted Husted

Evaluative ResearchNatalie Verter

Cues for Better Scent in DebuggingRui Maranhao Abreu

Web Application Software TestingAndrew Kandels

Lessons Learned in Test Automation From ZombielandMatt Barbour

Testing in a Continuous Delivery World - LondonCD Meetup - May 2014Amy Phillips

Jesse Hull - LOGIN Conference 2012Login Conference LLC

Chaining the Beast - Testing Wonder Applications in the Real WorldWO Community

Introduction to atddShawn Wallace

Agile Testing Days 2018 USA - API Testing FundamentalsJoEllen Carter

My Little Webap - DevOpsSec is MagicApollo Clark

JavaOne 2014 Security Testing for Developers using OWASP ZAPSimon Bennetts

Natural born conversion killers - Conversion JamCraig Sullivan

Ähnlich wie Better search engine testing for solr (20)

Better Search Engine Testing - Eric Pugh

So we're going no-QA - how do we get the devs to do enough testing?

Practical Security Testing for Developers using OWASP ZAP at Dot Net Bangalor...

10 Useful Testing Tools for Open Source Projects @ TuxCon 2015

Shawn Wallace - Test automation in brownfield applications

Sandbox vs manual malware analysis v1.1

5 cro tools that i can't live without

Open source-secret-sauce-rit-2010

Evaluative Research

Cues for Better Scent in Debugging

Web Application Software Testing

Lessons Learned in Test Automation From Zombieland

Testing in a Continuous Delivery World - LondonCD Meetup - May 2014

Jesse Hull - LOGIN Conference 2012

Chaining the Beast - Testing Wonder Applications in the Real World

Introduction to atdd

Agile Testing Days 2018 USA - API Testing Fundamentals

My Little Webap - DevOpsSec is Magic

JavaOne 2014 Security Testing for Developers using OWASP ZAP

Natural born conversion killers - Conversion Jam

Mehr von OpenSource Connections

EncoresOpenSource Connections

Test driven relevancyOpenSource Connections

How To Structure Your Search Team for SuccessOpenSource Connections

The right path to making search relevant - Taxonomy Bootcamp London 2019OpenSource Connections

Payloads and OCR with SolrOpenSource Connections

Haystack 2019 Lightning Talk - The Future of Quepid - Charlie HullOpenSource Connections

Haystack 2019 Lightning Talk - State of Apache Tika - Tim AllisonOpenSource Connections

Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...OpenSource Connections

Haystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj BharadwajOpenSource Connections

Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...OpenSource Connections

Haystack 2019 - Search-based recommendations at Politico - Ryan KohlOpenSource Connections

Haystack 2019 - Search with Vectors - Simon HughesOpenSource Connections

Haystack 2019 - Natural Language Search with Knowledge Graphs - Trey GraingerOpenSource Connections

Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...OpenSource Connections

Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...OpenSource Connections

Haystack 2019 - Architectural considerations on search relevancy in the conte...OpenSource Connections

Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...OpenSource Connections

Haystack 2019 - Establishing a relevance focused culture in a large organizat...OpenSource Connections

Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...OpenSource Connections

2019 Haystack - How The New York Times Tackles Relevance - Jeremiah ViaOpenSource Connections

Mehr von OpenSource Connections (20)

Encores

Test driven relevancy

How To Structure Your Search Team for Success

The right path to making search relevant - Taxonomy Bootcamp London 2019

Payloads and OCR with Solr

Haystack 2019 Lightning Talk - The Future of Quepid - Charlie Hull

Haystack 2019 Lightning Talk - State of Apache Tika - Tim Allison

Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...

Haystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj Bharadwaj

Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...

Haystack 2019 - Search-based recommendations at Politico - Ryan Kohl

Haystack 2019 - Search with Vectors - Simon Hughes

Haystack 2019 - Natural Language Search with Knowledge Graphs - Trey Grainger

Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...

Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...

Haystack 2019 - Architectural considerations on search relevancy in the conte...

Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...

Haystack 2019 - Establishing a relevance focused culture in a large organizat...

Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...

2019 Haystack - How The New York Times Tackles Relevance - Jeremiah Via

Kürzlich hochgeladen

Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity

Scale your database traffic with Read & Write split using MySQL RouterMydbops

"ML in Production",Oleksandr BaganFwdays

Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3

(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...AliaaTarek5

TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc

What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina

Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada

The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech

Rise of the Machines: Known As Drones...Rick Flair

Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3

unit 4 immunoblotting technique complete.pptxBkGupta21

The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3

The State of Passkeys with FIDO Alliance.pptxLoriGlavin3

Ryan Mahoney - Will Artificial Intelligence Replace Real Estate AgentsRyan Mahoney

Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited

"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays

Time Series Foundation Models - current state and future directionsNathaniel Shimoni

Gen AI in Business - Global Trends Report 2024.pdfAddepto

SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero

Kürzlich hochgeladen (20)

Dev Dives: Streamline document processing with UiPath Studio Web

Scale your database traffic with Read & Write split using MySQL Router

"ML in Production",Oleksandr Bagan

Digital Identity is Under Attack: FIDO Paris Seminar.pptx

(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

What is DBT - The Ultimate Data Build Tool.pdf

Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024

The Ultimate Guide to Choosing WordPress Pros and Cons

Rise of the Machines: Known As Drones...

Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx

unit 4 immunoblotting technique complete.pptx

The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx

The State of Passkeys with FIDO Alliance.pptx

Ryan Mahoney - Will Artificial Intelligence Replace Real Estate Agents

Ensuring Technical Readiness For Copilot in Microsoft 365

"Debugging python applications inside k8s environment", Andrii Soldatenko

Time Series Foundation Models - current state and future directions

Gen AI in Business - Global Trends Report 2024.pdf

SIP trunking in Janus @ Kamailio World 2024

Better search engine testing for solr

1. BETTER SEARCH ENGINE TESTING FOR SOLR CVILLE MEETUP | EPUGH@O19S.COM | @DEP4B 1

2. WHY AM I QUALIFIED TO BE UP HERE? • Principal of OpenSource Connections - Solr Search Consultancy • Member of Apache Software Foundation • Presenter at conferences (OSCON, ApacheCON, UberConf, ExpoQA, STPcon 2009/2011!) • Fascinated by the art of software development 2

3. 3. X Co So m on ing AUTHOR 3

4. FATHER 4

5. AGENDA Why Test Search? Two General Approaches Test Tools for Solr Wrap Up 5

6. REDUCE HUMAN ERROR 6

7. photo credit to Jay Grieves @ www.bigswingingdeveloper.com 7

8. TWO APPROACHES TO TEST Humans Automated Tests 8

9. HUMAN TESTING • Better deals with ambiguity 9

10. RELEVANCY IS LIKE PORNOGRAPHY.... 10

11. I KNOW IT WHEN I SEE IT! http://en.wikipedia.org/wiki/Les_Amants 11

12. AUTOMATED TESTING 12

13. SWEET SPOT • Repeat Testing for Relevancy Tuning • Performance Testing: hitting SLA? • Regression Testing (Typical use case for automated testing) 13

14. “ONE TRUE ANSWER” • Absolute Truth / Matrix / Grid / TREC / Relevancy Assertions • The correct answers for each search are known ahead of time • Humans judges often decide these correct answers, stored as Relevancy Assertions • Can be labor intensive to setup • A “Numerical Grade” is produced for comparision 14

15. SO WHAT IS A JUDGEMENT LIST? 15

16. 16

17. COMPARABLE RESULTS Original AjaxSolr WebSolr gata 4 40.0 80.0 80.0 HIF 1 alpha 90.0 100.0 30 HIF-1 alpha 100.0 100.0 30 HIF1 alpha 70.0 100.0 100.0 HIF1A 100.0 100.0 100.0 iNOS 60.0 60.0 60.0 IRE-1 alpha 30 90.0 90.0 17

18. WRAP UP 18

19. Search Stack User Interface Search Engine Informa3on 19

20. 20

21. 21

22. A HYBRID APPROACH? Humans Automated Tests 22

23. A HYBRID APPROACH? Tool Supported Testing Humans Automated Tests Judgements 23

24. TESTING IS A HUMAN ACTIVITY James Bach, satisfice.com 24

25. THANK YOU! • twitter: dep4b • speakerrate: http://www.speakerrate.com/epugh/ • email: epugh@opensourceconnections.com 25 25