1. Advanced Relevancy Ranking
Paul Nelson
Chief Architect / Search Technologies

2. 2
Search Technologies Overview
• Formed June 2005
• Over 100 employees and growing
• Over 400 customers worldwide
• Presence in US, Latin America, UK & Germany
• Deep enterprise search expertise
• Consistent revenue growth and profitability
• Search Engine Independent

3. 3
Lucene Relevancy: Simple Operators
• term(A)  TF(A) * IDF(A)
• Implemented with DefaultSimilarity / TermQuery
• TF(A) = sqrt(termInDocCount)
• IDF(A) = log(totalDocsInCollection/(docsWithTermCount+1)) + 1.0
• and(A,B)  A * B
• Implemented with BooleanQuery()
• or(A, B)  A + B
• Implemented with BooleanQuery()
• max(A, B)  max(A, B)
• Implemented with DisjunctionMaxQuery()
3

4. 4
Simple Operators - Example
and
or max
george martha washington custis
0.10 0.20 0.60 0.90
0.1 + 0.2 = 0.30 max(0, 0.9) = 0.90
0.3 * 0.9 = 0.27

5. 5
Less Used Operators
• boost(f, A)  (A * f)
• Implemented with Query.setBoost(f)
• constant(f, A)  if(A) then f else 0.0
• Implemented with ConstantScoreQuery()
• boostPlus(A, B)  if(A) then (A + B) else 0.0
• Implemented with BooleanQuery()
• boostMul(f, A, B)  if(B) then (A * f) else A
• Implemented with BoostingQuery()
5

6. 6
Problem: Need for More Flexibility
• Difficult / impossible to use all operators
• Many not available in standard query parsers
• Complex expressions = string manipulation
• This is messy
• Query construction is in the application layer
• Your UI programmer is creating query expressions?
• Seriously?
• Hard to create and use new operators
• Requires modifying query parsers - yuck
6

7. 7
Solr
Query Processing Language
7
User
Interface
QPL
Engine
Search
QPL
Script

8. 8
Introducing: QPL
• Query Processing Language
• Domain Specific Language for Constructing Queries
• Built on Groovy
• https://wiki.searchtechnologies.com/index.php/QPL_Home_Page
• Solr Plug-Ins
• Query Parser
• Search Component
• “The 4GL for Text Search Query Expressions”
• Server-side Solr Access
• Cores, Analyzers, Embedded Search, Results XML
8

9. 9
Solr Plug-Ins

10. 10
QPL Configuration – solrconfig.xml
<queryParser name="qpl"
class="com.searchtechnologies.qpl.solr.QPLSolrQParserPlugin">
<str name="scriptFile">parser.qpl</str>
<str name="defaultField">text</str>
</queryParser>
<searchComponent name="qplSearchFirst"
class="com.searchtechnologies.qpl.solr.QPLSearchComponent">
<str name="scriptFile">search.qpl</str>
<str name="defaultField">text</str>
<str name="isProcessScript">false</str>
</searchComponent>
Query Parser Configuration:
Search Component Configuration:

11. 11
QPL Example #1
myTerms = solr.tokenize(query);
phraseQ = phrase(myTerms);
andQ = and(myTerms);
return phraseQ^3.0 | andQ^2.0 | orQ;
Tokenize:
Phrase Query:
And Query:
Put It All Together:
orQ = (myTerms.size() <= 2) ? null :
orMin( (myTerms.size()+1)/2, myTerms);
Or Query:

12. 12
Thesaurus Example #2
myTerms = solr.tokenize(query);
thes = Thesaurus.load("thesaurus.xml")
thesQ = thes.expand(0.8f,
solr.tokenizer("text"), myTerms);
return and(thesQ);
Tokenize:
Load Thesaurus: (cached)
Thesaurus Expansion:
Put It All Together:
Original Query: bathroom humor
[or(bathroom, loo^0.8, wc^0.8), or(humor, jokes^0.8)]

13. 13
More Operators
Boolean Query Parser:
pQ = parseQuery("(george or martha) near/5 washington")
Relevancy Ranking Operators:
q1 = boostPlus(query, optionalQ)
q2 = boostMul(0.5, query, optionalQ)
q3 = constant(0.5, query)
Composite Queries:
compQ = and(compositeMax(
["title":1.5, "body":0.8],
"george", "washington"))

14. 14
News Feed Use Case
14
Order Documents Date
1 markets+terms Today
2 markets Today
3 terms Today
4 companies Today
5 markets+terms Yesterday
6 markets Yesterday
7 terms Yesterday
8 companies Yesterday
9 markets, companies older

15. 15
News Feed Use Case – Step 1
markets = split(solr.markets, "s*;s*")
marketsQ = field("markets", or(markets));
terms = solr.tokenize(query);
termsQ = field("body",
or(thesaurus.expand(0.9f, terms)))
compIds = split(solr.compIds, "s*;s*")
compIdsQ = field("companyIds", or(compIds))
Segments:
Terms:
Companies:

16. 16
News Feed Use Case – Step 2
todayDate = sdf.format(c.getTime())
todayQ = field("date_s",todayDate)
c.add(Calendar.DAY_OF_MONTH, -1)
yesterdayDate = sdf.format(c.getTime())
yesterdayQ = field("date_s",yesterdayDate)
Today:
Yesterday:
sdf = new SimpleDateFormat("yyyy-MM-dd")
cal = Calendar.getInstance()

17. 17
News Feed Use Case
17
Order Documents Date
1 markets+terms Today
2 markets Today
3 terms Today
4 companies Today
5 markets+terms Yesterday
6 markets Yesterday
7 terms Yesterday
8 companies Yesterday
9 markets, companies older

18. 18
News Feed Use Case – Step 3
sq1 = constant(4.0, and(marketsQ, termsQ))
sq2 = constant(3.0, marketsQ)
sq3 = constant(2.0, termsQ)
sq4 = constant(1.0, compIdsQ)
subjectQ = max(sq1, sq2, sq3, sq4)
tq1 = constant(10.0, todayQ)
tq2 = constant(1.0, yesterdayQ)
timeQ = max(tq1, tq2)
recentQ = and(subjectQ, timeQ)
Weighted Subject Queries:
Weighted Time Queries:
Put it All Together:
return max(recentQ, or(marketsQ,compIdsQ)^0.01))

19. 19
Embedded Search Example #1
results = solr.search('subjectsCore', or(qTerms), 50)
subjectsQ = or(results*.subjectId)
return field("title", and(qTerms)) | subjectsQ^0.9;
Execute an Embedded Search:
Create a query from the results:
Put it all together:
qTerms = solr.tokenize(qTerms);

20. 20
Embedded Search Example #2
results = solr.search('categories', and(qTerms), 10)
myList = solr.newList();
myList.add("relatedCategories", results*.title);
solr.addResponse(myList)
Execute an Embedded Search:
Create a Solr named list:
Add it to the XML response:
qTerms = solr.tokenize(qTerms);

21. 21
Other Features
• Embedded Grouping Queries
• Oh yes they did!
• Proximity operators
• ADJ, NEAR/#, BEFORE/#
• Reverse Lemmatizer
• Prefers exact matches over variants
• Transformer
• Applies transformations recursively to query trees
21

22. 22
Solr
Query Processing Language
22
User
Interface
QPL
Engine
Search
Data as entered
by user Boolean
Query Expression
QPL
Script
Application
Dev Team
Search Team

23. 23
Solr
QPL: Using External Sources to Build Queries
23
User
Interface
QPL
Engine
Search
QPL
Script
RDBMS
Other
Indexes
Thesaurus

24. CONTACT
Paul Nelson
pnelson@searchtechnologies.com