Presented at Lucene/Solr Revolution 2017

1. SEARCH LIKE %SQL%
INFIX SEARCH IN LUCENE / SOLR / ELASTIC
SEP 15, 2017

2. 2
Talk Title
Speaker Name
Company
SEARCH LIKE %SQL%
Mikhail Khludnev
EPAM

3. 3
• work in Search for 6 years
• Apache Lucene/Solr committer for 2 years
• speak at LuceneRevolution, BerlinBuzzwords
• chief search engineer in EPAM
ABOUT ME

4. 4
WE ARE

5. 5
ESTABLISHED & EXPANDING GLOBAL VERTICALS
Award-winning Wealth Management Platform
Deep Expertise in Current and
Emerging FinTech
Working with 5 of the 10 Largest
Investment Banks
Leading Digital Transformation for
Global Retailers
Working with largest online travel association (OTA)
& largest global hospitality company
Recognized M&E Leader by
Independent Research Analysts
Working with 4 out of the 4 Top Broadcast Networks
and 14 out of the top 30 TV Networks to transform
consumer-driven media
R&D Domain Experts with 700+ Complex
Solutions & Services Supporting the Entire
Drug Discovery Workflow
Working with 9 of the 10 Top
Pharma Companies
24-Year History of Leading
Product Development
Working with 30+ of the top 100 ISVs
FINANCIAL SERVICES TRAVEL & CONSUMER
SOFTWARE & HI-TECHLIFE SCIENCES AND HEALTHCARE
MEDIA & ENTERTAINMENT
EMERGING
Deep Expertise Offers
Innovative Solutions
Working with industries ranging from
Energy and Utilities to Telecom and Automotive

6. 6
• Term and boolean query
• Prefix*
query
• *suffix
query
• *infix*
query
• Approaching
Suggester
• Derivative
Terms
AGENDA

7. 7
• Endeca
• MarkLogic
• FAST, Google Search Appliance
• Sphinx
• Apache Lucene
• Apache Solr
• Elastic
SEARCH ENGINES

8. 8

9. 9

10. 10
CUSTOMER PROFILE
Any comprehensive text
search service
• Patent
• Legal
• Chemistry
• Bioinformatics
• SQL legacy

11. 11
WHERE LIKE %infix% *infix*

12. 12
Business Problem/Opportunity
• Ill searches for *infix*
CHALLENGE
Bank of England

13. 13
Business Problem/Opportunity
• Ill searches for *infix*
CHALLENGE

14. 14
…at all?
Or what’s fast at comparison to it?
WHY IT’S A PROBLEM?

15. 15
text:foo
OR
text:bar
text:foo
AND
text:bar
THESE SEARCHES ARE (CONSIDERED AS) FAST

16. 16
text:foo
OR
text:bar
text:foo
AND
text:bar
O(r)
<<
O(Dall)
r
–
results
Dall
–
all
docs
THESE SEARCHES ARE (CONSIDERED AS) FAST

17. 17
• text:[sci
TO
scj]
• text:sci*
WHY THESE ARE STILL FAST?

18. 18
TERM EXPANSION
• discipline
• luscious
• science
• scilla
• scissors
text:[sci
TO
scj]
text:sci*
text:(science
OR
scilla
OR
scissors)
O(t)+O(r)
t – query terms
r - results

19. 19
TERM EXPANSION
• discipline
• luscious
• science
• scilla
• scissors
text:[sci
TO
scj]
text:sci*
text:(science
OR
scilla
OR
scissors)
O(t)+O(r)

20. 20
PREFIX* SEARCH
24 ms
sci*

21. 21
ms

22. 22
WHAT’S THEN?
• asci
• disci
• discipline
• lemnisci
• luscious
• menisci
text:*sci

23. 23
WHAT’S THEN?
text:*sci
O(Tall)+O(r)
• asci
• disci
• discipline
• lemnisci
• luscious
• menisci
Tall – all terms
r - results

24. 24
*SUFFIX SEARCH
4948 ms
*sci

25. 25
ms

26. 26
0 1000 2000 3000 4000 5000 6000
prefix*
*suffix
RESPONSE TIME, ms

27. 27
text:*sci
ReversedWildcardFilterFactory
0enilpicsid
0icsa
0icsid
0icsinem
0icsinmel
0suoicsul
asci
disci
discipline
lemnisci
luscious
menisci
text:0ics*
WHAT’S THEN? – REVERSE!

28. 28
ReversedWildcardFilterFactory

29. 29

30. 30

31. 31
WHAT’S THEN? – REVERSE!
text:*sci
ReversedWildcardFilterFactory
0enilpicsid/0
0icsa/10
0icsid/20
0icsinem/30
0icsinmel/40
0suoicsul/50
asci/60
disci/70
discipline/80
lemnisci/90
luscious/100
menisci/110

32. 32
ReversedWildcardFilterFactory

33. 33
Well.. Postings
asci/0 8, 9, 10, 14, 18, 23, 24, 26, 31, 35
disci/10 8, 11, 14, 18, 18, 18, 21, 23, 25, 27
discipline/20 4, 5, 6, 6, 9, 13, 13, 14, 18, 22
lemnisci/30 3, 4, 7, 9, 9, 9, 12, 13, 17, 20
luscious/40 3, 3, 5, 9, 9, 12, 14, 19, 23, 28
menisci/50 0, 2, 5, 6, 11, 13, 17, 22, 27

34. 34
Well.. Postings .. ah yeah..
0enilpicsid
0icsa
0icsid
0icsinem
0icsinmel
0suoicsul
asci/0 8, 9, 10, 14, 18, 23, 24, 26, 31, 35
disci/10 8, 11, 14, 18, 18, 18, 21, 23, 25, 27
discipline/20 4, 5, 6, 6, 9, 13, 13, 14, 18, 22
lemnisci/30 3, 4, 7, 9, 9, 9, 12, 13, 17, 20
luscious/40 3, 3, 5, 9, 9, 12, 14, 19, 23, 28
menisci/50 0, 2, 5, 6, 11, 13, 17, 22, 27

35. 35
Well.. Postings .. ah yeah.. (and positions!)
0enilpicsid/0 4, 5, 6, 6, 9, 13, 13, 14, 18, 22
0icsa/10 8, 9, 10, 14, 18, 23, 24, 26, 31, 35
0icsid/20 8, 11, 14, 18, 18, 18, 21, 23, 25, 27
0icsinem/30 0, 2, 5, 6, 11, 13, 17, 22, 27
0icsinmel/40 3, 4, 7, 9, 9, 9, 12, 13, 17, 20
0suoicsul/50 3, 3, 5, 9, 9, 12, 14, 19, 23, 28
asci/60 8, 9, 10, 14, 18, 23, 24, 26, 31, 35
disci/70 8, 11, 14, 18, 18, 18, 21, 23, 25, 27
discipline/80 4, 5, 6, 6, 9, 13, 13, 14, 18, 22
lemnisci/90 3, 4, 7, 9, 9, 9, 12, 13, 17, 20
luscious/100 3, 3, 5, 9, 9, 12, 14, 19, 23, 28
menisci/110 0, 2, 5, 6, 11, 13, 17, 22, 27

36. 36
benchmark
khludnevm$
ant
run-­‐task
-­‐Dtask.alg=conf/index-­‐5m.alg
-­‐
Dtask.mem=1000m
…
[java]
-­‐-­‐>
Round
0-­‐-­‐>1:
solr.server:org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient
-­‐-­‐>org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient
[java]
[java]
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐>
starting
task:
StopSolrServer
[java]
[java]
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐>
Report
sum
by
Prefix
(AddDocs)
and
Round
(1
about
1
out
of
13)
[java]
Operation
round
recsPerRun
elapsedSec
avgUsedMem
avgTotalMem
[java]
AddDocs
0
5000001
1,
100.41
102,215,792
257,425,408
[java]
[java]
Reopen
Times:
[java]
1166
[java]
####################
[java]
###
D
O
N
E
!!!
###
[java]
####################
BUILD
SUCCESSFUL
Total
time:
19
minutes
28
seconds
$
ant
run-­‐task
-­‐Dtask.alg=conf/index-­‐5m-­‐reverse.alg
-­‐Dtask.mem=1000m
[java]
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐>
starting
task:
Rounds
…....
[java]
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐>
Report
sum
by
Prefix
(AddDocs)
and
Round
(1
about
1
out
of
13)
[java]
Operation
round
recsPerRun
rec/s
elapsedSec
avgUsedMem
avgTotalMem
[java]
AddDocs
0
5000001
3,556.96
1,405.69
75,075,400
257,425,408
[java]
[java]
Reopen
Times:
[java]
3114
[java]
####################
[java]
###
D
O
N
E
!!!
###
[java]
####################
BUILD
SUCCESSFUL
Total
time:
26
minutes
50
seconds
$
du
-­‐hs
../example/schemaless/solr/gettingstarted/data/*
28G
../example/schemaless/solr/gettingstarted/data/index-­‐reverse
13G
../example/schemaless/solr/gettingstarted/data/index-­‐simple

37. 37
0
5
10
15
20
25
30
Baseline (5M en
wiki)
Reversed
Main Index
INDEX SIZE, GB
13G
28G

38. 38

39. 39

40. 40
text:*sci*
AND THEN

41. 41
discipline
EdgeNGramFilter + ReversedWildcardFilter
EdgeNGram Sort
discipline cipline
iscipline discipline
scipline e
cipline ine
ipline ipline
pline iscipline
line line
ine ne
ne pline
e scipline*sci* -> sci*

42. 42
asci
ci
cious
cipline
disci
discipline
e
emnisci
enisci
i
ine
ious
ipline
isci
iscipline
lemnisci
line
luscious
menisci
mnisci
ne
nisci
ous
pline
s
sci
scious
scipline
us
uscious

43. 43
0
5
10
15
20
25
30
35
Baseline (5M en
wiki)
Reversed EdgeNGramm
Main Index
INDEX SIZE, GB
13G
28G
~60G

44. 44
https://discuss.codechef.com/questions/21385/a-tutorial-on-suffix-arrays
https://issues.apache.org/jira/browse/SOLR-9974
http://labs.carrotsearch.com/jsuffixarrays.html
SUFFIX ARRAY

45. 45
SUGGESTER

46. 46
AnalyzingInfixSuggester
LUCENE-3922: Add Japanese Kanji number normalization to Kuromoji
SOLR-4945: Japanese Autocomplete and Highlighter broken
4945
autocomplete
broken
highlighter
japanese
solr

47. 47
AnalysingInfixSuggester TO RESCUE!

48. 48

49. 49
AnalysingInfixSuggester FOR infix SEARCH
• feed AnalysingInfixSuggester with main index’s terms
• enable EdgeNGramFilter for AnalysingInfixSuggester
discipline
iscipline
scipline
cipline
ipline
pline
line
ine
ne
e

50. 50
• 14 M terms -> 79 M EdgeNGramms
• 10 min
• 3.3 G (25%)
BUILDING SUGGESTER INDEX
discipline
iscipline
scipline
cipline
ipline
pline
line
ine
ne
e

51. 51
ms

52. 52
0
5
10
15
20
25
30
35
Baseline (5M en
wiki)
Reversed EdgeNGramm Suggester
Main Index
Suggester Index
INDEX SIZE, GB
13G
28G
13G+3.3G
~60G

53. 53
http://localhost:8901/solr/gettingstarted/suggest?
suggest.dictionary = body_txt_en & suggest.q = sci
<response>
<lst name="responseHeader"><int
name="QTime">4</int>
</lst>
<lst name="suggest">
<lst name="body_txt_en">
<lst name="sci">
<int name="numFound">1000</int>
<arr name="suggestions">
<str>scienc</str>
<str>scientif</str>
<str>scientist</str>
<str>disciplin</str>
<str>sci</str>
<str>conscious</str>
<str>category:sci</str>
<str>fascin</str>
<str>discipl</str>
<str>consciou</str>
<str>unconsci</str>
<str>conscienc</str>
<str>oscil</str>
<str>neurosci</str>
<str>interdisciplinari</str>
<str>disciplinari</str>
<str>scissor</str>
<str>ascii</str>
<str>scientolog</str>
<str>scimitar</str>
<str>conscienti</str>
<str>pseudosci</str>
<str>rescind</str>
<str>priscilla</str>
<str>subconsci</str>
<str>brescia</str>
<str>scion</str>
<str>category:scientif</str>
<str>infobox_scientist</str>
<str>www.newscientist.com</str>
<str>resuscit</str>
<str>plebiscit</str>
<str>user:scimitar</str>
<str>multidisciplinari</str>
<str>fascia</str>
<str>scifi</str>
<str>geoscienc</str>
<str>www.scifi.com</str>
<str>www.sciencemag.org</str>
<str>omnisci</str>
<str>scipio</str>
<str>neuroscientist</str>
<str>scientologist</str>
….
<int
name="QTime">4</int>

54. 54
AnalysingInfixSuggester FOR infix SEARCH
• feed AnalysingInfixSuggester with main index’s terms
• enable EdgeNGramFilter for AnalysingInfixSuggester
• override wildcard expansion by calling AnalysingInfixSuggester

55. 55
*infix* SEARCH
3834 ms
*sci*
142 ms
*sci*

56. 56

57. 57

58. 58

59. 59
0 1000 2000 3000 4000 5000 6000
prefix*
*suffix
*substr*
*suggester*
RESPONSE TIME, ms

60. 60
AnalysingInfixSuggester FOR infix SEARCH
• existing scalable algorithm
• minor customization
• no postings explosion
• potentially supports NRT

61. 61
discipline
discipline
Suggester
asci
ci
cious
cipline
..
sci
scious
scipline
us
uscious
discipline
asci
disci
discipline
lemnisci
luscious
menisci School discipline
.. is a required set of
actions by a teacher
towards a student …
Main Index

62. 62
discipline
discipline
Derivative Terms
asci
ci
cious
cipline
..
sci
scious
scipline
us
uscious
discipline
asci
disci
discipline
lemnisci
luscious
menisci School discipline
.. is a required set of
actions by a teacher
towards a student …

63. 63
• A slight index format change
• many terms refer to the same postings list
• API is :
• indexWriter.deriveTerms(“name”, “name_edge”, new EdgeNgrammTokenFilter());
• search: name_edge:sci*
• Hijacking and Injecting codecs LUCENE-7863
• Promising for deep taxonomies.
Derivative terms

64. 64
*INFIX* SEARCH WITH DERIVATIVE TERMS
127 ms
*sci*

65. 65
ms

66. 66
0 1000 2000 3000 4000 5000 6000
prefix*
*suffix
*substr*
*suggester*
*derived*
RESPONSE TIME, ms

67. 67
0
5
10
15
20
25
30
35
Baseline (5M en
wiki)
Reversed EdgeNGramm Suggester Derived Terms
Main Index
Suggester Index
INDEX SIZE, GB
13G
28G
13G+3.3G 17G
~60G

68. 68
REFERENCES
What is in a Lucene index? Adrien Grand
https://www.youtube.com/watch?v=T5RmMNDR5XI
Automata Invasion. Robert Muir, Michael Mccandless
https://www.youtube.com/watch?v=pd2jvy2IbJE
• Lucene Search Essentials: Scorers, Collectors and Custom Queries, Mikhail Khludnev
https://www.youtube.com/watch?v=X9YovpYj6uo
A new Lucene suggester based on infix matches
http://blog.mikemccandless.com/2013/06/a-new-lucene-suggester-based-on-
infix.html

69. 69
REFERENCES
What is in a Lucene index? Adrien Grand
https://www.youtube.com/watch?v=T5RmMNDR5XI
Automata Invasion. Robert Muir, Michael Mccandless
https://www.youtube.com/watch?v=pd2jvy2IbJE
В поисках Tommy Hilfiger, Михаил Хлуднев
https://www.youtube.com/watch?v=Azf4oUL-Dqc
A new Lucene suggester based on infix matches
http://blog.mikemccandless.com/2013/06/a-new-lucene-suggester-based-on-infix.html

70. 70
ANYWAY
text:*a*

71. 71
CONTACTS
Mikhail_Khludnev@EPAM.COM
mkhl@apache.or g
https://plus.google.com/+MikhailKhludnev

72. 72
Thank YouThank You