This presentation gives an introduction to the Search Engines. What are they? How do they work? It also has a brief introduction to Solr and Lucene

1. ENTERPRISE SEARCH an introduction

2. Web Search Desktop Search Enterprise Search

3. so what is a Search Engine?

5. Any search application has two major components SEARCH component INDEXING component - of importance to us developers (read headache) - of importance to the users

6. data INDEX FILES is indexed user sends search query receives search results INDEXING component SEARCH component

7. Let’s start with INDEXING

8. is it easy to search here . . .

9. or here . . .

12. so what all needs to be Indexed and Searched ?

13. various FILE FORMATS Text Files HTML PDF MS Word PPT

14. coming from various DATA SOURCES Emails CMS File System Database Web Pages

15. data ( documents ) INDEX FILES user sends search query receives search results Analyzer fed to text that should be indexed removing stop words such as "a" or "the" converting all text to lowercase letters for case-insensitive searching Stemming (A stemming algorithm reduces the words "fishing", "fished", "fish", and "fisher" to the root word, "fish". )- Index Writer tokenized text

16. Document 1: Coffee isn't my cup of tea. Document 2: Chocolate, men, coffee - some things are better rich. INDEX coffee - 1,2 cup - 1 tea - 1 chocolate - 1 men - 1 things - 1 better - 1 rich - 1

17. And now the SEARCH Component

18. data INDEX FILES is indexed user receives search results sends search query search terms

19. Search Request Terms Taxonomy Spelling Index Correct Search Terms + Incorrect Search Terms Search Terms + Related Terms from Taxonomy + Concept IDs Search engine (INDEX) Search results with 1) Actual Location of the result 2) Rank 3) Details 4) Facet Categorization Results’ Page

20. introducing LUCENE

24. Ways of storing fields of any document: Indexed means it is searchable Stored you may chose not to make a field searchable, means the content can be displayed in the search results. Example : “ summary associated with a page ” Tokenized means it is run through an Analyzer , that converts the content into a sequence of tokens

25. introducing SOLR Solr Solr Lucene Index

27. Adding Documents to SOLR

29. Schema.xml field indexing and display definition

31. Solrconfig.xml file defines cache size, faceted field type, request handler customization

33. Search Results http://localhost:8983/solr/select?q=video&start=0&rows=2&fl=name,price

34. Default Parameters http://localhost:8983/solr/select?q=video&start=0&rows=2&fl=name,price param default description q The query start 0 Offset into the list of matches rows 10 Number of documents to return fl * Stored fields to return qt standard Query type; maps to query handler df (schema) Default field to search

35. <response><responseHeader><status>0</status> <QTime>1</QTime></responseHeader> <result numFound=&quot;16173&quot; start=&quot;0&quot;> <doc> <str name=&quot;name&quot;>Apple 60 GB iPod with Video</str> <float name=&quot;price&quot;>399.0</float> </doc> <doc> <str name=&quot;name&quot;>ASUS Extreme N7800GTX/2DHTV</str> <float name=&quot;price&quot;>479.95</float> </doc> </result> </response>

36. Solr Core Lucene Admin Interface Standard Request Handler Disjunction Max Request Handler Custom Request Handler Update Handler Caching XML Update Interface Config Analysis HTTP Request Servlet Concurrency Update Servlet XML Response Writer Replication Schema Search Requests hit here New document to be added here