Custom Solr Tokenizer Flexible Tokenizer with JFlex
•Als PPTX, PDF herunterladen•
0 gefällt mir•1,194 views
The document discusses building a custom tokenizer for Apache Solr using JFlex to provide flexible tokenization. It describes the motivation for custom tokenizers when dealing with custom metadata like IDs, types, and descriptions. It then provides examples of use cases and demonstrates how to set up a JFlex tokenizer project, define token patterns in a scanner description, and integrate it with Solr.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie Custom Solr Tokenizer Flexible Tokenizer with JFlex
Ähnlich wie Custom Solr Tokenizer Flexible Tokenizer with JFlex (20)
Mehr von SHI Search | Analytics | Big Data
Mehr von SHI Search | Analytics | Big Data (15)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Custom Solr Tokenizer Flexible Tokenizer with JFlex
- 1. Technology Drives Business CUSTOM SOLR TOKENIZER FLEXIBLE TOKENIZER WITH JFLEX 2014 BerlinBuzzword
- 2. Agenda • ME & SHI • JFLEX Tokenizer • Motivation • JFlex ?! • Solr implementation • Demo • Q & A
- 3. Markus Klose – Search Consultant • Expertise in Solr, Lucene, Elasticsearch, Fast ESP • Certified Apache Solr Trainer • Speaker, Blogger, Coder • Author “Einführung in Apache Solr” • @markus_klose
- 4. SHI GmbH & Co KG 2013 2011 Delivering mission-critical data-driven solution for multiple industries. Partnering with Partnering with LucidWorks 2000 Embracing Open Source. 1994 Foundation. Development of home-grown information retrieval platform. 2014
- 5. OUR MISSION Vendor-independent IT Consulting and Software Engineering company. Dedicated to deliver next generation Semantic Search, Big Data and Exploratory Data Analytics solutions. Using Enterprise Data Hub approach for 360° data integration. And helping customers to Accelerate (e)Business through better technology adoption and data utilization.
- 6. Technology Drives Business CUSTOM TOKENIZER WITH JFLEX JFlex based tokenizer - the idea is not new, but great
- 7. Motivation 1 • In customer projects we have to deal very often with custom „meta“ data • IDs • Type designation • Product description • How to face that problem? PatternTokenizer?
- 8. Motivation 2 • Use and combine existing tools to be more flexible • Configuration over Coding • JFlex allready used in ClassicTokenizer / StandardTokenizer
- 9. UseCase – Type designation • Product Data • nymj3x1,5 / nym-j 3x1,5 / nymj 3x1,5 / nym-j 3 x 1,5 • Search Input • nymj 3 1,5 / nym-j 3x1,5 • Index • nymj315 / nymj / nym / j / 315 / 3 / 15
- 10. JFlex - The Fast Scanner Generator • JFlex is a lexical analyzer generator (aka scanner generator) • Current version 1.5.1 • Download - http://jflex.de/download.html • Mailing Lists • BSD-style license • CLI API & GUI
- 11. JFlex - The Fast Scanner Generator • Berlin Buzzword 26.05.2014 • LETTERS -> „Berlin“, „Buzzword“ • LETTERS and SPACE -> „Berlin Buzzword“ • DIGITS -> „26“, „05“, „2014“ • DIGITS and . -> „26.05.2014“ • LETTERS and SPACE or DIGITS and . -> „Berlin Buzzword“ , „26.05.2014“
- 12. Custom Tokenizer – Project Setup • JAVA - TokenizerFactory –> typical factory, tokenizer configuration • JAVA - Tokenizer -> base class, token manipulation • JFLEX – Scanner -> description of token patterns • (JAVA – Scanner) -> Generated scanner
- 13. Demo ISBN Tokenizer / URL Tokenizer https://github.com/scherziglu
- 14. Resources • JFlex Tokenizer • GitHub (https://github.com/scherziglu) • Solr Source Code (e.g. ClassicTokenizer) • @markus-klose / @SHIEngineers • JFlex Websites http://jflex.de/ • Q & A
- 15. CONTACT SHI GmbH & Co KG Curt-Frenzel-Str. 12 86167 Augsburg Germany info@shi-gmbh.com +49.821.74 82 633 0 mma@shi-gmbh.com mk@shi-gmh.com @markus_klose dwr@sgi-gmbh.com @SHIEngineers @wrigley_dan
Hinweis der Redaktion
- 9
- 36/840 E+P USB-Kabel 000(VE10) 6-30 3S+1Ö M12FR-3L 1x2
- JFlex is a lexical analyzer generator (also known as scanner generator) for Java(tm), written in Java(tm). It is also a rewrite of the very useful tool JLex which was developed by Elliot Berk at Princeton University. As Vern Paxson states for his C/C++ tool flex: They do not share any code though. JFlex is designed to work together with the LALR parser generator CUP by Scott Hudson, and the Java modification of Berkeley Yacc BYacc/J by Bob Jamison. It can also be used together with other parser generators like ANTLR or as a standalone tool. JFlex has three mailing lists: jflex-announce is low traffic and read-only for announcements of new releases, jflex-users is for help and discussions, and jflex-devel for developer discussions. If you would like to subscribe to either of the first two, please enter your email address below, check the appropriate boxes, and press [subscribe]. For the developer list, see the mailing lists page. Creating java classes based on a grammar that parses input
- Show factory & solrconfig.xml Show Tokenizer -> incrementToken Show JFlex File + Compilation
- Step 1 nur text Step 2 simple kombination Step 3 kompex setup ISBN url protocol://subdomain.site.domain/directory