"TextMining with ElasticSearch", Saskia Vola, CEO at textminers.io

1. TextMining with
ElasticSearch
Saskia Vola | CEO@textminers.io
Big Data Berlin, 2016/12/15

2. TextMining with
ElasticSearch

3. What is TextMining?
Text mining, also referred to as text data mining, roughly equivalent to text analytics, is the process of deriving
high-quality information from text… (https://en.wikipedia.org/wiki/Text_mining)
● Information Extraction
● Sentiment Analysis
● Keyword Extraction
● Classification
● Clustering
● Natural Language Processing
● Information Retrieval

4. What is ElasticSearch?
● Scalable Search- and Analytics engine
● Based on Apache Lucene
● ~ NoSQL database
● Very popular and powerful
● Easy to use (Rest/HTTP)

5. 1. Linguistic Preprocessing
● Tokenization
● Stopword Removal
● Stemming
● Built in for >30 languages

6. 1. Linguistic Preprocessing

7. 2. Language Detection
Plugin: https://github.com/jprante/elasticsearch-langdetect

8. 3. Clustering
Plugin: https://github.com/carrot2/elasticsearch-carrot2
● Clusters your search results on the fly
● Different algorithms (k-means, suffix-tree, lingo*)
● Generates cluster titles
● Works best with many documents

9. 3. Clustering

10. 4. Classification
a) Training data:
● You need an index with minimum 2 fields
○ content - analyzed String (text)
○ category - not_analyzed String (keyword)

11. 4. Classification
a) Classification:
● MoreLikeThis Query:
○ Document = like_text
○ Aggregate categories of top 10 hits with scores
● Eval20News dataset:
○ Recall: 1.0 - Precision: 0.71
○ Recall: 0.12 - Precision: 0.92

12. Thanks for your attention!
saskia.vola@textminers.io