2. AGENDA
Use Cases
Lessons
Learned
WILDHORNMINER
Market
Situation
What is
required from
project
partners and
customers?
What is good,
what not and
what are the
next steps?
What are the
others
doing?
How looks Ontos’
approach of a flexible
text mining by using
Deep Learning?
How to analyze
texts from various
sources and
interlink with
existing knowledge
graphs?
12. September 20162
3. USE CASES
What is required from project partners and customers?
12. September 20163
4. USE CASES FROM LEDS
• Content Augmentation in E-Commerce
• Product descriptions imported from various sources as usually unstructured text
• Much manual work is resource consuming and expensive
http://www.walmart.com/ip/The-Revenant-Blu-ray-Digital-HD/50129277
12. September 20164
5. USE CASES FROM CUSTOMERS
• Brand or competitor monitoring
• What are the people or journalists writing about my brand?
• What are my competitors doing? How is the market changing?
Monitoring of
web sites
news feeds
social media
channels
…
Link with
reports
CRM
Open data
…
12. September 20166
6. USE CASES FROM CUSTOMERS
• Assist criminal investigations in the (dark) web
• More and more organized crime in the web
• Detect entities with their relation and additional facts
But Neumann and McKelvey eventually sold
the business to their landlord Joshua Guttman.
Neumann McKelvey
?
Joshua
Guttman
Business
sold_what
sold_to
a
12. September 20167
7. MAIN REQUIREMENTS
• Detection and classification of entities, relations and facts with good F1
• Long, high quality texts vs. short texts with bad / missing grammar in
multiple languages
• Training not by linguists but domain experts
• Flexible adaption to new domains and contexts
• Many different data sources
12. September 20168
9. MARKET
• MarketsAndMarkets: NLP (all) for 2020
• Market: $ 13,4 bn
• CAGR: ~ 18,4%
• ResearchMOZ: Text Analytics 2015-2019
• CAGR: ~ 16,1%
• Transparency Market Research: Text Analytics 2016–2024
• CAGR: ~ 17,6%
• Driving element: NLP-as-a-Service with CAGR of ~20,2%
http://www.marketsandmarkets.com/PressReleases/natural-language-processing-nlp.asp
http://www.researchmoz.us/global-natural-language-processing-market-2015-2019-report.html
http://www.transparencymarketresearch.com/pressrelease/global-text-analytics-market.htm
12. September 201610
10. MARKET
• ResearchAndMarkets: 2016 Top 10 Information and Communication
Technologies
• 7. Natural Language Processing (NLP)
• Increasing Demand for Automation Drives Growth
• Funding on the Rise - Start-ups Driving Key Innovations for NLP Applications
• High Acceptance of Technology enables the US to Remain on the Top
• Gartner‘s Top 10 Strategic Technology Trends für 2016
• 4 - Information of Everything:
• Access to heterogeneous data sources
• (Semantic) Linking of data items
• 5 - Advanced Machine Learning:
• Deep Learning and Neural Network for NLP
http://www.researchandmarkets.com/research/vd8fr9/2016_top_10
12. September 201611
11. GETTING AN OVERVIEW
• Understood domain by reviewing 30+ tools and frameworks
• Checked scientific prototypes as well as open source tools
• Investigated what other enterprise, companies, and startups are doing
• Defined a set of 15+ criteria to compare and understand
12. September 201613
12. GETTING AN OVERVIEW
• Understood domain by reviewing 30+ tools and frameworks
• Findings
• Startups and huge enterprises usually stick to Deep Learning (DL) high flexibility
• Some existing companies loss market share, e.g., Attensity
• Market saturation in U.S., only less companies in Europe (5)
• More and more NLP-as-a-Service
• Only a few use RDF data, e.g., for output or disambiguation (8)
• Only 8 tools could extract relations and facts
• Only 9 open source tools with available benchmarks
12. September 201614
13. GETTING AN OVERVIEW
• Rough comparison of the main concepts
• Conclusions for Ontos
• Use DL as foundation for text mining tasks
• Combine it with semantic technologies, e.g.
for disambiguation
• External view: NLP-as-a-Service
• Internal view: NLP pipelines in order to
combine different tools / technologies
based on http://www.deeplearningbook.org/contents/intro.html Fig 1.5
12. September 201616
14. MINER
How looks Ontos’ approach of a flexible text mining by using Deep Learning?
12. September 201617
15. MINER - OVERVIEW
Language model generated
from Corpora
Model as input for supervised
step
Supervised Model
Domain or task specific model
for sequence labeling
Trained on specific texts by
domain experts
Large collection of texts in a
given language or “dialect”,
e.g. Social Media / Twitter
Large text corpora Unsupervised Model
12. September 201618
16. MINER – CORPUS CREATION
• 1 corpus per language / dialect required
• the larger and more heterogeneous the better get and model more
contexts for words
• Sources: Common Crawl, Wikipedia, news feeds, domain specific texts,
Twitter, …
12. September 201619
17. MINER – UNSUPERVISED MODEL
• language model that could be reused in multiple domains / contexts
• reuse the concept of word2vec to create word embeddings
Language
Dictionary
Word2Vec Model
Word2Vec
Dictionary
Numeric Corpus
12. September 201620
18. MINER – SUPERVISED MODEL
• Sequence labeling task
• use a bi-directional LSTM
(BLSTM)
• Gated Recurrent Unit (GRU) as
specialized LSTM
• Output layer
• join and normalize
• classification
LEDS is a research project
Word Embeddings
GRU GRU GRU GRU GRU
GRU GRU GRU GRU GRU
Forward GRU
Backward GRU
OUT OUT OUT OUT OUTOutput Layers
0
0
Output
Textual Input
12. September 201621
19. MINER – IMPLEMENTATION
• Torch framework
• matured, efficient GPU support, good packages for neural networks, …
• Scripting with Lua language
• Service implementation with Go because of great integration with Lua
• Integrated in Ontos Eiger workbench
12. September 201622
20. • Create new “projects”
ad-hoc
• Reuse pre-labeled
corpora
• Free definition of
entity types as required
for domain
MINER – IMPLEMENTATION
12. September 201623
21. • List and manage
documents of project
corpora
MINER – IMPLEMENTATION
12. September 201624
23. MINER – IMPLEMENTATION
• English News
• ~ 400 English texts from CoNLL 2003 and news feeds
• 5 entity types defined and manually annotated by 2 experts
• 1st value: correct start of entity
• 2nd value: correct end of entity
Class: Person Organization Product Location Event
Per class F1:....... 0.976 | 0.990 0.932 | 0.962 0.873 | 0.942 0.958 | 0.976 0.891 | 0.928
Per class Recall:... 0.974 | 0.988 0.936 | 0.953 0.845 | 0.926 0.965 | 0.980 0.870 | 0.913
Per class Precision: 0.978 | 0.992 0.928 | 0.971 0.902 | 0.959 0.951 | 0.973 0.914 | 0.944
http://www.cnts.ua.ac.be/conll2003/ner/
12. September 201626
24. MINER – IMPLEMENTATION
• English Twitter
• ~ 2400 tweets from Twitter NLP tools / A. Ritter 2011
• 5 entity types defined, some are combined from original source
Class Person Organisation Place Product Thing
Per class F1:....... 0.797 | 0.883 0.548 | 0.141 0.725 | 0.669 0.195 | 0.207 0.409 | 0.551
Per class Recall:... 0.769 | 0.852 0.487 | 0.085 0.696 | 0.647 0.120 | 0.129 0.337 | 0.529
Per class Precision: 0.828 | 0.918 0.627 | 0.414 0.756 | 0.693 0.516 | 0.514 0.520 | 0.574
https://github.com/aritter/twitter_nlp
12. September 201627
25. WILDHORN
How to analyze texts from various sources and interlink with existing
knowledge graphs?
12. September 201628
26. NLP PIPELINES
• Problem: How to efficiently connect various tools in the data stream?
• Sources
• RSS feed reader, crawler, Twitter, FTP servers, …
• Analytics
• 1 MINER instance per language
• Disambiguate and link to Knowledge graphs
or taxonomies
• Sinks
• RDF stores, Apache Solr, HDFS, Apache Cassandra, …
http://www.computernewsme.com/wp-content/uploads/2011/06/Cable-clutter.jpg12. September 201629
27. NLP PIPELINES
• Best practices – by LinkedIn
• http://www.confluent.io/blog/stream-data-platform-1/
• Apache Kafka!
12. September 201630
32. CURRENT STATUS
• Detection and classification of entities, relations and facts with good F1
• Long, high quality texts vs. short texts with bad / missing grammar in
multiple languages
• Training not by linguists but domain experts
• Flexible adaption to new domains and contexts
• Many different data sources
12. September 201636
33. LESSONS LEARNED
• Neural networks / deep learning provide great concepts & frameworks for
flexible, high quality NLP tasks
• Apache Kafka / Confluent Platform in combination with Apache Spark
good foundation for data streaming and processing
• Disambiguation and linking of entities to taxonomies and Knowledge
Graphs via Semantic Web technologies is a core contribution for data
integration
• Hard to find employees with Deep Learning skills
12. September 201637
34. NEXT STEPS
• Try letter-trigram word hashing to overcome out of dictionary problem of
word2vec algorithm
• Relation and fact extraction in MINER
Adam Neumann is the CEO of super-hot office rental company WeWork, the most valuable startup in New York City.
PersonFirst Name
Sentiment
Entity Type
Adam super-hot
Entity
Adam Neumann
Legend Relation Relation Type
firstName1
firstName WhoHow
positive1
positiveSentiment
WeWork
Organization
ThingWhatHow
a has
A1 A2 A3 A4
Annotation
https://arxiv.org/abs/1608.06757
12. September 201638
35. NEXT STEPS
• Define NLP pipelines
in frontend
• Make MINER
scalable
• Usable search
interface
• Benchmark with
GERBIL
12. September 201639
36. Q & A
Dr. Martin Voigt
Managing Director
Ontos GmbH
D-04319 Leipzig
M: +49 178 40 222 58
E: martin.voigt@ontos.com
Twitter: m_a_r_t_i_n
12. September 201640
