Building natural language processing engines is really a complex work. It requires an architecture where we glue many algorithms, data processing, and data storage techniques together to solve a single most important problem—how effectively we can understand users query in the form of text, voice, or visual gestures fast and effectively and respond them with zero error. Identifying the best tools available in the market and how we can fit these tools and libraries in our pipelines makes a great edge to build these systems. In this talk, I will give a deep knowledge of how we’re using Apache Spark in our NLU engine to solve a complex problem with ease. At the end of the talk, attendees will get a high-level architecture overview of our NLU engine, some TensorFlow introduction, and how they can use Apache Spark in ML and deep learning system.

1. Rahul Kumar, I.AM Plus Electronics
Powering NLU Engine with Apache Spark
to Communicate with World

2. Agenda
• NLP platform features
• NLP Architecture
• NLP building blocks
• Deep learning for NLP
• TensorFlow Overview
2

3. NLP Platform Features
• Text & Voice based interaction
• 50K + Music
• Email / Calendar Integration
• News + weather
• Open API for various Skills
3

4. Platform Architecture
4

5. Core Building Blocks
• NLP/NLU engine
• Skills repository
• Data Processing & Analyzing
• User Interaction medium
5

6. Workflow
6

7. 7

8. Data Challenges
• Heterogenous data type
• Streaming + Batch data
• Data Cleaning, enrichments
• Data Annotation
8

9. Spark MLlib
• Classification and
Regression
• Clustering
• Collaborative filtering
• Frequent Pattern
Mining
9

10. Data Pipeline on Scale
10

11. What is Deep learning?
11

12. 12

13. 13

14. Word embedding
• Word embedding is the collective name for a
set of language modeling and feature learning
techniques in natural language processing
(NLP) where words or phrases from the
vocabulary are mapped to vectors of real
numbers
14

15. Word embedding model
• Word2Vec model
• Glove model
• fasttext model
15

16. Idea for word2vec model
• Word similarity = vector similarity
• Key idea: Predict surrounding words of every word
• Faster and can easily incorporate a new
sentence/document or add a word to the vocabulary
• Developed by Mikolov, Sutskever, Chen, Corrado and
Dean in 2013 at Google Research
16

17. Representation
• Word meaning and relationships between
words are encoded spatially
• Spatial distance corresponds to word similarity
words are close together ⇔ their "meanings"
are similar notation: word w → vec[w] its point
in space, as a position vector.
17

18. Similar words are closer together
18

19. Word relationships are displacements
• The displacement (vector) between the points
of two words represents the word relationship.
• Same word relationship ⇒ same vector
19

20. Architecture
20

21. • TensorFlow is an open-source machine
learning library for research and production
• TensorFlow provides a variety of different
toolkits that allow you to construct models at
your preferred level of abstraction
21

22. TensorFlow architecture
22

23. 23

24. Questions ?
24
https://www.linkedin.com/in/rahulkumar-aws/
@ rahul_kumar_aws