Deep learning for NLP and Transformer

1. Deep learning basics for NLP
Arvind Devaraj (Grouply.org)
Lead Data Scientist, Embibe

2. Agenda
Classical ML vs Deep Learning
What is Deep learning for NLP
Learning Resources
How Machine translation (NMT) works

3. ML Tools
Scikit-learn
Jupyter
Pandas
TensorFlow
Google colab
Nltk/spacy

4. ML Courses
MLcourse.Ai
Cs231n (stanford)
Fast.Ai
Deeplearning.Ai
Coursera(nlp)
Analyticsvidya
Edureka
Simplilearn
Siraj Raval

5. Youtube channels and Books
KrisNaik(deployment)
StatQuest
DeepLearning.TV
CodeEmporium
3blue1brown
Bob Trenwith
Data School
Cognitive Class
The Hundred-page Machine Learning Book
Hands on ML with scikit learn, keras and
tensorflow
Deep learning (Ian Goodfellow)
Refer Grouply.org

6. ML Types

7. ML Algorithms
1. Linear Regression
2. Logistic Regression
3. Decision Trees
4. Random Forest
5. Clustering (K means, Hierarchical)

8. Machine learning skills
1. Math skills
Probability and statistics
Linear Algebra
Calculus
2. Programming in Python/R
3. Data Engineer skills
1. Ability to work with large amounts of data
2. Data preprocessing
3. Knowledge of SQL and NoSQL
4. ETL (Extract Transform and Load) operations
5. Data Analysis and Visualization skills

9. Project ideas
NLP
Sentiment analysis
Spam classifier
Text classifier (Tagging documents)
Smartbook
Kaggle

10. Classical machine learning
More info

12. Deep learning for NLP

13. Introduction to Deep learning
Difference between classical ML and DL
Architectures - MLP, CNN, RNN
Frameworks - Tensorflow, Keras, Pytorch

14. Model 1
Neural Networks

15. Neural Network
Neural Networks act as a ‘black box’ that takes inputs and predicts an output.

16. Neural Network

17. Neural Network Training

18. Basic Neural Network

19. Neural Network

20. Neural Network
‘Black box’ that takes inputs and predicts an output.
Trained using known (input,output) , approximates the function and maps new inputs
Learns the function mapping inputs to output by adjusting the internal parameters (weights)

21. Word2vec using neural networks

22. Word2vec using neural networks

23. Word2vec using neural networks

24. Types of Neural Nets

25. Model 2
RNN

26. RNN
RNNs are used when the inputs have some state information
Examples include time series and word sequences
Can captures the essence of the sequence in its hidden state (context)
https://towardsdatascience.com/illustrated-guide-to-recurrent-neural-networks-79e5eb8049c9

27. RNN - unrolled in time

28. RNN

29. RNN can learn sentence representations

30. RNNs can predict
RNN can learn a probability function p( word | previous-words )

31. RNN - Summary
RNN can encode sentence meaning
Can predict next word given a sequence of words.
Can be trained to learn a language model

32. Translation
Two RNNs jointly trained (one using English, another with German)
Training using Parallel corpora

33. Models
1. Neural Networks
2. Recurrent Neural Networks
3. Encoder-Decoder
4. Attention Models
5. Transformer

34. Model 3
Encoder Decoder

35. Encoder Decoder
http://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/

36. Encoder Decoder

37. Encoder-Decoder in Translation

38. Encoder - Decoder summary
2014 -Google replaced their Statistical model with NMT
Due to its flexibility it is the goto framework for NLG with different models taking
roles of encoder and decoder
The decoder can not only be conditioned on a sequence but on arbitrary
representation enabling lot of use cases (like generating caption from image)

39. Translation using Encoder-Decoder
Issues
● Meaning is crammed
● Long term dependencies
● Word alignment
● Word interdependencies

40. Translation issues (long term dependencies)
The model cannot remember enough
difficult to retain >30 words.
"Born in France..went to EPFL Switzerland..I speak fluent ..."

41. Pay selective attention

42. Pay selective attention
First introduced in Image Recognition

43. Translation with Attention

44. Translation with transformer

45. Machine translation
NN can encode words
RNN can encode sentences
Long sentences need changes to RNN architecture
Two RNNs can act as encoder and decoder (of any representation)
Encoding everything into single context loses information
Selectively pay attention to the inputs we need.
Get rid of RNN and use only attention mechanism. Make it parallelizable
Richer representation of inputs using Self-attention.
Use Encoder-Decoder attention as usual for translation

47. Word2vec to BERT