https://github.com/sergii-gavrylov/ShowAndTell https://github.com/sergii-gavrylov/theano-tutorial-AACIMP2015

1. Word embeddings and
neural language modeling
AACIMP 2015
Sergii Gavrylov

2. Overview
● Natural language processing
● Word representations
● Statistical language modeling
● Neural models
● Recurrent neural network models
● Long short-term memory rnn models

3. Natural language
processing
● NLP mostly works with text data (but its methods could be applied
to music, bioinformatic, speech etc.)
● From the perspective of machine learning NL is a sequence of
variable-length sequences of high-dimensional vectors.

4. Word representation

5. One-hot encoding
V = {zebra, horse, school, summer}

6. One-hot encoding
V = {zebra, horse, school, summer}
v(zebra) = [1, 0, 0, 0]
v(horse) = [0, 1, 0, 0]
v(school) = [0, 0, 1, 0]
v(summer) = [0, 0, 0, 1]

7. One-hot encoding
V = {zebra, horse, school, summer}
v(zebra) = [1, 0, 0, 0]
v(horse) = [0, 1, 0, 0]
v(school) = [0, 0, 1, 0]
v(summer) = [0, 0, 0, 1]
(+) Pros:
Simplicity
(-) Cons:
One-hot encoding can be memory inefficient
Notion of word similarity is undefined with one-hot encoding

8. Distributional
representation
Is there a representation that preserves the similarities of word meanings?
d(v(zebra), v(horse)) < d(v(zebra), v(summer))

9. Distributional
representation
Is there a representation that preserves the similarities of word meanings?
d(v(zebra), v(horse)) < d(v(zebra), v(summer))
“You shall know a word by the company it keeps” -
John Rupert Firth

10. Distributional
representation
clic.cimec.unitn.it/marco/publications/acl2014/lazaridou-etal-wampimuk-acl2014.pdf
“A cute, hairy wampimuk is
sitting on the hands.”

11. Distributional
representation
www.cs.ox.ac.uk/files/6605/aclVectorTutorial.pdf

12. Distributional
representation
www.cs.ox.ac.uk/files/6605/aclVectorTutorial.pdf

13. Distributional
representation
(+) Pros:
Simplicity
Has notion of word similarity
(-) Cons:
Distributional representation can be memory inefficient

14. Distributed
representation
V is a vocabulary
wi
∈ V
v(wi
) ∈ Rn
v(wi
) is a low-dimensional, learnable,
dense word vector

15. Distributed
representation
colah.github.io/posts/2014-07-NLP-RNNs-Representations

16. Distributed
representation
(+) Pros:
Has notion of word similarity
is memory efficient (low dimensional)
(-) Cons:
is computationally intensive

17. Distributed representation as a
lookup table
W is a matrix whose rows are v(wi
) ∈ Rn
v(wi
) returns ith
row of W

18. Statistical language
modeling
A sentence s = (x1
, x2
, … , xT
)
How likely is s?
p(x1
, x2
, … , xT
)
according to the chain rule (probability)

19. n-gram models
n-th order Markov assumption

20. n-gram models
n-th order Markov assumption
bigram model of s = (a, cute, wampimuk, is, on, the, tree, .)
1. How likely does ‘a’ follow ‘<S>’?
2. How likely does ‘cute’ follow ‘a’?
3. How likely does ‘wampimuk’ follow ‘cute’?
4. How likely does ‘is’ follow ‘wampimuk’?
5. How likely does ‘on’ follow ‘is’?
6. How likely does ‘the’ follow ‘on’?
7. How likely does ‘tree’ follow ‘the’?
8. How likely does ‘.’ follow ‘tree’?
9. How likely does ‘<S>’ follow ‘.’?

21. n-gram models
n-th order Markov assumption
bigram model of s = (a, cute, wampimuk, is, on, the, tree, .)
the counts are obtained from a training corpus

22. n-gram models
Issues:
Data sparsity
Lack of generalization:
[ride a horse], [ride a llama]
[ride a zebra]

23. Neural language model
ride
a
lookup table

24. Neural language model
0
0
0
0
1
0
0
0
0
0
0
0
0
0
1
0
ride
a
lookup table

25. Neural language model
0
0
0
0
1
0
0
0
0
0
0
0
0
0
1
0
...
ride
a
lookup table

26. 0
0
0
0
1
0
0
0
0
0
0
0
0
0
1
0
...
ride
a
lookup table Neural language model

27. Neural language model
0
0
0
0
1
0
0
0
0
0
0
0
0
0
1
0
...
ride
a
lookup table
neural network

28. Neural language model
0
0
0
0
1
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
1
0
...
ride
a
lookup table
neural network

29. Neural language model
0
0
0
0
1
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
1
0
...
ride
a
lookup table
neural network
zebra should have
representation similar to
horse and llama
Now we can generalize to
the unseen n-grams
[ride a zebra]

30. Recurrent neural network
models
There is no Markov assumption

31. Recurrent neural network
models
There is no Markov assumption
arxiv.org/pdf/1503.04069v1.pdf

32. Recurrent neural network models
yesterday

33. Recurrent neural network models
yesterday

34. Recurrent neural network models
yesterday

35. Recurrent neural network models
yesterday

36. Recurrent neural network models
yesterday we

37. Recurrent neural network models
yesterday we

38. Recurrent neural network models
yesterday we

39. Recurrent neural network models
yesterday we

40. Recurrent neural network models
yesterday we were

41. Recurrent neural network models
yesterday we were

42. Recurrent neural network models
yesterday we were

43. Recurrent neural network models
yesterday we were

44. Recurrent neural network models
yesterday we were riding a

45. Recurrent neural network models
yesterday we were riding a
horse

46. Recurrent neural network
models
Vanishing/exploding gradient problem
www.jmlr.org/proceedings/papers/v28/pascanu13.pdf
Naïve transition has difficulty in handling long-term dependencies

47. Long short-term memory
rnn models
arxiv.org/pdf/1503.04069v1.pdf

48. Long short-term memory
rnn models
arxiv.org/pdf/1503.04069v1.pdf

49. Long short-term memory
rnn models
arxiv.org/pdf/1503.04069v1.pdf

50. Long short-term memory
rnn models
arxiv.org/pdf/1503.04069v1.pdf

51. Long short-term memory
rnn models
arxiv.org/pdf/1503.04069v1.pdf

52. Long short-term memory
rnn models
arxiv.org/pdf/1503.04069v1.pdf

53. Long short-term memory
rnn models
arxiv.org/pdf/1503.04069v1.pdf

54. Neural
machine
translation
devblogs.nvidia.com/parallelforall/introduction-
neural-machine-translation-with-gpus
devblogs.nvidia.com/parallelforall/introduction-
neural-machine-translation-gpus-part-2
devblogs.nvidia.com/parallelforall/introduction-
neural-machine-translation-gpus-part-3

55. Image captioning
cs.stanford.edu/people/karpathy/cvpr2015.pdf

56. Image captioning
cs.stanford.edu/people/karpathy/cvpr2015.pdf
arxiv.org/pdf/1411.4555v2.pdf

57. Conclusion
CS224d: Deep Learning for Natural Language Processing
cs224d.stanford.edu
● Neural methods provide us with a powerful set of tools for
embedding language.
● They provide better ways of tying language learning to extra-
linguistic contexts (images, knowledge-bases, cross-lingual data).