The document discusses various text analytics elements used for vector space modeling of text, including word2vec and other models. It provides examples of vector representations of words in a corpus and co-occurrence matrices. It also covers problems with high-dimensional models and potential solutions like single value decomposition. Skip-gram and n-gram models for capturing word relationships are described. The document concludes with noting it will provide Python code demonstrations of these techniques.

1. Text Analytics elements.
Word2Vec and others
Ilya Gerasimov
Software Engineering Team Leader, Saint-Petersburg
April 4, 2015

2. 2CONFIDENTIAL
Vector Space Model

3. 3CONFIDENTIAL
Discrete representation
In vector space terms, this is a vector with one 1 and a lot of
zeroes
[0 0 0 0 0 0 0 0 0 0 1 0 0 0 0]
Dimensionality: 20K (speech) – 50K (vocab) – 500K (big vocab)
– 13M (Google 1T)
motel [0 0 0 0 0 0 0 0 0 0 1 0 0 0 0] AND
hotel [0 0 0 0 0 0 0 1 0 0 0 0 0 0 0] = 0

4. 4CONFIDENTIAL
TF-IDF metric

5. 5CONFIDENTIAL
Window based cooccurence matrix
Example corpus:
• I like deep learning.
• I like NLP.
• I enjoy flying.
I like enjoy deep learnin
g
NLP flying
I x 2 1 0 0 0 0
like 2 x 0 1 0 1 0
enjoy 1 0 x 0 0 0 1
deep 0 1 0 x 1 0 0
learning 0 0 0 1 x 0 0
NLP 0 1 0 0 0 x 0
flying 0 0 1 0 0 0 x

6. 6CONFIDENTIAL
Problems
• Increase in size with vocabulary
• Very high dimensional: require a lot of storage
• Models are less robust

7. 7CONFIDENTIAL
Single Value Decomposition

8. 8CONFIDENTIAL
Cosine similarity

9. 9CONFIDENTIAL
Deep learning

10. 10CONFIDENTIAL
N-grams & Skip grams
London is the capital of Great Britain
N-grams: [London, is] [is the capital]
[capital of Great Britain]
Skip grams: [London the capital]
[capital Britain] [London Britain]

11. 11CONFIDENTIAL
Examples. I

12. 12CONFIDENTIAL
Examples. II

13. 13CONFIDENTIAL
Demo time.
Tools:
Python 2.7
Gensim https://radimrehurek.com/gensim/
Wikipedia corpus