The fourth lecture from the Machine Learning course series of lectures. This lecture first introduces a problem of visualising multi-dimensional data on fewer dimensions and later discusses one of the most popular methods for reducing dimensionality - principal component analysis (PCA). Later, also t-SNE is mentioned briefly as a non-linear alternative to PCA. A link to my github (https://github.com/skyfallen/MachineLearningPracticals) with practicals that I have designed for this course in both R and Python. I can share keynote files, contact me via e-mail: dmytro.fishman@ut.ee.

1. Introduction to Machine
Learning
(Dimensionality reduction)
Dmytro Fishman (dmytro@ut.ee)

2. So far we have been fearless drawing multi-dimensional
instances on two-dimensional planes

3. 28px
28px

4. 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 126,
213, 34, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 227, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 114, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 178, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 236, 254, 254,
84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 231, 56, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 191, 254, 254, 185, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 105,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 93, 248, 254, 243, 27, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
16, 192, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 32, 254, 254, 248, 95, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 254, 254, 231, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 186, 254, 254, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
21, 203, 254, 188, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 255, 179, 8, 4, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 254, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 68, 240, 254, 254, 250, 103, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 92, 255, 207, 91, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0
28px
28px

5. 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 126,
213, 34, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 227, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 114, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 178, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 236, 254, 254,
84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 231, 56, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 191, 254, 254, 185, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 105,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 93, 248, 254, 243, 27, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
16, 192, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 32, 254, 254, 248, 95, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 254, 254, 231, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 186, 254, 254, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
21, 203, 254, 188, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 255, 179, 8, 4, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 254, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 68, 240, 254, 254, 250, 103, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 92, 255, 207, 91, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0
28px
28px
102
41

6. 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 126,
213, 34, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 227, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 114, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 178, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 236, 254, 254,
84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 231, 56, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 191, 254, 254, 185, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 105,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 93, 248, 254, 243, 27, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
16, 192, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 32, 254, 254, 248, 95, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 254, 254, 231, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 186, 254, 254, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
21, 203, 254, 188, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 255, 179, 8, 4, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 254, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 68, 240, 254, 254, 250, 103, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 92, 255, 207, 91, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0
28px
28px
41
Pixel #X
Pixel#Y
102

7. 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 126,
213, 34, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 227, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 114, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 178, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 236, 254, 254,
84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 231, 56, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 191, 254, 254, 185, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 105,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 93, 248, 254, 243, 27, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
16, 192, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 32, 254, 254, 248, 95, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 254, 254, 231, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 186, 254, 254, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
21, 203, 254, 188, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 255, 179, 8, 4, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 254, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 68, 240, 254, 254, 250, 103, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 92, 255, 207, 91, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0
28px
28px Pixel #X
Pixel#Y
41
How about the rest of 782 pixels?
102

8. 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 126,
213, 34, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 227, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 114, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 178, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 236, 254, 254,
84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 231, 56, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 191, 254, 254, 185, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 105,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 93, 248, 254, 243, 27, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
16, 192, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 32, 254, 254, 248, 95, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 254, 254, 231, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 186, 254, 254, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
21, 203, 254, 188, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 255, 179, 8, 4, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 254, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 68, 240, 254, 254, 250, 103, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 92, 255, 207, 91, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0
28px
28px Pixel #X
Pixel#Y
41
How about the rest of 782 pixels?
But we cannot really visualise all 784
dimensions...
102

9. 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 126,
213, 34, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 227, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 114, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 178, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 236, 254, 254,
84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 254, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 254, 254, 231, 56, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 191, 254, 254, 185, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 105,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 233, 254, 254, 30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 93, 248, 254, 243, 27, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
16, 192, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 32, 254, 254, 248, 95, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 254, 254, 231, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 186, 254, 254, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
21, 203, 254, 188, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 255, 179, 8, 4, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 254, 254, 254, 254, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 68, 240, 254, 254, 250, 103, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 92, 255, 207, 91, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0
28px
28px Pixel #X
Pixel#Y
41
How about the rest of 782 pixels?
102
But we cannot really visualise all 784
dimensions... or can we?

10. Dimensionality reduction
Patients
Healthy

11. The following material was inspired by:
https://www.youtube.com/watch?v=_UVHneBUBW0

12. The following material was inspired by:
https://www.youtube.com/watch?v=_UVHneBUBW0
With a bit of fixes,
though

13. Patients
Healthy

14. Patients
Healthy
Each dot in this plot
represents a person

15. Patients
Healthy
Every person in its turn
could be described by
hundreds of protein
reactivities
Each dot in this plot
represents a person

16. Patients
Healthy
Every person in its turn
could be described by
hundreds of protein
reactivities
Each dot in this plot
represents a person
The idea is that people with
similar protein reactivity
profiles cluster together

17. Patients
Healthy
5, 98, 43, …, 9, 66, Patient

18. Patients
Healthy
5, 98, 43, …, 9, 66, Patient
62, 8, 14, …, 73, 2, Healthy

19. Patients
Healthy
5, 98, 43, …, 9, 66, Patient
62, 8, 14, …, 73, 2, Healthy
3, 45, 13, …, 7, 54, Patient

20. Patients
Healthy
5, 98, 43, …, 9, 66, Patient
62, 8, 14, …, 73, 2, Healthy
3, 45, 13, …, 7, 54, Patient
How is that even possible?

21. Patients
Healthy
PCA
5, 98, 43, …, 9, 66, Patient
62, 8, 14, …, 73, 2, Healthy
3, 45, 13, …, 7, 54, Patient
How is that even possible?

22. Introduction to dimensions

23. 1-Dimensional data
Person Protein #1
A 24
B 63
C 51
D 34
E 15

24. 1-Dimensional data
300 60
Person Protein #1
A 24
B 63
C 51
D 34
E 15

25. 1-Dimensional data
300 60
300 60
Uniformly
distributed data
Person Protein #1
A 24
B 63
C 51
D 34
E 15

26. 1-Dimensional data
300 60
300 60
Uniformly
distributed data
30 60
Non-uniform
distribution
0
Person Protein #1
A 24
B 63
C 51
D 34
E 15

27. 2-Dimensional data
300 60
Protein #1
Person Protein #1
A 24
B 63
C 51
D 34
E 15

28. 2-Dimensional data
30
60
Protein#2
Person Protein #1 Protein #2
A 24
B 63
C 51
D 34
E 15
300 60
Protein #1
0

29. 2-Dimensional data
Person Protein #1 Protein #2
A 24 29
B 63
C 51
D 34
E 15
A30
60
Protein#2 300 60
Protein #1
0

30. 2-Dimensional data
A
Person Protein #1 Protein #2
A 24 29
B 63
C 51
D 34
E 15
30
60
Protein#2 300 60
Protein #1
0

31. 2-Dimensional data
A
Person Protein #1 Protein #2
A 24 29
B 63 59
C 51
D 34
E 15
B
30
60
Protein#2 300 60
Protein #1
0

32. 2-Dimensional data
A
B
And so on…
Person Protein #1 Protein #2
A 24 29
B 63 59
C 51
D 34
E 15
30
60
Protein#2 300 60
Protein #1
0

33. 2-Dimensional data
Person Protein #1 Protein #2
A 24 29
B 63 59
C 51 32
D 34 56
E 15 4
… … …
30
60
Protein#2 300 60
Protein #1
0

34. 2-Dimensional data
Protein profiles are correlated
Person Protein #1 Protein #2
A 24 29
B 63 59
C 51 32
D 34 56
E 15 4
… … …
30
60
Protein#2 300 60
Protein #1
0

35. 2-Dimensional data
Here is no correlation
Person Protein #1 Protein #2
A 24 29
B 63 59
C 51 32
D 34 56
E 15 4
… … …
30
60
Protein#2 300 60
Protein #1
0

36. 3-Dimensional data
Person Protein #1 Protein #2
A 24 29
B 63 59
C 51 32
D 34 56
E 15 4
… … …
30
60
Protein#2 300 60
Protein #1
0

37. 3-Dimensional data
Person P1 P2 P3
A 24 29
B 63 59
C 51 32
D 34 56
E 15 4
… … …
Protein #330
60
Protein#2 300 60
Protein #1
0
30
60

38. 3-Dimensional data
Person P1 P2 P3
A 24 29 33
B 63 59
C 51 32
D 34 56
E 15 4
… … …
A
30
60
Protein #330
60
Protein#2 300 60
Protein #1
0

39. Dimensions
300 60
1-Dimensional

40. Dimensions
300 60
30 6
3
6
2-Dimensional
1-Dimensional

41. Dimensions
300 60
30 6
3
6
3-Dimensional
30 6
3
6
3
6
2-Dimensional
1-Dimensional

42. How about 200-D data?

43. Are all D are equally important?
How about 200-D data?

44. Are all D are equally important?
Let’s go back to 2-D example
How about 200-D data?

45. 30
60
Protein#2
300 60
Protein #1
0
Importance of axes

46. Variation is
from left to right
30
60
Protein#2
300 60
Protein #1
0
Importance of axes

47. Projected data -
does not look
too different
30
60
Protein#2
300 60
Protein #1
0
Importance of axes

48. We can safely
remove 2nd dimension
30
60
Protein#2
300 60
Protein #1
0
300 60
Protein #1
Importance of axes
Projected data -
does not look
too different

49. 30
60
Protein#2
300 60
Protein #1
0
300 60
Protein #1
The important variation is from left to right
We can safely
remove 2nd dimension
Importance of axes
Projected data -
does not look
too different

50. 30
60
Protein#2
300 60
Protein #1
0
Principle components

51. 30
60
Protein#2
300 60
Protein #1
0
Data is mostly spread
along this line
Principle components

52. 30
60
Protein#2
300 60
Protein #1
0
Data is mostly spread
along this line
and a little bit along
this line
Principle components

53. 30
60
Protein#2
300 60
Protein #1
0
Data is mostly spread
along this line
and a little bit along
this line
What if we make new axes from these lines?
Principle components

54. 30
60
Protein
#2
30
0
60
Protein
#1
0
New X and Y axes
Principle components

55. Principle components
Easier to see right/left and
above/below variation
New X and Y axes

56. Principle components
PC1
Easier to see right/left and
above/below variation
PC2
These new axes are called
Principle Components or
PCs
New X and Y axes

57. Principle components
PC1
Easier to see right/left and
above/below variation
PC1 spans along the
direction of the most
PC2
These new axes are called
Principle Components or
PCs
New X and Y axes

58. Principle components
PC1
Easier to see right/left and
above/below variation
PC1 spans along the
direction of the most
PC2
PC2 spans along the
direction of the most
These new axes are called
Principle Components or
PCs
New X and Y axes

59. Principle components
PC1
Easier to see right/left and
above/below variation
PC1 spans along the
direction of the most
PC2
PC2 spans along the
direction of the most
These new axes are called
Principle Components or
PCs
Principle Components are
always orthogonal one to
another
New X and Y axes

60. Principle components
PC1
PC2
If original data would have 3 dimensions, we would
have 3rd PC
PC3

61. Principle components
PC1
PC2
If original data would have 3 dimensions, we would
have 3rd PC
PC3
There is always as
many PCs as there are
dimensions

62. Principle components
If original data would have 3 dimensions, we would
have 3rd PC
There is always as
many PCs as there are
dimensions
Each new PCs is
guaranteed to explain
less variance
PC1
PC2
PC3

63. Principle components
If original data would have 3 dimensions, we would
have 3rd PC
There is always as
many PCs as there are
dimensions
Usually it is enough to project data points onto first
two PCs to see important patterns
PC1
PC2
PC3
Each new PCs is
guaranteed to explain
less variance

64. PCA
Now we know what those
PC1 and PC2 stand for!

65. PCA
Now we know what those
PC1 and PC2 stand for!
Wait! But so far we have
been talking only about
2 proteins!

66. PCA
Now we know what those
PC1 and PC2 stand for!
We are coming to this part
Wait! But so far we have
been talking only about
2 proteins!

67. Where PCs come from
PC1
PC2

68. PC1
PC2
These are vectors
Where PCs come from

69. PC1
PC2
30
3
Where PCs come from
These are vectors
All vectors have
coordinates

70. PC1
PC2
30
3
For example, this one
has coordinates (3,3)
Where PCs come from
These are vectors
All vectors have
coordinates

71. PC1
PC2
30
3
For example, this one
has coordinates (3,3)
In 200-D space vectors have 200 coordinates
Where PCs come from
These are vectors
All vectors have
coordinates

72. 30
60
Protein#2
300 60
Protein #1
0
How did we choose PC1?
Where PCs come from

73. 30
60
Protein#2
300 60
Protein #1
0
How did we choose PC1?
By minimising the distance
from points to the vector
Where PCs come from

74. 30
60
Protein#2
300 60
Protein #1
0
How did we choose PC1?
By minimising the distance
from points to the vector
Where PCs come from

75. 30
60
Protein#2
300 60
Protein #1
0
How did we choose PC1?
Coordinates of this vector
are coordinates of PC1
By minimising the distance
from points to the vector
Where PCs come from
PC1

76. 30
60
Protein#2
300 60
Protein #1
0
How did we choose PC1?
Coordinates of this vector
are coordinates of PC1
By minimising the distance
from points to the vector
Where PCs come from
Coordinates of PC2 are
chosen in similar fashion but
PC2 should be orthogonal
to PC1
PC1
PC2

77. PCA Example
Reactivities PCs coordinates
PC1 PC2
2 3
4 12
… …
-5 0
Let’s find coordinates of Person A in terms of PCs
Person Pr1 Pr2 … PrN
A 24 29 … 11
B 63 59 … 1
C 51 32 … 23
D 34 56 … 2
E 15 4 … 8
… … … … …

78. PCA Example
Reactivities PCs coordinates
PC1 PC2
2 3
4 12
… …
-5 0
Let’s find coordinates of Person A in terms of PCs
Person Pr1 Pr2 … PrN
A 24 29 … 11
B 63 59 … 1
C 51 32 … 23
D 34 56 … 2
E 15 4 … 8
… … … … …
asmanyasthere
areproteins(N)

79. PCA Example
Reactivities PCs coordinates
Let’s find coordinates of Person A in terms of PCs
Person A (PC1 score) =
PC1 PC2
2 3
4 12
… …
-5 0
Person Pr1 Pr2 … PrN
A 24 29 … 11
B 63 59 … 1
C 51 32 … 23
D 34 56 … 2
E 15 4 … 8
… … … … …

80. PCA Example
Reactivities PCs coordinates
Let’s find coordinates of Person A in terms of PCs
Person A (PC1 score) = 24*2 + 29*4 +… + 11*(-5) = 11
PC1 PC2
2 3
4 12
… …
-5 0
Person Pr1 Pr2 … PrN
A 24 29 … 11
B 63 59 … 1
C 51 32 … 23
D 34 56 … 2
E 15 4 … 8
… … … … …

81. PCA Example
Reactivities PCs coordinates
Person A (PC2 score) = 24*3+ 29*12 + … +11*0 = 21
Reactivities PCs coordinates
Let’s find coordinates of Person A in terms of PCs
Person A (PC1 score) = 24*2 + 29*4 +… + 11*(-5) = 11
PC1 PC2
2 3
4 12
… …
-5 0
Person Pr1 Pr2 … PrN
A 24 29 … 11
B 63 59 … 1
C 51 32 … 23
D 34 56 … 2
E 15 4 … 8
… … … … …

82. Principle components
Principle component 1
Principle
component 2
21
110
0
A
Person A (PC2 score) = 24*3+ 29*12 + … +11*0 = 21
Person A (PC1 score) = 24*2 + 29*4 +… + 11*(-5) = 11

83. Principle components
Principle component 1
Principle
component 2
Person B (PC2 score) = 65
Person B (PC1 score) = 60
65
600
0
A
B

84. Principle components
Principle component 1
Principle
component 2
0
0
A
B
C
D E
G

85. Principle components
Principle component 1
Principle
component 2
0
0
A
B
C
D E
G
The idea is that similar points in
multi-dimensional space get
located close in fewer dimensions
Healthy
Patients

86. PCA is good, but it is a linear algorithm, meaning
that it cannot capture complex relationship between
features

87. PCA is good, but it is a linear algorithm, meaning
that it cannot capture complex relationship between
features
There are always alternative options to consider…

88. https://www.analyticsvidhya.com/blog/2017/01/t-sne-implementation-r-python/
t-Distributed Stochastic Neighbour
Embedding (t-SNE)
visualisation is form http://distill.pub/2016/misread-tsne/
t-SNE is non-linear
dimensionality reduction
algorithm
t-SNE uses conditional
probabilities to represent
similarity between points
Has been shown to better
represent underlying
structure of
multidimensional data

89. Practice

90. https://www.analyticsvidhya.com/blog/2017/01/t-sne-implementation-r-python/
t-SNE
visualisation is form http://distill.pub/2016/misread-tsne/
PCA
O(N2) makes it very slow
Meaning of features is lost
Has been shown to capture the
structure of multi-dimensional
data better (than PCA)
Works relatively fast even on big
datasets
Transformed features could be
traced back
Usually is not so good in figuring
out hidden structure

91. Demo

92. References
• Machine Learning by Andrew Ng (https://www.coursera.org/learn/machine-
learning)
• Introduction to Machine Learning by Pascal Vincent given at Deep Learning
Summer School, Montreal 2015 (http://videolectures.net/
deeplearning2015_vincent_machine_learning/)
• Welcome to Machine Learning by Konstantin Tretyakov delivered at AACIMP
Summer School 2015 (http://kt.era.ee/lectures/aacimp2015/1-intro.pdf)
• Stanford CS class: Convolutional Neural Networks for Visual Recognition by
Andrej Karpathy (http://cs231n.github.io/)
• Data Mining Course by Jaak Vilo at University of Tartu (https://courses.cs.ut.ee/
MTAT.03.183/2017_spring/uploads/Main/DM_05_Clustering.pdf)
• Machine Learning Essential Conepts by Ilya Kuzovkin (https://
www.slideshare.net/iljakuzovkin)
• From the brain to deep learning and back by Raul Vicente Zafra and Ilya
Kuzovkin (http://www.uttv.ee/naita?id=23585&keel=eng)

93. www.biit.cs.ut.ee www.ut.ee www.quretec.ee