https://telecombcn-dl.github.io/2017-dlsl/
Winter School on Deep Learning for Speech and Language. UPC BarcelonaTech ETSETB TelecomBCN.
The aim of this course is to train students in methods of deep learning for speech and language. Recurrent Neural Networks (RNN) will be presented and analyzed in detail to understand the potential of these state of the art tools for time series processing. Engineering tips and scalability issues will be addressed to solve tasks such as machine translation, speech recognition, speech synthesis or question answering. Hands-on sessions will provide development skills so that attendees can become competent in contemporary data analytics tools.
4. 4
Previously… A Perceptron
J. Alammar, “A visual and interactive guide to the Basics of Neural Networks” (2016)
F. Van Veen, “The Neural Network Zoo” (2016)
5. 5
Two Perceptrons + Softmax classifier
J. Alammar, “A visual and interactive guide to the Basics of Neural Networks” (2016)
11. 11
Recurrent Neural Network (RNN)
Alex Graves, “Supervised Sequence Labelling with Recurrent Neural Networks”
The hidden layers and the
output depend from previous
states of the hidden layers
12. 12
Alex Graves, “Supervised Sequence Labelling with Recurrent Neural Networks”
The hidden layers and the
output depend from previous
states of the hidden layers
Recurrent Neural Network (RNN)
17. 17
Convolutional Neural Network (CNN)
Slide credit: Junting Pan, “Visual Saliency Prediction using Deep Learning Techniques” (ETSETB-UPC 2015)
Orange
A Krizhevsky, I Sutskever, GE Hinton “Imagenet classification with deep convolutional neural networks” Part of: Advances in
Neural Information Processing Systems 25 (NIPS 2012)
18. 18
Convolutional Neural Network (CNN)
F. Van Veen, “The Neural Network Zoo” (2016)
More details:
D1L3, “Convolutional Neural Networks”
21. 21
Autoencoder (AE)
“Deep Learning Tutorial”, Dept. Computer Science, Stanford
Autoencoders:
● Predict at the output the
same input data.
● Do not need labels:
Unsupervised
learning
22. 22
Autoencoder (AE)
“Deep Learning Tutorial”, Dept. Computer Science, Stanford
Dimensionality reduction:
● Use hidden layer as
a feature extractor of
the desired size.
Unsupervised
learning
25. 25
Variational Autoencoder (VAE)
Kevin Frans, “Variational Autoencoders explained” (2016)
The latent vector learned in the hidden layer of the basic autoencoder (in green)...
30. 30
Restricted Boltzmann Machine (RBM)
Figure: Geoffrey Hinton (2013)
Salakhutdinov, Ruslan, Andriy Mnih, and Geoffrey Hinton. "Restricted Boltzmann machines for
collaborative filtering." Proceedings of the 24th international conference on Machine learning. ACM, 2007.
● Shallow two-layer net.
● Restricted=No two nodes in a layer share a
connection
● Bipartite graph.
● Bidirectional graph
○ Shared weights.
○ Different biases.
31. 31
Restricted Boltzmann Machine (RBM)
Figure: Geoffrey Hinton (2013)
Salakhutdinov, Ruslan, Andriy Mnih, and Geoffrey Hinton. "Restricted Boltzmann machines for
collaborative filtering." Proceedings of the 24th international conference on Machine learning. ACM, 2007.
32. 32
Restricted Boltzmann Machine (RBM)
DeepLearning4j, “A Beginner’s Tutorial for Restricted Boltzmann Machines”.
RBMs are a specific type of
autoencoder.
Unsupervised
learning
33. 33
Deep Belief Networks (DBN)
Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief
nets." Neural computation 18, no. 7 (2006): 1527-1554.
● Architecture like an MLP.
● Training as a stack of
RBMs…
● ...so they do not need
labels:
Unsupervised
learning
34. 34
Deep Belief Networks (DBN)
Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief
nets." Neural computation 18, no. 7 (2006): 1527-1554.
After the DBN is trained, it can
be fine-tuned with a reduced
amount of labels to solve a
supervised task with superior
performance.
Supervised
learning
Softmax
35. 35
Deep Belief Networks (DBN)
F. Van Veen, “The Neural Network Zoo” (2016)
More details:
D2L1,”Deep Belief Networks”
38. 38
Deep Belief Networks (DBN)
Geoffrey Hinton, "Introduction to Deep Learning & Deep Belief Nets” (2012)
Geoorey Hinton, “Tutorial on Deep Belief Networks”. NIPS 2007.