Deep learning presentation
•
12 likes•9,619 views
Short introduction to deep learning and to the DLL Library (C++, https://github.com/wichtounet/dll). Nothing fancy.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Deep learning presentation
Similar to Deep learning presentation (17)
Recently uploaded
Recently uploaded (20)
Deep learning presentation
- 1. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Deep Learning Baptiste Wicht baptiste.wicht@gmail.com September 12, 2014 Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 2. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Table of Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 3. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Contents 1 Deep Learning Definition History Usages Difficulties 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 4. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Definition Deep Learning (Wikipedia) Deep learning is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using model architectures composed of multiple non-linear transformations Deep Learning (deeplearning.net) Deep Learning is a new area of Machine Learning research, which has been introduced with the objective of moving Machine Learning closer to one of its original goals: Artificial Intelligence. Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 5. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Definition (cont.d) Goal: Imitate the nature Set of algorithms Generally structures with multiple layers Often unsupervised feature learning Time-consuming training Sometimes large amount of data Generally complex data New name for an old thing hot topic Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 6. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties History 1960: Neural networks 1985: Multilayer Perceptrons 1986: Restricted Boltzmann Machine 1995: Support Vector Machine 2006: Hinton presents the Deep Belief Network (DBN) New interests in deep learning and RBM State of the art MNIST 2009: Deep Recurrent Neural Network 2010: Convolutional DBN 2011: Max-Pooling CDBN Many competitions won and state of the art results Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 7. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Names Geoffrey Hinton Andrew Y. Ng Yoshua Bengio Honglak Lee Yann LeCun ... Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 8. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Algorithms Deep Neural Networks Deep Belief Networks Convolutional Deep Belief Networks Deep SVM Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 9. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Usages Text recognition Facial Expression Recognition Object Recognition Audio classification Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 10. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Difficulties Large number of free variables Few insights on how to set them Complex to implement Large variations between papers Lot of refinements were proposed Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 11. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contents 1 Deep Learning 2 Restricted Boltzmann Machine Definition Training Units Variants 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 12. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Definition Restricted Boltzmann Machine Function: Learn a probability distribution over the input Generative stochastic neural network Visible and hidden neurons Neurons form a bipartite graph V visible units and visible biases H hidden units and hidden biases VxH weights Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 13. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Definition (Cont.d) Binary units (Bernoulli RBM) p(hj = 1|v) = (cj + mX i viwi,j ) p(vi = 1|h) = (bi + nX j hjwi,j ) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 14. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 15. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 16. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 17. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 18. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 19. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Usages Unsupervised feature learning Classification with other techniques (linear classifier, SVM, ...) Limited to one layer of abstraction Stacking for higher-level models and classification Deep Belief Network Deep Boltzmann Machines Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 20. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Training Objective: Maximizing the log-likelihood Intractable Other methods have been developed: Markov Chain Monte Carlo (MCMC) (Too slow) Contrastive Divergence (CD) (Hinton) Persistent CD Mean-Field CD (mf-CD) Parallel Tempering Annealed Importance Sampling Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 21. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contrastive Divergence For each data point 1 Compute gradients g between t = k and t = k − 1 2 Add g to the weights and the biases Repeat for several epochs Experiments have shown that CD1 (k = 1) works well Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 22. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contrastive Divergence When to stop training ? 1 Proxies to log-likelihood: Reconstruction error Pseudo-likelihood (PCD) 2 Visual inspection of the filters Training is relatively fast Can be trained on GPU Hard to compare two RBMs Hard to test an implementation correctly Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 23. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contrastive Divergence Options Mini-batch training Momentum Weight decay Sparsity Target ... Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 24. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Units RBM Was initially developed with binary units Different types of units can be used: Gaussian visible units for real-value inputs Softmax hidden unit for classification (last layer) Rectified Linear Unit (ReLU) units for hidden/visible Can be capped Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 25. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Variants Convolutional RBM (see later) mean-covariance RBM (mcRBM) Sparse RBM (SRBM) Third-Order RBM Spike And Slab RBM Nonnegative RBM ... Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 26. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network Definition Training 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 27. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Definition Deep Belief Network Generative graphical model Type of Deep Neural Network Multiple layer of hidden units Stack of RBMs Can be implemented with other autoencoders Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 28. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Definition (Cont.d) Each RBM takes input from previous layer output Each layer forms a higher-level representation of the data Number of hidden units in each layer can be tuned Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 29. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Training 1 Train each layer, from bottom to top, with Contrastive Divergence (Unsupervised) 2 Then treat the DBN as a MLP 3 If necessary, fine-tune the last layer for classification (Supervised) Back propagation nonlinear Conjugate Gradient method Limited Memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) Hessian-Free CG (Martens) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 30. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM Definition Training Probabilistic Max Pooling 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 31. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Definition Convolutional RBM Motivation: Translation-invariance Scaling to full-size images Variant of RBM, concepts remain the same NV xNV binary visible units K groups of hidden units NKxNK binary hidden units per group Each group has a NWxNW filter (NW , NV − NH + 1) A bias bk for each hidden group A single bias c for all visible units Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 32. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Definition (Cont.d) Binary units: p(hk j = 1|v) = (bk + (W~ k v v)j ) p(vi = 1|h) = (c + KX k (Wk f hk )i) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 33. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Training Contrastive Divergence Gradients computations are done with convolutions Same refinements can be used (weight decay, momentum, ...) CRBM is highly overcomplete Sparse learning is very important Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 34. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Probabilistic Max Pooling Shrink the representation by a constant factor C Allows higher-level to be invariant to small translations Reduces computational effort Generative version of standard Max Pooling Pooling layer with K groups of pooling units Each group has NPxNP units NP , NH/C Each hidden block (CxC) is connected to exactly one pooling unit Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 35. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Definition (Cont.d) Binary units: p(vi = 1|h) = (c + KX k (Wk f hk )i) I(hk j ) , bk + (W~ k v v)j p(hk j = 1|v) = exp(I(hk i )) 1 + P j02
- 36. exp(I(hk i0)) p(pk = 0|v) = 1 1 + P j02
- 37. exp(I(hk i0)) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 38. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 39. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Stack of Convolutional RBM With or without Probabilistic Max Pooling Each RBM takes input from previous layer output Each layer forms a higher-level representation of the data Number of hidden units in each layer can be tuned Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 40. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Feature Learning Source: Honglak Lee Each layer learns a different abstraction of features 1 Stroke 2 Parts of faces 3 Faces Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 41. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Implementation Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 42. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Implementation Deep Learning Library (DLL) https://github.com/wichtounet/dll RBM Binary, Gaussian, Softmax, ReLU units CD and PCD Momentum, Weight Decay, Sparsity Target Convolutional RBM Standard version Probabilistic Max Pooling Various units CD and PCD Momentum, Weight Decay, Sparsity Target Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 43. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Implementation DBN Pretraining with RBM Fine-tuning with Conjugate Gradient Fine-tuning with Stochastic Gradient Descent Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 44. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Future Work Use CDBN for text detection Convolutional DBN SVM classification layer for DBN Refinements New training methods for RBM/DBN Reduce compute time Maxout, Dropout Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
- 45. Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Conclusion Deep Learning solutions are very powerful State of the art in several problems , Still room for improvement , Still young solutions (hype) , However They are complex to implement / Free variables need to be configured with care / Results from paper are hard to reproduce / Heavy to train / Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning