Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Introduction to
Deep Learning
Dmytro Fishman, University of Tartu
What is Machine Learning?
Was born with a goal to create Artificial Intelligence
!
There were two main branches:
• Symbolic...
Machine Learning
founding disciplines
Computer
Science
Statistics
Information
Theory Databases
Probability
Theory
Optimisa...
What is Machine Learning?
Machine learning is a scientific field that develops
computer programs that make data-driven decis...
What is Machine Learning?
Machine learning is
• a scientific field
• develops computer programs
• make data-driven decisions...
Examples of Machine Learning
Spam Filtering
Examples of Machine Learning
Face Recognition
Examples of Machine Learning
Recommendation Systems
Comes from nature…
http://www.borgenmagazine.com/wp-content/uploads/2014/06/natural_disasters.jpg
…and industry
http://forums.vwvortex.com/showthread.php?3208922-2008-Dodge-Challenger-Factory-production-pics./page2
Often is very messy
http://insights.wired.com/profiles/blogs/what-s-the-big-deal-with-unstructured-data
Ideally a long !
list of examples
of the same
nature
Types of data
255,255,255,255
255,255,255,255
255, 56, 65,255
255, 98, 89,110
To be or not to be.
x = (0.5, 0.9, 0.7, -0.3...
Machine Learning tasks
Supervised Learning
• Classification
• Regression!
!
Unsupervised Learning!
• Clustering
• Dimension...
Machine Learning tasks
Supervised Learning
• Classification
• Regression!
!
Unsupervised Learning!
• Clustering
• Dimension...
Supervised Learning
Cat!
Cat
???
Dog
x1 = (87, 205, 255, 155, …); y1 = 1
x2 = (95, 195, 245, 155, …); y2 = 1
x3 = (43, 159...
Evolution of ML methods
Rule-based
Classic Machine
Learning
Representation
Learning
Deep Learning
Adopted from Y.Bengio ht...
Mimicking the brain
Engineering perspective
• Compact!
• Energy efficient (12 watts—a fifth of
the power of a lightbulb)
• 1...
Biological Neuron
Caution!
Here comes the slide with
some math!
Artificial Neurone
x0
w0
x1w1
x2w2
X
i
xiwi + b f
axon from neurone
dendrite
synapse
output axon
activation!
function
f(
X
...
Feed Forward
x0
w0
x1w1
x2w2
X
i
xiwi + b f
axon from neurone
dendrite
synapse
output axon
activation!
function
f(
X
i
xiw...
Training Neural Netowrks
43
255
32
Output
Hidden
…
Input
Image
32
143
255
Output
Hidden
…
Input
Image
Training Neural Netowrks
32
143
255
Output
Hidden
…
Input
Image
Training Neural Netowrks
0.6
32
143
Input
255
0.4
Output
Hidden
Image
…
Cat
Dog
Error
Training Neural Netowrks
Deep Neural Networks
GoogLeNet
Types of Deep Neural
Networks
• Convolutional Neural Networks (CNN)
• Convolutional Deep Belief Networks (CDBN)
!
• Recurr...
Convolutional Neural
Networks
ImageNet Large Scale Visual Recognition
Challenge (ILSVRC)
Breed?
http://cs.stanford.edu/people/karpathy/ilsvrc/
Misclassified examples
Overall:
trained expert got 5.1% of errors
GoogLeNet 6.7% of errors
Before Deep Neural Networks the ...
Galaxy Zoo Challenge
61578 JPG images, made by Hubble
Galaxy Zoo Challenge
http://benanne.github.io/2014/04/05/galaxy-zoo.html
Diabetic Retinopathy Detection
Challenge
0 stage
4 stage
and 80 000 more images
Diabetic Retinopathy Detection Challenge
…
…
Diabetic Retinopathy Detection Challenge
DeepMind
DeepMind
0.6
32
143
Input
255
0.4
Output
Hidden
Image
…
Right
Left
Lost
one
life?
Training DeepMind
Recurrent Neural
Networks
Shakespeare?
Recurrent Neural Network
PANDARUS:
Alas, I think he shall be come approached and the day
When little srain wo...
Recurrent Neural Network
PANDARUS:
Alas, I think he shall be come approached and the day
When little srain would be attain...
Linux
474MB of the
source code
/*!
* Increment the size file of the new incorrect UI_FILTER group information!
* of the si...
Tools
• Python (many implementations)
• Torch (http://torch.ch/)
• Theano (http://deeplearning.net/software/theano/)
• Caf...
Online courses
CS224d: Deep Learning for Natural Language
Processing
by Richard Socher
http://cs224d.stanford.edu/
Machine...
Offline places
Computer Science Bachelor’s program
at CS@UCU
http://cs.ucu.edu.ua/en/
Applied Computer Science Stream
at Su...
Popular Blogs
10 Machine Learning Terms Explained in Simple English
by Aylien
http://blog.aylien.com/post/121281850733/10-...
Kaggle
Platform for data science competitions
Sponsored
Research
Public competitions
Computers are useless.
They can only give
answers.
Pablo Picasso
Nächste SlideShare
Wird geladen in …5
×

Introduction to Deep Learning (Dmytro Fishman Technology Stream)

Lviv IT Arena is a conference specially designed for programmers, designers, developers, top managers, inverstors, entrepreneur and startuppers. Annually it takes place on 2-4 of October in Lviv at the Arena Lviv stadium. In 2015 conference gathered more than 1400 participants and over 100 speakers from companies like Facebook. FitBit, Mail.ru, HP, Epson and IBM. More details about conference at itarene.lviv.ua.

Introduction to Deep Learning (Dmytro Fishman Technology Stream)

  1. 1. Introduction to Deep Learning Dmytro Fishman, University of Tartu
  2. 2. What is Machine Learning? Was born with a goal to create Artificial Intelligence ! There were two main branches: • Symbolic AI, based on if/then like rules (GOFAI) ! • Inspired by Neuroscience principle - learn through examples (later developed into ML)
  3. 3. Machine Learning founding disciplines Computer Science Statistics Information Theory Databases Probability Theory Optimisation Machine Learning
  4. 4. What is Machine Learning? Machine learning is a scientific field that develops computer programs that make data-driven decisions without being explicitly programmed and also automatically improve over time.
  5. 5. What is Machine Learning? Machine learning is • a scientific field • develops computer programs • make data-driven decisions • without being explicitly programmed to • also automatically improve over time.
  6. 6. Examples of Machine Learning Spam Filtering
  7. 7. Examples of Machine Learning Face Recognition
  8. 8. Examples of Machine Learning Recommendation Systems
  9. 9. Comes from nature… http://www.borgenmagazine.com/wp-content/uploads/2014/06/natural_disasters.jpg
  10. 10. …and industry http://forums.vwvortex.com/showthread.php?3208922-2008-Dodge-Challenger-Factory-production-pics./page2
  11. 11. Often is very messy http://insights.wired.com/profiles/blogs/what-s-the-big-deal-with-unstructured-data
  12. 12. Ideally a long ! list of examples of the same nature
  13. 13. Types of data 255,255,255,255 255,255,255,255 255, 56, 65,255 255, 98, 89,110 To be or not to be. x = (0.5, 0.9, 0.7, -0.3, …) x = (255, 255, 255, 255, …) x = (0, …, 0, 2, 1, 0, 0, 1, 2, …) to or not be the cat low John Adopted from P.Vincent http://videolectures.net/deeplearning2015_vincent_machine_learning/
  14. 14. Machine Learning tasks Supervised Learning • Classification • Regression! ! Unsupervised Learning! • Clustering • Dimensionality reduction • Density Estimation ! Reinforcement Learning - taking actions in an environment to maximise reward
  15. 15. Machine Learning tasks Supervised Learning • Classification • Regression! ! Unsupervised Learning! • Clustering • Dimensionality reduction • Density Estimation ! Reinforcement Learning - taking actions in an environment to maximise reward
  16. 16. Supervised Learning Cat! Cat ??? Dog x1 = (87, 205, 255, 155, …); y1 = 1 x2 = (95, 195, 245, 155, …); y2 = 1 x3 = (43, 159, 255, 5, …); y3 = -1 x4 = (180, 66, 245, …); ! y4 = f(x4) = 1 Features Labels Adopted from P.Vincent http://videolectures.net/deeplearning2015_vincent_machine_learning/
  17. 17. Evolution of ML methods Rule-based Classic Machine Learning Representation Learning Deep Learning Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/ Input Input Input Input Fixed set of rules Output Hand designed features Learning Automated feature extraction Output Learning Output Low level features High level features Learning Output
  18. 18. Mimicking the brain Engineering perspective • Compact! • Energy efficient (12 watts—a fifth of the power of a lightbulb) • 10^11 neurons • for each neurone there are 10-50 supporting cells (glial cells) • 10^14 synapses (connections) Computing power • Fairly slow in arithmetic calculations • Very efficient in computer vision, speech, language etc. Adopted from L.Bottou http://videolectures.net/deeplearning2015_bottou_neural_networks/
  19. 19. Biological Neuron
  20. 20. Caution! Here comes the slide with some math!
  21. 21. Artificial Neurone x0 w0 x1w1 x2w2 X i xiwi + b f axon from neurone dendrite synapse output axon activation! function f( X i xiwi + b) Slide from CS231n: Convolutional Neural Networks for Visual Recognition http://cs231n.github.io/ by Andrej Karpathy
  22. 22. Feed Forward x0 w0 x1w1 x2w2 X i xiwi + b f axon from neurone dendrite synapse output axon activation! function f( X i xiwi + b) Slide from CS231n: Convolutional Neural Networks for Visual Recognition http://cs231n.github.io/ by Andrej Karpathy class Neuron: # ... def forward(inputs): """ assume inputs and weights are 1-D numpy arrays and bias is a number """ cell_body_sum = np.sum(inputs * self.weights) + self.bias firing_rate = 1.0 / (1.0 + math.exp(-cell_body_sum)) # sigmoid activation function return firing_rate
  23. 23. Training Neural Netowrks 43 255 32 Output Hidden … Input Image
  24. 24. 32 143 255 Output Hidden … Input Image Training Neural Netowrks
  25. 25. 32 143 255 Output Hidden … Input Image Training Neural Netowrks
  26. 26. 0.6 32 143 Input 255 0.4 Output Hidden Image … Cat Dog Error Training Neural Netowrks
  27. 27. Deep Neural Networks
  28. 28. GoogLeNet
  29. 29. Types of Deep Neural Networks • Convolutional Neural Networks (CNN) • Convolutional Deep Belief Networks (CDBN) ! • Recurrent Neural Networks (RNN) • Long Short-Term Memory Networks (LSTM) ! https://en.wikipedia.org/wiki/Deep_learning
  30. 30. Convolutional Neural Networks
  31. 31. ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
  32. 32. Breed? http://cs.stanford.edu/people/karpathy/ilsvrc/
  33. 33. Misclassified examples Overall: trained expert got 5.1% of errors GoogLeNet 6.7% of errors Before Deep Neural Networks the best result was 20.7%
  34. 34. Galaxy Zoo Challenge 61578 JPG images, made by Hubble
  35. 35. Galaxy Zoo Challenge http://benanne.github.io/2014/04/05/galaxy-zoo.html
  36. 36. Diabetic Retinopathy Detection Challenge 0 stage 4 stage and 80 000 more images
  37. 37. Diabetic Retinopathy Detection Challenge
  38. 38. … … Diabetic Retinopathy Detection Challenge
  39. 39. DeepMind
  40. 40. DeepMind
  41. 41. 0.6 32 143 Input 255 0.4 Output Hidden Image … Right Left Lost one life? Training DeepMind
  42. 42. Recurrent Neural Networks
  43. 43. Shakespeare? Recurrent Neural Network PANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, I should not sleep. ! Second Senator: They are away this miseries, produced upon my soul, Breaking and strongly should be buried, when I perish The earth and thoughts of many states. ! DUKE VINCENTIO: Well, your wit is in the care of side and that. ! Second Lord: They would be ruled after this chamber, and my fair nues begun out of the fact, to be conveyed, Whose noble souls I'll have the heart of the wars. ! Clown: Come, sir, I will make did behold your worship. ! VIOLA: I'll drink it. VIOLA. On your attendance, my lord; here. ! DUKE. Stand you awhile aloof.—Cesario, Thou know'st no less but all; I have unclasp'd To thee the book even of my secret soul: Therefore, good youth, address thy gait unto her; Be not denied access, stand at her doors, And tell them there thy fixed foot shall grow Till thou have audience. ! VIOLA. Sure, my noble lord, If she be so abandon'd to her sorrow As it is spoke, she never will admit me. ! DUKE. Be clamorous and leap all civil bounds, Rather than make unprofited return. ! VIOLA. Say I do speak with her, my lord. What then? 4.4МB Shakespeare texts Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy
  44. 44. Recurrent Neural Network PANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, I should not sleep. ! Second Senator: They are away this miseries, produced upon my soul, Breaking and strongly should be buried, when I perish The earth and thoughts of many states. ! DUKE VINCENTIO: Well, your wit is in the care of side and that. ! Second Lord: They would be ruled after this chamber, and my fair nues begun out of the fact, to be conveyed, Whose noble souls I'll have the heart of the wars. ! Clown: Come, sir, I will make did behold your worship. ! VIOLA: I'll drink it. VIOLA. On your attendance, my lord; here. ! DUKE. Stand you awhile aloof.—Cesario, Thou know'st no less but all; I have unclasp'd To thee the book even of my secret soul: Therefore, good youth, address thy gait unto her; Be not denied access, stand at her doors, And tell them there thy fixed foot shall grow Till thou have audience. ! VIOLA. Sure, my noble lord, If she be so abandon'd to her sorrow As it is spoke, she never will admit me. ! DUKE. Be clamorous and leap all civil bounds, Rather than make unprofited return. ! VIOLA. Say I do speak with her, my lord. What then? Shakespeare? Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy
  45. 45. Linux 474MB of the source code /*! * Increment the size file of the new incorrect UI_FILTER group information! * of the size generatively.! */! static int indicate_policy(void)! {! int error;! if (fd == MARN_EPT) {! /*! * The kernel blank will coeld it to userspace.! */! if (ss->segment < mem_total)! unblock_graph_and_set_blocked();! else! ret = 1;! goto bail;! }! segaddr = in_SB(in.addr);! selector = seg / 16;! setup_works = true;! for (i = 0; i < blocks; i++) {! seq = buf[i++];! bpf = bd->bd.next + i * search;! if (fd) {! current = blocked;! }! }! rw->name = "Getjbbregs";! bprm_self_clearl(&iv->version);! regs->new = blocks[(BPF_STATS << info->historidac)] | PFMR_CLOBATHINC_SECONDS << 12;! return segtable;! } Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy
  46. 46. Tools • Python (many implementations) • Torch (http://torch.ch/) • Theano (http://deeplearning.net/software/theano/) • Caffe (http://caffe.berkeleyvision.org/) ! ! Deep Learning Libraries by programming language: http://www.teglor.com/b/deep-learning-libraries- language-cm569
  47. 47. Online courses CS224d: Deep Learning for Natural Language Processing by Richard Socher http://cs224d.stanford.edu/ Machine Learning by Andrew Ng https://www.coursera.org/learn/machine-learning Neural Networks for Machine Learning 
 by Geoffrey Hinton https://www.coursera.org/course/neuralnets CS231n: Convolutional Neural Networks for Visual Recognition by Andrej Karpathy http://cs231n.github.io/
  48. 48. Offline places Computer Science Bachelor’s program at CS@UCU http://cs.ucu.edu.ua/en/ Applied Computer Science Stream at Summer School AACIMP (Kyiv) http://summerschool.ssa.org.ua/ The Deep Learning Summer School 
 at Montreal, Canada https://sites.google.com/site/deeplearningsummerschool/ Computer Science Master’s program 
 at Tartu, Estonia https://sites.google.com/site/deeplearningsummerschool/ International Conference on Machine Learning 
 at New York, USA http://icml.cc/2016/
  49. 49. Popular Blogs 10 Machine Learning Terms Explained in Simple English by Aylien http://blog.aylien.com/post/121281850733/10-machine-learning-terms-explained-in-simple The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy http://karpathy.github.io/2015/05/21/rnn-effectiveness/ A Neural Network in 11 lines of Python
 by iamtrask http://iamtrask.github.io/2015/07/12/basic-python-network/ Understand LSTM Networks 
 by Christopher Olah https://colah.github.io/posts/2015-08-Understanding-LSTMs/ Sign up for Data Elixir mailing list: http://dataelixir.com/
  50. 50. Kaggle Platform for data science competitions Sponsored Research Public competitions
  51. 51. Computers are useless. They can only give answers. Pablo Picasso

×