1. Deep learning
A primer for the curious developer
Uwe Friedrichsen & Dr. Shirin Glander –codecentric AG – 2018

2. @ufried
Uwe Friedrichsen
uwe.friedrichsen@codecentric.de
@ShirinGlander
Dr. Shirin Glander
shirin.glander@codecentric.de

3. Why should I care about Deep Learning?

4. Deep learning has the potential to affect white collar workers (including IT)
in a similar way as robots affected blue collar workers.

5. What is Deep Learning?
Some success stories

16. What is Deep Learning?
A rough classification

17. AI
Artificial
Intelligence
ML
Machine
Learning
RL
Representational
Learning
DL
Deep
Learning

18. Traditional AI
Focus on problems that are ...
• ... hard for humans
• ... straightforward for computers
• ... can be formally described
Deep Learning
Focus on problems that are ...
• ... intuitive for humans
• ... difficult for computers
(hard to be described formally)
• ... best learnt from experience

19. Where does Deep Learning come from?

20. General evolution
• Two opposed forces
• Recreation of biological neural processing
• Abstract mathematical models (mostly linear algebra)
• Results in different models and algorithms
• No clear winner yet

21. Cybernetics (ca. 1940 - 1960)
• ADALINE, Perceptron
• Linear models, typically no hidden layers
• Stochastic Gradient Descent (SGD)
• Limited applicability
• E.g., ADALINE could not learn XOR
• Resulted in “First winter of ANN” (Artificial Neural Networks)

22. Connectionism (ca. 1980 - 1990)
• Neocognitron
• Non-linear models, distributed feature representation
• Backpropagation
• Typically 1, rarely more hidden layers
• First approaches of sequence modeling
• LSTM (Long short-term memory) in 1997
• Unrealistic expectations nurtured by ventures
• Resulted in “Second winter of ANN”

23. Deep Learning (ca. 2006 -)
• Improved algorithms, advanced computing power
• Enabled training much larger and deeper networks
• Enabled training much larger data sets
• Typically several to many hidden layers
• Overcame the “feature extraction dilemma”

24. What is Deep Learning used for?

25. Deep Learning application areas
• Classification (incl. missing inputs)
• Regression (value prediction)
• Function prediction
• Density estimation
• Structured output (e.g., translation)
• Anomaly detection
• Synthesis and sampling
• Denoising
• Compression (dimension reduction)
• ...

26. How does Deep Learning work?
A first (scientifically inspired) approach

27. „A computer program is said to learn
• from experience E
• with respect to some class of tasks T
• and performance measure P
if its performance at tasks in T,
as measured by P,
improves with experience E.”
-- T. Mitchell, Machine Learning, p. 2, McGraw Hill (1997)
Supervised learning,
unsupervised learning,
reinforcement learning, ...
Too difficult to solve
with fixed programs
designed by humans
Accuracy vs. error rate,
training vs. test set, ...

28. Err ...
Hmm ...
Well ...
I don’t get it!

29. How does Deep Learning work?
A second (more down-to-earth) approach

30. Operating
principle
Training
Network
types
Deep
Learning

31. Deep
Learning
Operating
principle
Training
Network
types
Structure
Behavior
Weight
Operation
Neuron
Data
CNN
Types
Challenges
Quality
measure
RNN
LSTM
Auto-
encoder
GAN
MLP
Training
set
Cost
function
Transfer
learning
Regulari-
zation
Layer
Connection
Hyper-
parameter
Activation
function
Reinforce-
ment
Unsuper-
vised
Supervised
Stochastic
gradient
descent
Back-
propagation
Under-/
Overfitting
Validation/
Test set
Optimization
procedure

32. Deep
Learning
Operating
principle
Training
Network
types

33. Structure
Behavior
Operating
principle

34. Structure
Behavior
Operating
principle

35. Operating
principle
Structure
Behavior
Neuron

36. Neuron
• Design inspired by biological neurons
• One or more inputs
• Processing (and state storage) unit
• One or more outputs
• In practice often implemented as tensor transformations
• Relevance of internal state depends on network type
• Usually negligible for feed-forward networks
• Usually relevant for recurrent networks
Neuron
Processing
(+ State)
Output(s)
Input(s)
...
...

37. Layer
Operating
principle
Structure
Behavior
Neuron

38. Layer
• Neurons typically organized in layers
• Input and output layer as default
• Optionally one or more hidden layer
• Layer layout can have 1-n dimensions
• Neurons in different layers can have different properties
• Different layers responsible for different (sub-)tasks
Output layer
Input layer
...
N
1
2
Hidden layer(s)
...

39. Connection
Operating
principle
Structure
Behavior
Neuron
Layer

40. Connection
• Usually connect input and output tensor in a 1:1 manner
• Connect between layers (output layer N-1 à input layer N)
• Layers can be fully or partially (sparsely) connected
• RNNs also have backward and/or self connections
• Some networks have connections between neurons
of the same layer (e.g., Hopfield nets, Boltzmann machines)
Input tensor(s)
Output tensor(s)

41. Weight
Operating
principle
Structure
Behavior
Neuron
Layer
Connection

42. Weight
• (Logically) augments a connection
• Used to amplify or dampen a signal sent over a connection
• The actual “memory” of the network
• The “right” values of the weights are learned during training
• Can also be used to introduce a bias for a neuron
• By connecting it to an extra neuron that constantly emits 1
W
Weight

43. Operation
Operating
principle
Structure
Behavior
Neuron
Layer
Weight
Connection

44. Input tensor(s)
Output tensor(s)
Step 1
• For each neuron of input layer
• Copy resp. input tensor’s value to neuron’s input
• Calculate state/output using activation function
(typically linear function, passing value through)
Step 2-N
• For each hidden layer and output layer in their order
• For each neuron of the layer
• Calculate weighted sum on inputs
• Calculate state/output using activation function
(see examples later)
Final step
• For each neuron of output layer
• Copy neuron’s output to resp. output tensor’s value

45. Input tensor(s)
Output tensor(s)
Step 1
Final step
Step 2-N
• Default update procedure (most widespread)
• All neuron per layer in parallel
• Different update procedures exist
• E.g., some Hopfield net implementations
randomly pick neurons for update

46. Activation
function
Operating
principle
Structure
Behavior
Neuron
Layer
Weight
Connection
Operation

47. Linear function
• Easy to handle
• Cannot handle
non-linear problems

48. Logistic sigmoid function
• Very widespread
• Delimits output to [0, 1]
• Vanishing gradient
problem

49. Hyperbolic tangent
• Very widespread
• Delimits output to [-1, 1]
• Vanishing gradient
problem

50. Rectified linear unit (ReLU)
• Easy to handle
• No derivative in 0
• Dying ReLU problem
• Can be mitigated, e.g.,
by using leaky ReLU

51. Softplus
• Smooth approximation
of ReLU
• ReLU usually performs
better
• Thus, use of softplus
usually discouraged

52. Hyper-
parameter
Operating
principle
Structure
Behavior
Neuron
Layer
Weight
Connection
Operation
Activation
function

53. Hyperparameter
• Influence network and algorithm behavior
• Often influence model capacity
• Not learned, but usually manually optimized
• Currently quite some research interest in
automatic hyperparameter optimization
Examples
• Number of hidden layers
• Number of hidden units
• Learning rate
• Number of clusters
• Weight decay coefficient
• Convolution kernel width
• ...

54. Training
Deep
Learning
Operating
principle
Network
types
Structure
Behavior
Weight
Operation
Neuron
Layer
Connection
Hyper-
parameter
Activation
function

55. Quality
measure
Training

56. Cost function
Training
Quality
measure

57. Cost function (a.k.a. loss function)
• Determines distance from optimal performance
• Mean squared error as simple (and widespread) example

58. Cost function (a.k.a. loss function)
• Determines distance from optimal performance
• Mean squared error as simple (and widespread) example
• Often augmented with regularization term
for better generalization (see challenges)

59. Optimization
procedure
Training
Quality
measure
Cost function

60. Training
Quality
measure
Stochastic
gradient
descent
Cost function
Optimization
procedure

61. Stochastic gradient descent
• Direct calculation of minimum often not feasible
• Instead stepwise “descent” using the gradient
à Gradient descent

62. Stochastic gradient descent
• Direct calculation of minimum often not feasible
• Instead stepwise “descent” using the gradient
à Gradient descent
• Not feasible for large training sets
• Use (small) random sample of training set per iteration
à Stochastic gradient descent (SGD)

63. Stochastic gradient descent
Gradient
Direction
Steepness
x

64. Stochastic gradient descent
x
ε * gradient
x’
Learning
rate ε

65. Training
Quality
measure
Stochastic
gradient
descent
Back-
propagation
Cost function
Optimization
procedure

66. Backpropagation
• Procedure to calculate new weights based on loss function
Depends on
cost function
Depends on
activation function
Depends on
input calculation

67. Backpropagation
• Procedure to calculate new weights based on loss function
• Usually “back-propagated” layer-wise
• Most widespread optimization procedure
Depends on
cost function
Depends on
activation function
Depends on
input calculation

68. Data
Training
Quality
measure
Stochastic
gradient
descent
Back-
propagation
Cost function
Optimization
procedure

69. Training set
Validation/
Test set
Data
Training
Quality
measure
Stochastic
gradient
descent
Back-
propagation
Cost function
Optimization
procedure

70. Data set
• Consists of examples (a.k.a. data points)
• Example always contains input tensor
• Sometimes also contains expected output tensor
(depending on training type)
• Data set usually split up in several parts
• Training set – optimize accuracy (always used)
• Test set – test generalization (often used)
• Validation set – tune hyperparameters (sometimes used)

71. Data
Types
Training
Quality
measure
Stochastic
gradient
descent
Back-
propagation
Training set
Validation/
Test set
Cost function
Optimization
procedure

72. Supervised
Data
Types
Training
Quality
measure
Stochastic
gradient
descent
Back-
propagation
Training set
Validation/
Test set
Cost function
Optimization
procedure

73. Supervised learning
• Typically learns from a large, yet finite set of examples
• Examples consist of input and output tensor
• Output tensor describes desired output
• Output tensor also called label or target
• Typical application areas
• Classification
• Regression and function prediction
• Structured output problems

74. Unsupervised
Data
Types
Supervised
Training
Quality
measure
Stochastic
gradient
descent
Back-
propagation
Training set
Validation/
Test set
Cost function
Optimization
procedure

75. Unsupervised learning
• Typically learns from a large, yet finite set of examples
• Examples consist of input tensor only
• Learning algorithm tries to learn useful properties of the data
• Requires different type of cost functions
• Typical application areas
• Clustering, density estimations
• Denoising, compression (dimension reduction)
• Synthesis and sampling

76. Reinforcement
Data
Types
Supervised
Training
Quality
measure
Unsupervised
Stochastic
gradient
descent
Back-
propagation
Training set
Validation/
Test set
Cost function
Optimization
procedure

77. Reinforcement learning
• Continuously optimizes interaction with an environment
based on reward-based learning
Agent
Environment
State t
Reward t
State t+1
Reward t+1
Action t

78. Reinforcement learning
• Continuously optimizes interaction with an environment
based on reward-based learning
• Goal is selection of action with highest expected reward
• Takes (discounted) expected future rewards into account
• Labeling of examples replaced by reward function
• Can continuously learn à data set can be infinite
• Typically used to solve complex tasks in (increasingly)
complex environments with (very) limited feedback

79. Challenges
Data
Types
Supervised
Training
Quality
measure
Unsupervised
Reinforcement
Stochastic
gradient
descent
Back-
propagation
Training set
Validation/
Test set
Cost function
Optimization
procedure

80. Data
Types
Supervised
Training
Quality
measure
Unsupervised
Reinforcement
Stochastic
gradient
descent
Back-
propagation
Under-/
Overfitting
Training set
Validation/
Test set
Cost function
Challenges
Optimization
procedure

81. Underfitting and Overfitting
• Training error describes how good training data is learnt
• Test error is an indicator for generalization capability
• Core challenge for all machine learning type algorithms
1. Make training error small
2. Make gap between training and test error small
• Underfitting is the violation of #1
• Overfitting is the violation of #2

82. Good fit
Underfitting
Overfitting
Training data
Test data

83. Underfitting and Overfitting
• Under- and overfitting influenced by model capacity
• Too low capacity usually leads to underfitting
• Too high capacity usually leads to overfitting
• Finding the right capacity is a challenge

84. Data
Types
Supervised
Training
Quality
measure
Unsupervised
Reinforcement
Stochastic
gradient
descent
Back-
propagation
Under-/
Overfitting
Training set
Validation/
Test set
Cost function
Regularization
Challenges
Optimization
procedure

85. Regularization
• Regularization is a modification applied to learning algorithm
• to reduce the generalization error
• but not the training error
• Weight decay is a typical regularization measure

86. Data
Types
Supervised
Training
Quality
measure
Unsupervised
Reinforcement
Stochastic
gradient
descent
Back-
propagation
Under-/
Overfitting
Transfer
learning
Training set
Validation/
Test set
Cost function
Regularization
Challenges
Optimization
procedure

87. Transfer learning
• How to transfer insights between related tasks
• E.g., is it possible to transfer knowledge gained while training
to recognize cars on the problem of recognizing trucks?
• General machine learning problem
• Subject of many research activities

88. Network
types
Deep
Learning
Operating
principle
Training
Structure
Behavior
Weight
Operation
Neuron
Data
Types
Challenges
Quality
measure
Training
set
Cost
function
Transfer
learning
Regulari-
zation
Layer
Connection
Hyper-
parameter
Activation
function
Reinforce-
ment
Unsuper-
vised
Supervised
Stochastic
gradient
descent
Back-
propagation
Under-/
Overfitting
Validation/
Test set
Optimization
procedure

89. MLP
Multilayer
Perceptron
Network
types

90. Multilayer perceptron (MLP)
• Multilayer feed-forward networks
• “Vanilla” neural networks
• Typically used for
• Function approximation
• Regression
• Classification
Image source: https://deeplearning4j.org

91. CNN
Convolutional
Neural Network
Network
types
MLP
Multilayer
Perceptron

92. Convolutional neural network (CNN)
• Special type of MLP for image processing
• Connects convolutional neuron only with receptive field
• Advantages
• Less computing
power required
• Often even better
recognition rates
• Inspired by organization of visual cortex
Image source: https://deeplearning4j.org

93. RNN
Recurrent
Neural Network
Network
types
MLP
Multilayer
Perceptron
CNN
Convolutional
Neural Network

94. Recurrent neural network (RNN)
• Implements internal feedback loops
• Provides a temporal memory
• Typically used for
• Speech recognition
• Text recognition
• Time series processing
Image source: https://deeplearning4j.org

95. LSTM
Long
Short-Term
Memory
Network
types
MLP
Multilayer
Perceptron
CNN
Convolutional
Neural Network
RNN
Recurrent
Neural Network

96. Long short-term memory (LSTM)
• Special type of RNN
• Uses special LSTM units
• Can implement very long-term memory
while avoiding the vanishing/exploding
gradient problem
• Same application areas as RNN
Image source: https://deeplearning4j.org

97. Auto-
encoder
Network
types
MLP
Multilayer
Perceptron
CNN
Convolutional
Neural Network
RNN
Recurrent
Neural Network
LSTM
Long
Short-Term
Memory

98. Autoencoder
• Special type of MLP
• Reproduces input at output layer
• Consists of encoder and decoder
• Usually configured undercomplete
• Learns efficient feature codings
• Dimension reduction (incl. compression)
• Denoising
• Usually needs pre-training for not only
reconstructing average of training set
Image source: https://deeplearning4j.org

99. GAN
Generative
Adversarial
Networks
Network
types
MLP
Multilayer
Perceptron
CNN
Convolutional
Neural Network
RNN
Recurrent
Neural Network
Auto-
encoder
LSTM
Long
Short-Term
Memory

100. Generative adversarial networks (GAN)
• Consists of two (adversarial) networks
• Generator creating fake images
• Discriminator trying to identify
fake images
• Typically used for
• Synthesis and sampling
(e.g., textures in games)
• Structured output with variance (e.g., variations of a design or voice generation)
• Probably best known for creating fake celebrity images
Image source: https://deeplearning4j.org

101. Deep
Learning
Operating
principle
Training
Network
types
Structure
Behavior
Weight
Operation
Neuron
Data
CNN
Types
Challenges
Quality
measure
RNN
LSTM
Auto-
encoder
GAN
MLP
Training
set
Cost
function
Transfer
learning
Regulari-
zation
Layer
Connection
Hyper-
parameter
Activation
function
Reinforce-
ment
Unsuper-
vised
Supervised
Stochastic
gradient
descent
Back-
propagation
Under-/
Overfitting
Validation/
Test set
Optimization
procedure

102. How does Deep Learning feel in practice?

132. What issues might I face if diving deeper?

133. Issues you might face
• Very fast moving research domain
• You need the math. Really!
• How much data do you have?
• GDPR: Can you explain the decision of your network?
• Meta-Learning as the next step
• Monopolization of research and knowledge

134. Wrap-up

135. Wrap-up
• Broad, diverse topic
• Very good library support and more
• Very active research topic
• No free lunch
• You need the math!
à Exciting and important topic – become a part of it!

136. References
• I. Goodfellow, Y. Bengio, A. Courville, ”Deep learning",
MIT press, 2016, also available via https://www.deeplearningbook.org
• C. Perez, “The Deep Learning AI Playbook”,
Intuition Machine Inc., 2017
• F. Chollet, "Deep Learning with Python",
Manning Publications, 2017
• OpenAI, https://openai.com
• Keras, https://keras.io
• Deep Learning for Java, https://deeplearning4j.org/index.html
• Deep Learning (Resource site), http://deeplearning.net

137. @ShirinGlander
Dr. Shirin Glander
shirin.glander@codecentric.de
@ufried
Uwe Friedrichsen
uwe.friedrichsen@codecentric.de