2019/02/06 論文輪読会 #17 発表資料

1. Batch Normalization:
Accelerating Deep Network Training
by Reducing Internal Covariate Shift
#17
2019/02/06
@iiou16_tech

2. abstract
Deep Neural Networks
Batch Normalization
dropOut
Batch Normalization 14 1
ImageNet
4.9 5 4.8

3. outline
1. Introduction
2. Towards Reducing Internal Covariate Shift
3. Normalization via Mini-Batch Statistics
1. Training and Inference with Batch-Normalized
Networks
2. Batch-Normalized ConvolutionalNetworks
3. Batch Normalization enables higher learning rates
4. Batch Normalization regularizes the model
4. Experiments
1. Activations over time
2. ImageNet classification
5. Conclusion

4. Introduction
• Deep Learning SGD
– x θ l
θ
–
• ( ) ( m
)
• 1
•

5. Introduction
• covariate shift
–
DNNx
DNNx’
x DNN x’

6. Introduction
• covariate shift
x
F1 F2
DNN
F2 F1
→F1 F2 x

7. 2 Towards Reducing Internal Covariate
Shift
•
–
(DNN)
– DNN
– 0 1
( )
– 1

8. 2 Towards Reducing Internal Covariate
Shift
•
•
• itr
• (SGD)
•

9. 3 Normalization via Mini-Batch Statistics
•
• 1
• 0 1
•
• γ β x
• 2

10. 3 Normalization via Mini-Batch Statistics
•
• 2
• 0 1 SGD
DNN
•
/
• itr

11. 3.1 Training and Inference with
BatchNormalized Networks
•
• /
•
• /
•
activation
/
/

12. 3.2 Batch-Normalized Convolutional
Networks
• Convolutionarl
• ( )
•
BN
• m
• Conv BN p*q
m*p*q
• Conv BN 2*

13. 3.3 Batch Normalization enables higher
learning rates
•
• BN
a
1/a

14. 3.4 Batch Normalization regularizes the
model
•
• BN
• DropOut

15. 4 Experiments
4.1 Activations over time
•
BN
• MNIST
• 3 NN ( )
• 60 50000
BN BN

16. 4.2 ImageNet classification
• Inception ImageNet
• Relu
• CNN layer 5*5( )→3*3 ×2
• batch size = 32
• Optimiser : Momentum SGD
https://arxiv.org/pdf/1409.4842.pdf

17. 4.2.1 Accelerating BN Networks
• BASE Inception BN
• BN
•
• DropOut
• L2 Weight regularization 1/5
• 6
•
• 1%
• photometric distortion
•
• Local Response Normalization
https://arxiv.org/pdf/1409.4842.pdf

18. 4.2.2 Single-Network Classification
BN-x5LSVRC2012
lr=0.0015
BN
4.2.1
lr=0.0075
4.2.1
lr=0.045
BN-x5
Leru→sigmoid

19. 4.2.3 Ensemble Classification
• ImageNet Best Result
• BN-x30 6
SoTA
• DropOut (5% or 10%)

20. Conclusion(1/2)
•
• NN
• activation
DNN
• SGD
BN 2
• BN
• BN
• dropOut
• BN ImageNet

21. Conclution(2/2)
• Standardization layer
• BN
• future work
• Recurrent Neural Networks BN
• / BN
• domain adaptation
•
Batch Normalization BN→ , 2
Standardization layer SL no paramater
activation
activation

22. 1
• BN google
• https://patents.google.com/patent/US20160217368A1/en
A neural network system implemented by one or more computers, the neural network system comprising:
a batch normalization layer between a first neural network layer and a second neural network layer, wherein the
first neural network layer generates first layer outputs having a plurality of components, and wherein the batch
normalization layer is configured to, during training of the neural network system on a batch of training examples:
receive a respective first layer output for each training example in the batch;
compute a plurality of normalization statistics for the batch from the first layer outputs;
normalize each component of each first layer output using the normalization statistics to generate a respective
normalized layer output for each training example in the batch;
generate a respective batch normalization layer output for each of the training examples from the normalized layer
outputs; and
provide the batch normalization layer output as an input to the second neural network layer.
• ※
• https://www.slideshare.net/YosukeShinya/ss-125937523
• by 50 @2018/12/15

23. 2
• BN
• Group Normalization
• fixup