Sunghoon Joo, Paper review slide. Title : MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks

1. MorphNet: Fast & Simple Resource-Constrained Structure
Learning of Deep Networks
주성훈, 삼성SDS AI선행연구Lab.
2019. 8. 11.
PR-187
Ariel Gordon1, Elad Eban1, Ofir Nachum2, Bo Chen1, Hao Wu1, Tien-Ju Yang1,3, and Edward Choi1,4
1 Google Research
2 Google Brain
3 Energy-efficient multimedia systems group, MIT
4 Georgia Institute of Technology
CVPR 2018

2. 1. Research Background

3. 1. Research Background
Introduction
2/20
• What & Why - Automated architecture search
- Automating structure design in DNNs is an active research field that is gaining significance as DNNs become
more ubiquitous in a variety of applications and platforms.
• Previous works
- Sparsifying regularizers (pruning, L1 regularization on weight matrices, designed regularizers
They do not target reduction of a particular resource (e.g., the number of floating point operations, or FLOPs, per inference).
- Optimizing every aspect of network structure (from trial-and-error attemps – RL, evolutionary
algorithm)
but, this methods require months or years of GPU time
• Automatic neural network architecture design is currently effective only under limited
conditions and given knowledge of the right tool to use.
Authors propose a simple and general technique for resource-constrained optimization of DNN architectures.

4. 1. Research Background 3/20
Three advantages :
• (1) it is scalable to large models and large datasets
• (2) it can optimize a DNN structure targeting a specific resource (FLOPs, model size).
• (3) it can learn a structures that improves performance while reducing the targeted resource usage.
MorphNet takes an existing neural network as input and produces a new
neural network that is smaller, faster, and yields better performance tailored
to a new problem.
Objective

5. 2. Methods

6. 2. Methods
Problem setup
5/20
• To optimize over the output widths of all layers
Seed network
Constraint (FLOPs, model size)
Output channels of layer L
a loss measuring a combination of how well the neural network fits the data
and any additional regularization terms

7. 2. Methods
A cycle of shrinking and expanding phases
6/20
Step 1-2: Shrinking
Consuming resource ↓
Performance ↓
(Sparsifying regularizer)
Step 3 : Expansion
(width multiplier)

8. 2. Methods
Two types of constraints
7/20
Model size constraint
FLOPs constraint
Input spatial dimensions : w,x
Output spatial dimensions : y, z
Filter dimensions : f, g

9. 2. Methods
1) Shrinking phase - Sparsifying regularizer (pruning)
8/20
• When shrinking a network, we wish to minimize the loss of the DNN subject to a constraint.
L0 norm : Corresponds to the total number of nonzero elements in a vector.
It is necessary to replace the discontinuous L0 norm with a continuous proxy norm.
Differentiable almost everywhere!
"Batch normalization: Accelerating deep network training by reducing internal covariate shift." (2015).

10. 2. Methods
2) Expanding phase - Width multiplier
9/20
• Uniformly expand all layer sizes
- "Mobilenets: Efficient convolutional neural networks for mobile vision applications." (2017).
- Application of a width multiplier is essentially free.
- The approach suffers, however, with decreased quality of the initial network design.

11. 3. Experimental Results

12. 3. Experimental Results
The effect of the regularizer on the actual FLOPs during training
11/20
• Inception V2 on ImageNet
Seed network : Inception V2
A learning rate of 10-3 (constant in time)

13. 3. Experimental Results
Applying constraint with different strengths (different values of λ)
12/20
0.7×10-9
1.0×10-9
1.3×10-9
2.0×10-9
3.0×10-9
• The accuracy of the DNN can be improved while maintaining a constrained resource usage (FLOPs in
this case).
Seed network : Inception V2, trained on ImageNet
A learning rate of 10-3 (constant in time)
Baseline method : Naïve width multiplier

14. 3. Experimental Results
Improvement the accuracy by shrink-expansion
13/20
λ=1.3×10-9
• Although the number of FLOPs is constant, our method is capable of and chooses to increase the
number of weights in the model.

15. 3. Experimental Results
Improved Performance at No Cost
14/20
• Evaluation MAP (mean-average-precision) on various seeds and data.
* JTL dataset : 350M images and about 20K labels
* Two iteration
* AudioSet : 20M labelled audio segments
1) "CNN architectures for large-scale audio classification." 2017
1)
• Each result requires up to three training runs.
• The first training run is run until the convergence of the FLOPs cost, which is approximately 20 times
faster than the convergence of the performance metric (MAP).

16. 3. Experimental Results
The regularizer methodically targets a particular resource
15/20
AudioResNet-AudioSet
ResNet 101-JFT
FLOP regularizer Size regularizer

17. • FLOP regularizer tends to remove neurons from the lower layers near the input,
whereas the model size regularizer tends to remove neurons from upper layers near the output.
3. Experimental Results
The regularizer methodically targets a particular resource
16/20
MAP : 0.405 MAP : 0.428 MAP : 0.421

18. 3. Experimental Results
Morphing Networks
17/20
https://ai.googleblog.com/2019/04/morphnet-towards-faster-and-smaller.html
• MorphNet can remove residual connections in ResNet-style networks,
and parallel towers in Inception-style networks.

19. 3. Experimental Results
The MorphNet is capable of generating pretty stable DNN architectures.
18/20
The relative standard deviation for FLOPs and test accuracy across 10 runs
are 1.12% and 0.208% respectively.

20. 4. Conclusion

21. 4. Conclusions 20/20
• Furthermore, we have applied MorphNet to large scale
problems to achieve improvements over human-designed
DNN structures, with little extra training cost compared to
training the DNN once.
Thank you.
• MorphNet can successfully navigate this tradeoff when
targeting either FLOPs or model size.
• As future work, the iterative process of shrinking and
expanding easily lends itself to optimizing over other aspects
(not width of layer output) of network design.