안녕하세요 딥러닝 논문읽기 모임 입니다! 오늘 소개 드릴 논문은 Once-for-All: Train One Network and Specialize it for Efficient Deployment 라는 제목의 논문입니다.
모델을 실제로 하드웨어에 Deploy하는 그 상황을 보고 있는데 이 페이퍼에서 꼽고 있는 가장 큰 문제는 실제로 트레인한 모델을 Deploy할 하드웨어 환경이 너무나도 많다는 문제가 하나 있습니다 모든 디바이스가 갖고 있는 리소스가 다르기 때문에 모든 하드웨어에 맞는 모델을 찾기가 사실상 불가능하다는 문제를 꼽고 있고요
각 하드웨어에 맞는 옵티멀한 네트워크 아키텍처가 모두 다른 상황에서 어떻게 해야 될건지에 대한 고민이 일반적 입니다. 이제 할 수 있는 접근중에 하나는 각 하드웨어에 맞게 옵티멀한 아키텍처를 모두 다 찾는 건데 그게 사실상 너무나 많은 계산량을 요구하기 때문에 불가능하다라는 문제를 갖고 있습니다 삼성 노트 10을 예로 한 어플리케이션의 requirement가 20m/s로 그 모델을 돌려야 된다는 요구사항이 있으면은 그 20m/s 안에 돌 수 있는 모델이 뭔지 accuracy가 뭔지 이걸 찾기 위해서는 파란색 점들을 모두 찾아야 되고 각 점이 이제 트레이닝 한번을 의미하게 됩니다 그래서 사실상 다 수의 트레이닝을 다 해야지만 그 중에 뭐가 최적인지 또 찾아야 합니다. 실제 Deploy해야 되는 시나리오가 늘어나면 이게 리니어하게 증가하기 때문에
각 하드웨어에 맞는 그런 옵티멀 네트워크를 찾는게 사실상 불가능합니다.
그래서 이제 OFA에서 제안하는 어프로치는 하나의 네트워크를 한번 트레이닝 하고 나면 다시 하드웨어에 맞게 트레이닝할 필요 없이 그냥 각 환경에 맞게 가져다 쓸 수 있는 서브네트워크를 쓰면 된다 이게 주로 메인으로 사용하고 있는 어프로치입니다.
오늘 논문 리뷰를 위해 펀디멘탈팀 김동현님이 자세한 리뷰를 도와주셨습니다 많은 관심 미리 감사드립니다!
Vastral Call Girls Book Now 7737669865 Top Class Escort Service Available
Once-for-All: Train One Network and Specialize it for Efficient Deployment
1. Once-for-All: Train One Network and
Specialize it for Efficient Deployment
[ICLR 2020]
2022. 03. 20. (Sun)
Presented by: 김동현
w/ Fundamental Team: 김채현, 박종익, 양현모, 이근배, 이재윤, 송헌
1
2. Contents
● Problem and Approach
● Key Challenge
● How to Train Once-for-all Network
● How to Deploy Once-for-all Network
● Evaluations
● Discussions
● Conclusion
2
3. Contents
● Problem and Approach
● Key Challenge
● How to Train Once-for-all Network
● How to Deploy Once-for-all Network
● Evaluations
● Discussions
● Conclusion
3
4. Main Problem to Solve
● There are various hardware platforms to deploy DNN models.
○ Survey says there are 23.14 billion IoT devices until 2018.
○ The devices have different resource constraints;
It is impossible to deploy the same model to all devices.
● The optimal neural network architecture varies by deployment environments
(e.g., #arithmetic units, application requirements).
4
5. Main Problem to Solve
● It is computationally prohibitive to find all the optimal architecture by training
on each environment.
● Then, how is it possible to cost-efficiently find the specialized model on
each platform?
5
target latency
= 20ms
6. Suggested Approach
● Train a Once-for-all(OFA) network, which enables serving on various
environment without additional training.
○ Various scales of sub-networks (about 1019
) are available from one OFA network.
○ Each hardware can find the specialized model for its requirements (e.g, latency).
6
7. Key Challenges for Once-for-All Network
Requirements
1. The sub-network architecture should be part of the largest network.
2. Sub-networks should share parameters with larger networks.
3. Optimal model architecture for specified hardwares should be easily found.
7
8. Key Challenges for Once-for-All Network
Requirements
1. The sub-network architecture should be part of the largest network.
2. Sub-networks should share parameters with larger networks.
3. Optimal model architecture for specified hardwares should be easily found.
Challenges
1. How to design sub-network architecture space based on a OFA network.
2. How to let sub-networks share parameters with larger networks.
3. How to select the optimal model for the hardware (in terms of latency,
accuracy).
8
9. Contents
● Problem and Approach
● Key Challenge
● How to Train Once-for-all Network : Challenges #1, #2
● How to Deploy Once-for-all Network: Challenges #3
● Evaluations
● Discussions
● Conclusion
9
11. ● Assumption: Follow the common practice of CNN models (e.g., ResNet).
○ A model consists of groups of Layers (i.e., units).
● Architecture Search Space
○ # Layers(L): the depth of each unit is chosen from {2, 3, 4}
○ # Channels(C): expansion ratio in each layer is chosen from {3, 4, 6}
○ Kernel Size(Ks): {3, 5, 7}
○ Input Dimension: ranges from 128 to 224 with a stride
● Num available sub-networks: ((3 * 3)2
+ (3 * 3)3
+ (3 * 3)4
)5
= about 1019
Training OFA Network - Network Architecture
… … …
…
L1 L2 L3
C
…
Ks
# units
11
12. How sub-networks share parameters:
● Elastic Kernel Size
○ Merely sharing the parameters of larger kernel can affect the performance.
○ When changing kernel size, pass through Transform Matrix:
■ For each layer, hold parameters for elastic kernels.
● # 25*25 parameters for 7x7 -> 5x5.
● # 9*9 parameters for 5x5 -> 3x3.
● E.g., 5x5 kernel = (Center of 7x7) * Transform Matrix
Training OFA Network - Sharing Parameters
12
13. How sub-networks share parameters:
● Elastic Depth (= #Layers)
○ The first D layers are shared when L layers exist in a unit.
○ Simpler depth settings compared to selecting random layers from L layers.
Training OFA Network - Sharing Parameters
L D
13
14. How sub-networks share parameters:
● Elastic Width (= #Channels)
○ For the given expansion ratio, select channels through a channel sorting method:
1. Calculate L1 Norm for each channel’s weights.
2. Sort the channels by the L1 Norm order.
3. Choose the top-K channels.
Training OFA Network - Sharing Parameters
L1 Norm
14
15. Progressive Shrinking
1. Train a full model (i.e. max vaule for each configuration).
● With the trained full-size model, Knowledge-Distillation techniques are leveraged.
● Note: Full model != Best model
Training OFA Network - Training Process
… … …
…
L1 L2 L3
Note1: Input image size is randomly chosen for each training batch
15
16. Progressive Shrinking
1. Train a full model (i.e. max vaule for each configuration).
2. Sample sub-networks varying kernel sizes and fine-tune.
a. For each step, sample one sub-net with different kernel sizes.
b. Calculate Loss. Loss = Full model loss * KD_raio + sub-net loss
c. Update the weights (updating sub-net’s weight -> updating the full model’s weight)
Training OFA Network - Training Process
… … …
L1 L2 L3 16
Note1: Input image size is randomly chosen for each training batch
17. Progressive Shrinking
1. Train a full model (i.e. max vaule for each configuration).
2. Sample sub-networks varying kernel sizes and fine-tune.
3. Sample sub-networks varying depth and fine-tune.
4. Sample sub-networks varying channel expansion ratio and fine-tune.
Training OFA Network - Training Process
… … …
L1 L2 L3
Note2: Refer to Appendix B for impl. details of progressive shrinking
Note1: Input image size is randomly chosen for each training batch
17
18. Deploying Specialized Model w/ OFA Network
Problem:
● derive the specialized sub-network for a given deployment scenario (e.g.,
latency constraints).
Solution:
● Train an accuracy predictor (3-layer FFNN)
○ f(architecture, input image size) => accuracy
○ randomly sample 16K sub-networks, measure the accuracy on 10K validation images
● Latency Lookup Table (Details in the ProxylessNAS paper)
○ On each hardware platform, build a latency lookup table .
● Conduct an evolutionary search leveraging the above information.
○ Mutate from the known sub-network by sampling and predicting the performance.
○ add the mutated sub-network to the child pool if it satisfies the constraint (latency).
18
20. Evaluation
● ImageNet Dataset
● Eval on Various Hardware Platforms:
○ Samsung S7 Edge, Note8, Note10, Google Pixel1, Pixel2, LG G8, NVIDIA 1080Ti, V100
GPUs, Jetson TX2, Intel Xeon CPU, Xilinx ZU9EG, and ZU3EG FPGAs
● Please refer to the paper for the detailed training configurations.
20
21. Evaluation
Performance of sub-networks on ImageNet
● top-1 accuracy under 224x224 resolution.
● Can achieve higher performance through Progressive Shrinking.
○ 74.8% top1 accuracy (D=4, W=3, K=3), which is on par with MobileNetV3-Large.
○ Without PS, it achieves 71.5%, which is 3.3% lower.
21
get the same architecture from
the full model w/o PS
22. Evaluation
Reduced Design Cost
● reports comparison between OFA and hardware-aware NAS methods
○ NAS: The design cost is linear to the number of deployment scenarios (N).
○ the total CO2 emissions of OFA is:
■ 16× fewer than ProxylessNAS
■ 19× fewer than FBNet
■ 1,300× fewer than MnasNet
22
23. Evaluation
OFA under Different Computational Resource Constraints
● Better accuracy under the same constraints:
○ (Left): MACs, (Right): Latency
○ Achieves higher accuracy, Requires lower computations
○ Better than “OFA - Train from scratch”, which is trained from the scratch without pretraining.
23
24. Discussions
● Would it work if the same approach is applied to other models, tasks (e.g.,
Transformer, NLP)?
● The architecture search space is limited to certain models.
○ e.g. How to apply the method to models such as HRNet?
24
25. Conclusion
● Once-for-all(OFA) Network allows training one large model and deploying
various sub-networks without additional training.
● OFA suggests Progressive Shrinking algorithm to share and find
sub-networks, which highly reduces the design cost.
● The paper shows OFA can achieve higher performance with ImageNet
dataset.
● With a trained OFA network, optimal sub-networks can be found on various
deployment environments.
25