요즘 대형 비전 트랜스포머(ViT)의 발전에 비해, 합성곱 신경망(CNN)을 기반으로 한 대형 모델은 아직 초기 단계에 머물러 있습니다. 본 연구는 InternImage라는 새로운 대규모 CNN 기반 모델을 제안합니다. 이 모델은 ViT와 같이 매개변수와 학습 데이터를 늘리는 이점을 얻을 수 있습니다. 최근에는 대형 밀집 커널에 초점을 맞춘 CNN과는 달리, InternImage는 변형 가능한 컨볼루션을 핵심 연산자로 사용합니다. 이를 통해 모델은 감지 및 세분화와 같은 하향 작업에 필요한 큰 유효 수용영역을 갖게 되며, 입력 및 작업 정보에 의존하는 적응형 공간 집계도 가능합니다. 이로 인해, InternImage는 기존 CNN의 엄격한 귀납적 편향을 줄이고, ViT와 같은 대규모 매개변수와 대규모 데이터로 더 강력하고 견고한 패턴을 학습할 수 있게 됩니다. 논문에서 제시한 모델의 효과성은 ImageNet, COCO 및 ADE20K와 같은 어려운 벤치마크에서 입증되었습니다. InternImage-H는 COCO test-dev에서 65.4 mAP, ADE20K에서 62.9 mIoU를 달성하여 현재 최고의 CNN 및 ViT를 능가하는 새로운 기록을 세웠습니다

1. InternImage: Exploring Large-Scale Vision
Foundation Models with
Deformable Convolutions
발표자: 김병현
이미지처리팀:
강인하, 김현진, 안종식, 이주영, 이희재, 현청천

2. InterImage
 State-of-the-Art (COCO test-dev) Backbone Network
Released by OpenGVLab
2

3. Introduction
 Success of Transformers in Computer Vision Tasks
 CNN-based foundation models can also achieve
comparable or even better performance than ViTs when
equipped with similar operator-/architecture-level designs,
scaling-up parameters, and massive data.
3

4. Introduction
 Gap between CNNs and ViTs
Operator Level
• Long-range dependency
• Adaptive spatial aggregation
Architecture View
• Advanced components
– Layer Normalization
– Feed Forward Networks
– GELU
Recent Long-Range CNNs
• Very large kernels (31x31)
• Gap with SOTA ViTs
4

5. Introduction
 Comparison of different core operators
5

6. Introduction
 Global Attention: Vision Transformer
6
Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers
for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).
Architecture of ViT

7. Introduction
 Local Attention: Swin Transformer
7
Architecture of Swin Transformer
Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using
shifted windows." Proceedings of the IEEE/CVF international conference
on computer vision. 2021.

8. Introduction
 Large Kernel: SLaK
8
Liu, Shiwei, et al. "More convnets in the 2020s: Scaling up kernels beyond
51x51 using sparsity." arXiv preprint arXiv:2207.03620 (2022).

9. Introduction
 Concentration on CNN-based Model
InterImage
• Brand-New CNN-based Backbone Network
• Characteristics
– Dynamic sparse convolutional layer
» Only with 3x3 kernels
» Adaptive spatial aggregation
» Reduce inductive bias
» Low computational cost compared to large convolutional layers
– Overall Architecture of ViT
9

10. Introduction
 Contributions
1st CNN-based backbone with more than 1 billion params.
Add long-rage dependencies and adaptive spatial aggregation
with 3x3 DCN
SOTA accuracy in COCO dataset
10

11. Q & A
Q & A
11

12. Overall Architecture
12
1. Deformable
Convolutional Layer V3
2. Architecture Design

13. Deformable Convolutional Layer v3
 Revisiting DCNv2
13

14. Deformable Convolutional Layer v3
14
https://www.taokong.org/report/DCN_v2_paper_reading.pdf

15. Deformable Convolutional Layer v3
15
https://www.taokong.org/report/DCN_v2_paper_reading.pdf

16. Deformable Convolutional Layer v3
1. Sharing weights among convolutional neurons.
• Heavy computational cost of DCNv2
• independent linear projection weights
• memory complexity is linear with the total number of sampling points
• To remedy this problem, we borrow the idea from the separable
convolution and detach the original convolution weights into
depth-wise and point-wise parts
2. Introducing multi-group mechanism
• Split the spatial aggregation process into G groups
3. Normalizing modulation scalars along sampling points
• Change element-wise sigmoid normalization to softmax
normalization along sample points.
16

17. Deformable Convolutional Layer v3
17

18. Deformable Convolutional Layer v3
 Normal Convolutional Layer
18
ℎ
𝑤
𝑑
𝑘
𝐷
𝑘
𝐻
𝑊
𝐷
Input Tensor
Convolutional Kernel
Output Tensor

19. Deformable Convolutional Layer v3
 Deformable Convolutional Layer v1
19
ℎ
𝑤
𝑑
Input Tensor
𝑘
𝐷
𝑘
Convolutional Kernel
𝐻
𝑊
𝐷
Output Tensor
𝑘
2𝑁(𝑁 = 𝑘2
)
𝑘
Offset Layer
(Convolutional Kernel)
𝐻
𝑊
Offset map
2𝑁(𝑁 = 𝑘2
)

20. Deformable Convolutional Layer v3
 Deformable Convolutional Layer v2
20
𝑘
2𝑁(𝑁 = 𝑘2
)
𝑘
Offset Layer
(Convolutional Kernel)
𝐻
𝑊
Offset map
2𝑁(𝑁 = 𝑘2
)
ℎ
𝑤
𝑑
Input Tensor
𝑘
𝐷
𝑘
Convolutional Kernel
𝐻
𝑊
𝐷
Output
Tensor
𝑘
𝑁(𝑁 = 𝑘2
)
𝑘
Modulation Layer
(Convolutional Kernel)
𝐻
𝑊
Modulation map
𝑁(𝑁 = 𝑘2
)
Sigmoid

21. Deformable Convolutional Layer v3
 Deformable Convolutional Layer v3
21
𝑘
2𝑁(𝑁 = 𝑘2
)
𝑘
Offset Layer
(Convolutional Kernel*)
𝐻
𝑊
Offset map
2𝑁(𝑁 = 𝑘2
)
ℎ
𝑤
𝑑 = 𝐶′ × 𝐺
Input Tensor
𝑘
𝐶′
𝑘 Convolutional Kernel
𝐻
𝑊
𝐷
Output
Tensor
𝑘
𝑁(𝑁 = 𝑘2
)
𝑘
Modulation Layer
(Convolutional Kernel*)
𝐻
𝑊
Modulation map
𝑁(𝑁 = 𝑘2
)
Softmax
(Axis -> Depth)
× 𝐺
*Details not found in the paper
Shared Params
in Kernels

22. InterImage Model
22

23. InterImage Model
23
1. Basic Block

24. InterImage Model
24
2. Stem layer & Downsampling

25. InterImage Model
25
3. Stacking Rules

26. InterImage Model
26
1. Basic Block

27. InterImage Model
27
2. Stem & Downsampling

28. InterImage Model
28
3. Stacking Rules

29. InterImage Model
29
4. Scaling Rules

30. InterImage Model
30
5. Hyper-parameters for models of different scales
4 Stages Models are prevalent since DETR
• Swin Transformer
• MetaFormer

31. Q & A
Q & A
31

32. Experiments
 Image Classification (Tiny Model)
32

33. Experiments
 Image Classification (Large Model)
33

34. Experiments
 Object Detection & Instance Segmentation
34

35. Experiments
 Object Detection & Instance Segmentation
35

36. Experiments
 Semantic Segmentation
36

37. 이미지처리팀 리뷰 의견
 Deformable Conv V3에 대한 분석이 부재
Ablation Study 부재
• ResNet 등 기존 Conv 기반 모델에서 성능향상을 가지는지 확인해줬으면
정말 좋았을 듯
Deformable Conv V3의 장점에 대한 정성적인 분석 부재
• Convolution을 Group으로 나누면서 생기는 단일 레이어의 다양한 Offset
Map의 장점에 대한 시각화 자료가 있었으면 좋았을 듯
Inductive Bias를 줄일 수 있었다는 주장에 대한 근거자료 부재
 코드가 아직 공개되지 않아 정확한 검증은 어려움
37

38. Q & A
Q & A
38