Terry Taewoong Um proposes deformable convolutional networks. The document discusses introducing learnable offsets to convolutional filters and region of interest pooling layers to allow the networks to spatially transform based on the input data. This helps the networks better adapt to objects of different scales and aspect ratios. Experimental results show deformable convnets achieve state-of-the-art performance on semantic segmentation, object detection and other tasks. Code is available online for others to experiment with these techniques.

1. Terry Taewoong Um (terry.t.um@gmail.com)
University of Waterloo
Department of Electrical & Computer Engineering
Terry T. Um
DEFORMABLE
CONVOLUTIONAL NETWORKS
1

2. TODAY’S PAPER
Terry Taewoong Um (terry.t.um@gmail.com)
Convolution
RoI pooling
Convolution + learnable offset
RoI pooling + learnable offset

3. 1. INTRODUCTION
Terry Taewoong Um (terry.t.um@gmail.com)
- data augmentation
- SIFT (scale invariant feature)
- Label-preserving augmentation?
https://goo.gl/GCf6q8
cs231n, Stanford
https://goo.gl
/fKvx8V

4. 1. INTRODUCTION
Terry Taewoong Um (terry.t.um@gmail.com)
There is no reason to use
“fixed-size” convolution filters
 Introduce learnable offset
Fig.5.

5. 1. INTRODUCTION
Terry Taewoong Um (terry.t.um@gmail.com)
Fig.1.
• RoI pooling
https://deepsense.io/region-of-
interest-pooling-explained/

6. 1. INTRODUCTION
Terry Taewoong Um (terry.t.um@gmail.com)
?
• Insert simple networks that determine
parameters for effective spatial transformations

7. 2. DEFORMABLE CONVNET
Terry Taewoong Um (terry.t.um@gmail.com)
x(3.7,2.3) = 0.7*0.3*x(4.0,3.0) +
0.7*0.7*x(4.0,2.0) + …
• Bilinear interpolation

8. 2. DEFORMABLE CONVNET
Terry Taewoong Um (terry.t.um@gmail.com)
https://deepsense.io/region-of-
interest-pooling-explained/

9. 2. DEFORMABLE CONVNET
Terry Taewoong Um (terry.t.um@gmail.com)
• Deformable convolution
• Deformable RoI pooling
 Any processes that are differentiable can
be learned by back propagation

10. 2. DEFORMABLE CONVNET
Terry Taewoong Um (terry.t.um@gmail.com)
- Deep Lab : SOTA semantic segmentator
- Category-aware RPN : a simplified SSD
- Faster R-CNN : SOTA object detector
- R-FCN : SOTA object detector
(per-RoI computation cost )
(I hope other members will have a chance
to present on these SOTA methods in the
near future)

11. 3. UNDERSTANDING D-CONVNET
Terry Taewoong Um (terry.t.um@gmail.com)
background small obj large obj
Fig.4.
Fig.5.
Table.2.

12. 3. UNDERSTANDING D-CONVNET
Terry Taewoong Um (terry.t.um@gmail.com)
Fig.6.
3*3 bins
 deformed

13. 3. UNDERSTANDING D-CONVNET
Terry Taewoong Um (terry.t.um@gmail.com)
https://github.com/felixlaumon

14. 3. UNDERSTANDING D-CONVNET
Terry Taewoong Um (terry.t.um@gmail.com)

15. 3. UNDERSTANDING D-CONVNET
Terry Taewoong Um (terry.t.um@gmail.com)

16. 4. EXPERIMENT
Terry Taewoong Um (terry.t.um@gmail.com)

17. Terry Taewoong Um (terry.t.um@gmail.com)