Presentations on Video Resampling algorithms

1. Tejus Adiga M
Department of Electronics and Communication, NMAMIT, Nitte.
Presented By:

2. Resampling
• Ideal resampling: Discrete -> Continuous -> Discrete.
• Practical resampling: Done entirely in discrete domain.
• Types of Resampling:
• Downsampling: Decrease size by M.
• Upsampling: Increase size by N.
• Fractional Resampling: Increase size my M and decrease by N (M/N).
• Traditional Methods:
• Blind Resampling: 2D Convolution. Eg Kernels Nearest Neighbor, Bilinear,
Bicubic, Bspline.
• Content Aware Resampling: Seam Carving, Edge Directed Interpolation (EDI),
Super Resolution.
• Seperability: 2D filtering = Performing 1D filtering two times in
each dimension one after another.
28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
2

3. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
3
Downsampling
• General Approach:
Anti Alias Filter
(LPF)
Downsampler
↓ 𝑁
Image 𝑊𝑥𝐻
Downsampled
Image
𝑊
2
𝑥
𝐻
2
• Practical Approach
𝑦 𝑚 =
𝑘=−𝑁/2
𝑁/2
𝐶 𝑘 𝑥2𝑚 −𝑘−1 0 ≤ 𝑚 ≤ 𝑊, 𝐻. 𝑁 𝑖𝑠 𝑓𝑖𝑙𝑡𝑒𝑟 𝑙𝑒𝑛𝑔𝑡ℎ
Downsampling
Kernel
Downsampled Image
𝑊
2
𝑥
𝐻
2
Image 𝑊𝑥𝐻
• Convolution
m-2 m-1 m m+1 m+2 m+3
Downscaled Image
Original Image
Fig 1: Downsampling process
x
Ck1Ck1 Ck2Ck2
hd(x)
0
1 Pixel
Distance
Fig 2: Downsampling Kernel

4. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
4
Upsampling
• General Approach:
Upsampler
↑ 𝑀
Low Pass FilterImage 𝑊𝑥𝐻
Upsampled Image
2𝑊 𝑥 2𝐻
• Practical Approach
𝑦2𝑚+1 =
𝑘=−𝑁/2
𝑁/2
𝐶 𝑘 𝑥 𝑚 −𝑘−1 0 ≤ 𝑚 ≤ 𝑊, 𝐻. 𝑁 𝑖𝑠 𝑓𝑖𝑙𝑡𝑒𝑟 𝑙𝑒𝑛𝑔𝑡ℎ
Upsampling Kernel
Upsampled Image
2𝑊 𝑥 2𝐻
Image 𝑊𝑥𝐻
• Convolution
Upscaled Image
m-2 m-1 m m+1 m+2 m+3
Original Image
m-3 m+4
Fig 3: Upsampling process
hu(k)
x
0
Fig 4: Upsampling Kernel

5. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
5
Blind Resampling – Nearest Neighbor
Fig 5: Spatial kernel and Frequency Response
• Downsampling: Discard Every alternate pixel.
• Upsampling: Replicate the Nearest Pixel.
• Artifacts: Aliasing-Increase 4 times for two fold resample.
• Kernel: Rectangular spatial kernel. Infinite frequency contents.

6. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
6
Blind Resampling – Nearest Neighbor
Downsampled by 4
Downsampled by 2
Captured Image
Upsampled by 4
Fig 6: Downsampled and
Upsampled by factor of 2 and 4
Upsampled by 2

7. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
7
Blind Resampling – Bilinear
Fig 7: Spatial kernel and Frequency Response
• Downsampling and Upsampling: Average of two pixels. (4 pixels in 2D)
• Artifacts: Aliasing, Blurring.
• Filter Coefficients: ℎ 𝑏 𝑥 =
1
2
,
1
2
2
= 0.5, 0.5
• Kernel: Triangular or Tent Spatial kernel.
• Frequency response: Stop band attenuation better than Nearest Neighbor.
• Aliasing is reduced when compared to nearest Neighbor.

8. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
8
Blind Resampling – Bilinear
Captured Image
Upsampled by 4
Fig 8: Downsampled and
Upsampled by factor of 2 and 4
Upsampled by 2
Downsampled by 4
Downsampled by 2

9. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
9
Blind Resampling – Bicubic
Fig 9: Spatial kernel and Frequency Response
• Downsampling and Upsampling: Weighted average of 4 pixels.
• Artifacts: Blurring.
• Filter: ℎ 𝑏 𝑥 =
3
2
𝑥 3 −
5
2
𝑥 2 + 1, 0 ≤ 𝑥 ≤ 1
−
1
2
𝑥 3 +
5
2
𝑥 2 − 4 𝑥 + 2, 1 < 𝑥 ≤ 2
0, 𝑥 > 2
• Frequency response: Stop band attenuation better than Bilinear.
• The 1st negative side lobe introduce controlled Ringing effect which makes image
appear sharper than they actually are.

10. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
10
Blind Resampling – Bicubic
Captured Image
Upsampled by 4
Fig 10: Downsampled and
Upsampled by factor of 2 and 4
Upsampled by 2
Downsampled by 4
Downsampled by 2

11. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
11
Blind Resampling – Windowed Sinc
Fig 11: Windowed Sinc Kernel
• Artifacts: Ringing, Blurring.
• Filter: Truncated Sinc function. ℎ 𝑏 𝑥 =
sin
𝜋𝑥
2
𝜋𝑥
2
• Side lobes significantly contributes to Ringing.

12. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
12
Blind Resampling – Lanczos
Fig 12: Lanczos Kernel for a=2 and a=3
• Weighted Average of 4 pixels.
• Artifacts: Blurring.
• Filter: ℎ 𝑏 𝑥 = 𝑠𝑖𝑛𝑐 𝑥 𝑠𝑖𝑛𝑐
𝑥
𝑎
− 𝑎 ≤ 𝑥 ≤ 𝑎 𝑎𝑛𝑑 𝑎 = 2, 3
• ‘a’ indicates number of lobes in one half of the filter.
• Effect of side lobes is decreased by multiplying another scaled sinc function.
But stronger enough to make image look sharper.
• Upsampled image sharper than Bicubic, Bilinear.

13. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
13
Blind Resampling – Lanczos
Captured Image
Upsampled by 4
Fig 13: Downsampled and
Upsampled by factor of 2 and 4
Upsampled by 2
Downsampled by 4
Downsampled by 2

14. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
14
Artifacts in Image Resampling
• Aliasing:
• Jagged Edges.
• Introduced in Downsampling and Enhanced in Upsampling.
• Priority: Lanczos, Bicubic, Bilinear, nearest Neighbor.
• Ringing:
• Side lobes of lengthy filter contribute to false edges.
• Optimal filter length 4.
• Windowed Sinc.
• Blurring:
• LPF gains get multiplied while Downsampling and Upsampling.
• Information lost during Downsampling is irreversible. So in upsampling pixels
are filled with he help of existing information in Downsampled image.
• Priority: Nearest Neighbor, Bilinear, Lanczos, Bicubic.

15. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
15
Artifacts
Ringing Example
Aliasing Example
Blurring Example
Fig 14: Examples of Ringing,
Aliasing and Blurring

16. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
16
Content Aware Resampling – Seam Carving
• Seam: 8-Connected set of pixels that runs from top to bottom or Left to Right.
• Principle: Low energy seam is not appealing to eyes.
• Applications:
• Image Retargeting: Resizing image, Changing Aspect Ratio.
• Object removal or insertion.
• Algorithm:
• Find the Gradient Map of the input image I.
𝐺 𝑥, 𝑦 =
𝜕𝐼
𝜕𝑥
+
𝜕𝐼
𝜕𝑦
• In Gradient Map search a unique path (seam) from top to bottom or left
to right such that Energy of the seam is minimum than all other possible
seam.
𝑠 = min
𝑠
𝑊/𝐻
𝑔(𝑥, 𝑦)
• Remove the seam or Duplicate the seam from the image I and G which
reduces/increases the width/height by 1 pixel.
• Iterate the above steps until desired size is achieved.

17. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
17
Content Aware Resampling – Seam Carving
Fig 15: Seam Calculation using Gradient.

18. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
18
Content Aware Resampling – Seam Carving
Fig 16: Comparison of Seam Carving with Scaling

19. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
19
Seam Carving – Failure Cases

20. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
20
Seam Carving – Failure Cases

21. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
21
Content Aware Resampling –
New Edge Directed interpolation (NEDI)
• Principle: Inter pixel relations are retained while Downscaling. Hence
Covariance in original Image and Downscaled image are nearly same.
• New Pixel Value = Weighted Sum of nearest 4 pixels. Weights are computed
dynamically according to local image characteristics.
𝑌2𝑖+1,2𝑗+1 =
𝑘=0
1
𝑙=0
1
𝛼2𝑘+𝑙 𝑌2 𝑖+𝑘 ,2 𝑗+𝑙
𝛼 = 𝑅−1
𝑟
𝑅 =
1
22
𝐶 𝑇
𝐶, 𝑎𝑛𝑑 𝑟 =
1
22
𝐶 𝑇
𝑦
Where 𝑦 = [𝑦1 … 𝑦 𝑘 … 𝑦22] 𝑇
is the data vector
containing the 2x2 pixels inside the local window
and C is a 4x22 data matrix whose kth column
vector is the four nearest neighbors of along the
diagonal direction.

22. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
22
Content Aware Resampling – (NEDI)
Fig 16: Comparison of NEDI with Bicubic filter
Downscaled Image
Upscaled 4X using NEDI Upscaled 4X using Bicubic

23. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
23
Conclusion
• Trade off:
• Quality
• Speed of Operation
• Requirement
• Information lost during Downsampling cannot be recovered while upsampling.
• Future Work:
• Improvement of Content Aware Resizing methods.
• Adding Resolution.

24. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
24
References
1. New Edge-Directed Interpolation, Xin Li and Michael T. Orchard. IEEE TRANSACTIONS ON
IMAGE PROCESSING, VOL. 10, NO. 10, OCTOBER 2012.
2. Local and Nonlocal Regularization to Image Interpolation, Yi Zhan, Sheng Jie Li, and Meng Li,
Mathematical Problems in Engineering, Volume 2014 (2014), Article ID 230348.
3. Adaptive multidirectional edge directed interpolation for selected edge regions. TENCON 2011 -
2011 IEEE Region 10 Conference.
4. V.R. Algazi, G.E. Ford and R. Potharlanka, "Directional interpolation of images based on visual
properties and rank order filtering", Proceeding of ICASSP' 1991, pp.3005-3008.
5. Seam Carving for Content-Aware Image Resizing. Shai Avidan and Ariel Shamir. Proceedings of
ACM SIGGRAPH, 417–424.
6. J. Allebach and P.W. Wong, "Edge-directed interpolation", Proceeding of ICIP 1996, Page No
707-710.
7. Keys, R., “Cubic Convolution Interpolation for Digital Image Processing”, IEEE Trans on ASSP, vol
ASSP-29, No. 6, Page No 1153-1160. Dec 1981.
8. New Filters for Image Interpolation and Resizing, Amir Said, IEEE International Conference on
Image Processing, VOL. 8, 2007.
9. Image Zooming Methods, Bax Smith.
10. “Interpolation Theory”
http://sepwww.stanford.edu/public/docs/sep107/paper_html/node20.html