This Master Thesis includes three main parts: first is a real-time video de-noising filter, second is a rate-distortion oriented joint video pre-filtering and compression algorithm with pre-filtering by Video Block-Matching and 3D Collaborative Filter (VBM3D) and compression by the H.264/AVC standard, and third is a rate-distortion oriented joint video in-loop filtering and compression algorithm. Practical results show that the proposed real-time filter achieves good de-noising performance in terms of both Peak-to-noise ratio and subjective visual quality, and the proposed joint de-noising and compression approaches also enhance the performance of the H.264/AVC standard by improving object visual quality and saving bitrates.

1. 1/30
Master Thesis Presentation:
Real-time Rate-distortion Oriented Joint
Video Denoising and Compression
Junsheng Fu
Signal Processing Department
Tampere University of Technology
03/12/2011

2. 2/30
Outline
1. A real-time filter
2. Joint video pre-filtering and compression
3. Joint video in-loop filtering and compression
03/12/2011

3. 3/30
Part 1:
A real-time filter
03/12/2011

4. 4/30
VBM3D (Video Block-Matching 3D Filtering)
K. Dabov, A. Foi, and K. Egiazarian, “Video denoising by sparse 3D transform-domain collaborative filtering,” in Proc. 15th
European Signal Processing Conference, EUSIPCO, Poznan, Poland, September 2007
03/12/2011

5. 5/30
Filter performance
Res: 640x480 Standard
Sigma:5 VBM3D
Noisy (dB) 34.15
Vassar Denoised (dB) 40.81
Speed (fps) 1.12
Noisy (dB) 34.15
Computer Platform
ballroom Denoised (dB) 41.55 Processor: Intel Core 2 Duo 3GHz
Speed (fps) 1.14 Memory: 3.2Gb of RAM
Video sequences vassar and ballroom are from http://www.merl.com/pub/avetro/mvc-testseq/orig-yuv/.
03/12/2011

6. 6/30
Optimization Task
1. Accelerate filter speed: above 25 fps for video (640 x 480)
In a computer
• Processor: Intel Core 2 Duo 3GHz
• Memory: 3.2Gb of RAM
Approaches
1. Simplify VBM3D filter by only using most influential parts for
noise attenuation.
2. Propose a real-time integer implementation of simplified
VBM3D.
03/12/2011

7. 7/30
Approach 1:
What to remove to Simplify VBM3D?
Thresholding filtering Wiener filtering
Temporal Max
Filters searching grouping
Spatial Spatial Temporal Thres- Temporal Spatial Spatial Temporal Wiener Temporal frames blocks
search transform search holding transform search transform search filtering transform
Standard
VBM3D + + + + + + + + + + 9 8
*Simplified
VBM3D - - + + + - - - - - 5 4
* Simplified VBM3D means turn off some features of VBM3D filter.
03/12/2011

8. 8/30
Filter performance
Res: 640x480 Standard Simplified
Sigma:5 VBM3D VBM3D
Noisy (dB) 34.15
Vassar Denoised (dB) 40.81 38.61
Speed (fps) 1.12 7.62
Noisy (dB) 34.15
ballroom Denoised (dB) 41.55 37.58
Speed (fps) 1.14 7.61
Video sequences vassar and ballroom are from http://www.merl.com/pub/avetro/mvc-testseq/orig-yuv/.
03/12/2011

9. 9/30
Approach 2:
Proposed real-time implementation
Filter version Proposed implementation VBM3D standard version
Data type Integer float
Memory buffer only 4 frames buffer whole video
Block matching Modified diamond search Full search
Temporal searching
4 5
frames
03/12/2011

10. 10/30
Practical Results
Res: 640x480 Standard Simplified Proposed
Sigma: 5 VBM3D VBM3D Implementation
Noisy (dB) 34.15
Vassar Denoised (dB) 40.81 38.61 38.49
Speed (fps) 1.12 7.62 30.49
Noisy (dB) 34.15 34.15 34.15
ballroom Denoised (dB) 41.55 37.58 37.43
Speed (fps) 1.14 7.61 29.59
Video sequences vassar and ballroom are from http://www.merl.com/pub/avetro/mvc-testseq/orig-yuv/.
03/12/2011

11. 11/30
Part 1 summary: A real-time filter
1. 30 time faster than standard VBM3D
2. Ready to be used for real-time application.
3. Good video denoising performance in a low noise level.
03/12/2011

12. 12/30
Part 2:
Joint video pre-filtering and compression
03/12/2011

13. 13/30
Typical scheme of Pre-filtering and
compression
R max
Filter control Rate control
F* Q*
x(t) Pre-filtering y(t) Video Encoder z(t)
Usually we have two separate processes for pre-filtering and compression, and we
choose parameters separately.
03/12/2011

14. 14/30
Optimization Task
R max
Joint Rate
control
F* Q*
x(t) Pre-filtering y(t) Video Encoder z(t)
(1)
03/12/2011

15. 15/30
Practical Results
Pre-filtering: VBM3D
Video Encoder: H.264/AVC (JM codec)
Experiments modes:
• Constant quantization mode
• Constant bitrates mode
03/12/2011

16. 16/30
Practical Results
Constant Quantization Mode
In compression part, quantization parameters {Q} include:
• QPI ={21,22,…45}, respectively QPP = QPI + 5
• Other fixed codec setting
In filtering part, filtering parameters {F} include:
• Sigma ={0,0.5,1,…5},
• Other fixed filter setting
Full search:
• For each Q, find the best output under rate-distortion
framework.
03/12/2011

17. 17/30
Practical Results
hall.yuv, 25 points, QPI=21:1:45, QPP=26:1:50
38
36
34
Y-PSNR
32
30
28
26
H.264/AVC compression
joint VBM3D prefiltering and H.264/AVC compression
0 50 100 150 200 250 300 350 400 450
Bitrate (kbit/s)
PSNR gains up to 0.5 dB.
Bitrates savings up to 13.4%.
03/12/2011

18. 18/30
Practical Results
Constant Bitrates Mode
In compression part:
• Enable the constant bitrates control.
In filtering part, filtering parameters {F} include:
• Sigma ={0,0.5,1,…5},
• Other fixed filter setting
Full search:
• For each Q, find the best output under rate-distortion
framework.
03/12/2011

19. 19/30
Practical Results
hall.yuv, Constant bitrate=215 kbit/s
38.5
H.264/AVC compression
joint VBM3D prefiltering and H.264/AVC compression
38
37.5
37
Y-PSNR
36.5
36
35.5
35
0 50 100 150 200 250 300
frames
PSNR gains up to 1.2 dB .
03/12/2011

20. 20/30
H.264/AVC, frame 23 VBM3D+H.264/AVC, frame 23
03/12/2011

21. 21/30
Part 2 summary:
Joint video pre-filtering and compression
1. Output video frames have less compression artifacts.
2. Output videos have consistent PSNR gains.
• can be up to 1.2 dB under constant bitrates mode.
• can be up to 0.5 dB under constant quantization mode.
3. Can save the bitrates up to 13.4% in comparison with only
compression.
03/12/2011

22. 22/30
Part 3:
Joint video in-loop filtering and
compression
03/12/2011

23. 23/30
Typical scheme of in-loop filtering and
compression
03/12/2011

24. 24/30
Optimization Task
03/12/2011

25. 25/30
Practical Results
Enhanced in-loop filtering: real-time filter from Part 1.
Video Encoder: H.264/AVC (JM codec)
Experiments modes:
• Inter mode
• QPI ∈ {21, 22 . . . 45} for I frame, and respective QPP =
QPI + 5 for P frames.
• Intra mode
• QPI ∈ {21, 22 . . . 45} for I frame.
03/12/2011

26. 26/30
Practical Results
hall.yuv, 24 points, QPI=21:1:45, QPP=26:1:50
42
40
38
36
Y-PSNR
34
32
30
28 H.264/AVC(intra)
H.264/AVC(intra) + Enhancing filter
500 1000 1500 2000 2500 3000 3500
Bitrate (kbit/s)
PSNR gains up to 0.87 dB.
Bitrates savings up to 10.5%.
03/12/2011

27. 27/30
Practical Results
hall.yuv, 24 points, QPI=21:1:45, QPP=26:1:50
38
36
34
Y-PSNR
32
30
28
26 H.264/AVC(inter)
H.264/AVC(inter) + Enhancing filter
200 250 300 350 400 450 500 550
Bitrate (kbit/s)
PSNR gains up to 0.35 dB.
Bitrates savings up to 6.3%.
03/12/2011

28. 28/30
Part 3 summary :
Joint video in-loop filtering and compression
1. This joint approach consistently improves the compression
performance of H.264/AVC under intra mode, but it gains little
under inter mode.
2. Under intra mode
• PSNR gains up to 0.87 dB.
• Bitrates saving up to 10.5% in comparison with only compression.
3. Under inter mode
• PSNR gains up to 0.35 dB.
• Bitrates saving up to 6.3% in comparison with only compression.
03/12/2011

29. 29/30
Conclusion
1. We propose
• a real-time video filter which has good de-noising performance.
• a joint pre-filtering and compression approach
• a joint in-loop de-noising and compression approach .
2. Results show that these two joint approaches enhance the
performance of the H.264/AVC standard:
• less compression artifact
• increased PSNR
• saved the bitrates.
03/12/2011

30. 30/30
Thank you!
03/12/2011