Apollon - 22/5/12 - 09:00 - User-driven Open Innovation Ecosystems
200081003 Friday Food@IBBT
1. Advances and challenges in image and
video restoration
Aleksandra Pizurica
Ghent University
Image Processing and Interpretation Group
2. Median filter: Reduction of impulse noise
impulse noise median over 3x3
Median filter removes isolated noise peaks, without blurring the image
IBBT Friday food talk, October 3, 2008 Image and video restoration 2
3. Median filter and reduction of white noise
original median over 3x3
For not-isolated noise peaks (e.g., white Gaussian noise) median filter
is not very efficient.
IBBT Friday food talk, October 3, 2008 Image and video restoration 3
4. Why is denoising important
original denoised
Not only visual
enhancement, but
also: automatic
processing is
facilitated!
Example:
edge detection
IBBT Friday food talk, October 3, 2008 Image and video restoration 4
6. Overview
• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video
IBBT Friday food talk, October 3, 2008 Image and video restoration 6
7. Discrete Wavelet Transform (DWT)
DWT algorithm: a filter bank iterated on the lowpass output
wavelet coefficients
highpass
w j+1
g 2
w j+2
sj g 2
w j+3
h 2 g 2
lowpass s j+1 h 2
s j+1 h 2 s j+3
scaling coefficients
IBBT Friday food talk, October 3, 2008 Image and video restoration 7
9. Two dimensional DWT
APPROXIMATION
scaling coefficients
Wavelet coefficient values
DETAIL IMAGES
wavelet coefficients
0
Peaks indicate image edges
IBBT Friday food talk, October 3, 2008 Image and video restoration 9
10. Noise in the wavelet domain
Noise-free reference
Wavelet coefficient values
0
Peaks indicate image edges
IBBT Friday food talk, October 3, 2008 Image and video restoration 10
11. Marginal priors: Generalized Laplacian
Generalized Laplacian (generalized Gaussian) distribution
noise-free histogram
f(y)=Aexp(-|y /s|ν )
s: scale parameter
ν: shape parameter
(0 ≤ ν ≤ 1)
noisy Parameters accurately
estimated from a signal
corrupted by additive
white Gaussian noise
Often yields complicated expressions
Extension to higher dimensions (joint histograms) difficult
IBBT Friday food talk, October 3, 2008 Image and video restoration 11
12. Gaussian Scale Mixture (GSM) models
Efficient for modelling joint histograms
wavelet coefficient Gaussian of the neighboring wavelet coefficients
random variable
f(y) y= zu
f(u)
y u
z: mixture variable, random multiplier
A state-of the art denoiser for many years BLS-GSM: Bayesian Least
Squares estimator using GSM prior [Portilla et al, IEEE TIP’03]
∞
x = y + n = zu + n E ( y | x ) = ∫−∞ zE ( y | x, z ) f ( z )dz
zC u C u , Cn : signal and noise
noise E ( y | x, z ) = x
zC u + C n covariances
IBBT Friday food talk, October 3, 2008 Image and video restoration 12
13. Locally adaptive denoising ProbShrink
[Pizurica&Philips, IEEE TIP 2006]
LSAI – Local Spatial Activity Indicator
ESTIMATE
LSAI H1 signal of interest present
yl
OBSERVATION H0 signal of interest absent
LSAI zl
ηξµ
y = β + n, ˆ
β = P( H1 | y, z ) y = y
1 + ηξµ
f ( y | H1 ) P ( H1 )
η= f ( z | H1 ) µ=
f ( y | H0 ) ξ= P( H 0 )
f (z | H0 )
f(z|H0)
f(y|H0) P(H0)
f(y|H1)
f(z|H1) P(H1)
noisy coefficient y LSAI z subband statistics
IBBT Friday food talk, October 3, 2008 Image and video restoration 13
14. Locally adaptive denoising: ProbShrink…
[Pizurica&Philips, IEEE TIP 2006]
IBBT Friday food talk, October 3, 2008 Image and video restoration 14
15. ProbShrink for correlated noise…
local window
X 22
20
40
X
23
60
80
100
X22 X23
120
140
vector of coefficients
160
180
50 100 150
H0
H1
X23 H1
X22
X22
[B. Goosens, A. Pizurica, W. Philips; IEEE TIP 2008, in press]
IBBT Friday food talk, October 3, 2008 Image and video restoration 15
16. … ProbShrink for correlated noise
[B. Goosens, A. Pizurica, W. Philips; IEEE TIP 2008, in press]
IBBT Friday food talk, October 3, 2008 Image and video restoration 16
17. Denoising by singularity detection
Input signal Rate of increase of the
modulus of the wavelet
transform across scales is
w1 wavelet coefficients proportional to the
local Lipschitz regularity
w2
w3
[Mallat&Zhong, IEEE IT 1992]
IBBT Friday food talk, October 3, 2008 Image and video restoration 17
18. Statistics: magnitude and the rate of increase
Noise standard deviation=25.5
0.05 150
0.04
noise 100
edges
x=1 l
Magnitude
0.03
0.02
50
0.01 edges noise
x= 0
l
0
-50 0 50 100 150 200 250 300
0
noisy magnitude -3 -2 -1 0 1 2 3
ACR
1
0.8
scale
0.6
noise edges
0.4 Average Cone Ratio – an estimate
0.2 of the local Lipschitz exponent –
0
measures the rate of increase of
-4 -2 0 2 4 6
the coefficients across the scales
cone of influence ACR
[A. Pizurica et al; IEEE TIP 2002]
IBBT Friday food talk, October 3, 2008 Image and video restoration 18
19. Inter- and intrascale dependencies
• Bivariate models
• Hidden Markov Tree models
• Markov Random Field models
IBBT Friday food talk, October 3, 2008 Image and video restoration 19
20. Statistical modeling: MRF models
x0 xMAP neighborhood cliques
P(x)
Prior model
1
P(x) = exp− ∑VC (x) Example: penalize isolated peaks
Z C∈ς
negative potential
clique
potentials positive potential
IBBT Friday food talk, October 3, 2008 Image and video restoration 20
21. Statistical modeling: MRF models
x0 xMAP neighborhood cliques
P(x)
Prior model
Initial edges Iteration 1 Iteration 2 Iteration 3
IBBT Friday food talk, October 3, 2008 Image and video restoration 21
22. MRF based wavelet denoising
Original
Gamma MAP filter wavelet filter
[A. Pizurica et al; ICIP 2001]
IBBT Friday food talk, October 3, 2008 Image and video restoration 22
23. Current trends in Bayesian wavelet denoising
Gaussian Scale Mixture (GSM) model MRF models
[Portilla et al, IEEE TIP 2003]
Shortcoming: assumes the same but scaled
covariance for the whole subband Fields of GSM
[Liu and Simoncelli,
Mixture of GSM PAMI’08]
(MGSM)
Spatially variant GSM
[Portilla et al, Spie
(SVGSM) 2008] Field of Experts (FoE)
[Guerrero-Colon et al,
Computationally expensive [Roth and Black, CVPR’05]
IEEE TIP,08]
GSM in non-overlapping blocks [Tappen, Adelson, Freeman,
• Ignores non-local correlations CVPR’07, CVPR’08]
• Block size?
Mixture of projected GSM
Dimension reduction
(MPGSM) ?
in MGSM [Goossens, in review TIP 2008]
IBBT Friday food talk, October 3, 2008 Image and video restoration 23
24. Overview
• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video
IBBT Friday food talk, October 3, 2008 Image and video restoration 24
25. Why other multiresolution representations
Classical wavelets are well suited for point-like singularities,
but not for curvilinear singularities in images
• Poor orientation selectivity; no difference between 45 and -45o
• Checkerboard pattern appears also as an artifact in denoising
An example of wavelet base functions
Many wavelet-like representations with a better orientation selectivity:
complex wavelets [Kingsbury, Selesnick] , steerable pyramids [Freeman,
Adelson], curvelets [Donoho, Candes], contourlets [Do, Vetterli], …
IBBT Friday food talk, October 3, 2008 Image and video restoration 25
26. Curvelet-domain image denoising…
Curvelets: specific tiling of the frequency plane:
localized + directional
IBBT Friday food talk, October 3, 2008 Image and video restoration 26
27. …Curvelet domain image denoising…
Wavelet ProbShrink
Noisy Image Curvelet Hard Thresholding
Curvelet ProbShrink
PSNR=29.50dB
PSNR=22.16dB PSNR=29.02dB
PSNR=30.43dB
[L. Tessens, A. Pizurica, W. Philips, J Electr Imag 2008 in press]
IBBT Friday food talk, October 3, 2008 Image and video restoration 27
28. Results
…Curvelet domain image denoising…
Wavelet ProbShrink Curvelet ProbShrink
IBBT Friday food talk, October 3, 2008 Image and video restoration 28
29. …Curvelet domain image denoising
Wavelet ProbShrink Curvelet ProbShrink
IBBT Friday food talk, October 3, 2008 Image and video restoration 29
30. Overview
• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video
IBBT Friday food talk, October 3, 2008 Image and video restoration 30
31. MRI denoising: signal dependent noise
p(m) low SNR noisy m
(f=0)
magnitude
contrast
high SNR
( f =f1 )
noise-free f
f1 m SNR
IBBT Friday food talk, October 3, 2008 Image and video restoration 31
32. MRI denosing: algorithm
Step 1: Bias removal
Square magnitude MRI image – after squaring constant bias, proportional
to noise standard deviation.
For better results: square root the result before denoising!
Step 2: Denoising (coarse-to-fine, empirical density estimation)
Mask
A noisy detail
T?
p(z|H1)
log( )
p(z|H0) histograms p(z|H0)
Coarser, processed detail p(z|H1)
[Pizurica et al IEEE TMI 2003]
IBBT Friday food talk, October 3, 2008 Image and video restoration 32
33. MRI denosing: some results
Noisy image Denoised image Ground truth
IBBT Friday food talk, October 3, 2008 Image and video restoration 33
34. 3D MRI volume denoising
using 3D dual-tree complex wavelet transform
[J. Aelterman et al, EUSIPCO 2008]
IBBT Friday food talk, October 3, 2008 Image and video restoration 34
35. Denoising low-dose CT images
• Reducing radiation dose increases noise level
• Can we use denoising on low dose CT to obtain the same diagnostic
quality as in a higher dose CT image? [IBBT Ica4dt project]
• Difficulties:
- non-stationary correlated noise
- Streak artefacts
- How to estimate noise
IBBT Friday food talk, October 3, 2008 Image and video restoration 35
36. Denoising algorithm
segmentation
wavelet Inverse wavelet
Vector
transform transform
ProbShrink
(WT) (IWT)
(Dual-tree H0
complex) H1 H1
[B. Goossens et al, EMBS 2007]
IBBT Friday food talk, October 3, 2008 Image and video restoration 36
37. …Results
Watershed segmentation
denoised
IBBT Friday food talk, October 3, 2008 Image and video restoration 37
39. Optical Coherence Tomography (OCT) images
[IBBT Ica4dt project, with AGFA Healthcare]
3D OCT data 2D signal
OCT – “echography with light”
Noise: speckle similar to that in radar and ultrasound images
IBBT Friday food talk, October 3, 2008 Image and video restoration 39
40. Denoising OCT images
A developed 3D OCT denoiser combines [IBBT Ica4dt project]
• wavelet domain speckle filter
• motion compensated video denoising method
coarse-to-fine
processing 3D OCT data
Signal and noise statistics
Image:g120406breastseconformolnogelLR 050 Detail:Dx1 Parameter:b=10.3752
0 Image:g120406breastseconformolnogelLR 050 Detail:Dx2 Parameter:a=4.8348
0
0.035 0.2
Gamma, b=10.3752 Laplace, a=4.8348
0.03
0.025 0.15
likelihood p(m|1)
likelihood p(m|0)
0.02
0.1
0.015
0.01
0.05
0.005
0
0 50 100 150
magnitude m
200 250
0
0 50
magnitude m
100 150
Video denoising
Locally adaptive denoising
IBBT Friday food talk, October 3, 2008 Image and video restoration 40
41. Results and evaluation of OCT denoising
Noisy
Image Our method GTF
SAT
RKT
Lee
IBBT Friday food talk, October 3, 2008 Image and video restoration 41
42. Results and evaluation of OCT denoising
Noisy Our method Lee
SAT
[Pizurica et al; CMIR 2008 in press]
IBBT Friday food talk, October 3, 2008 Image and video restoration 42
43. Results and evaluation of OCT denoising
BLS-GSM
input Our method (2D version)
[Pizurica et al; CMIR 2008 in press]
IBBT Friday food talk, October 3, 2008 Image and video restoration 43
44. Overview
• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video
IBBT Friday food talk, October 3, 2008 Image and video restoration 44
45. Noise variance estimation
Block-based Smoothing based
Search for blocks of
nearly uniform intensity
smooth
estimate
σ
Gradient distribution based Wavelet based
noise (Rayleigh distr.)
HL1
signal+noise
use HH1
LH1
this part
σ Median{|HH1|}
or compensate for σ=
ˆ 0.6745
the peak shift
IBBT Friday food talk, October 3, 2008 Image and video restoration 45
46. Blur estimation using wavelet coefficients
• A well known approach: kurtosis of the wavelet coefficient histogram
• An alternative: examine the propagation of the wavelet coefficients across
the scales
0.35
0.3 original
blurred
0.25
0.2
PDF
0.15
0.1
0.05
0
-4 -2 0 2 4 6 8
Original image Blurred image ACR 1-2
ACR - Average Cone Ratio – an estimate of the local Lipschitz exponent –
measures the rate of increase of the coefficients across the scales
IBBT Friday food talk, October 3, 2008 Image and video restoration 46
48. Overview
• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video
IBBT Friday food talk, October 3, 2008 Image and video restoration 48
49. Video denoising
Input Noisy 2D Wavelet Noise
Frame Transform Estimation
Motion Time delay
Estimation
Recursive Temporal Filtering
Time delay
Adaptive Spatial Inverse 2D Wavelet Denoised
Filtering Transform Frame
[V. Zlokolica, A. Pizurica, W. Philips; IEEE TCSVT 2006]
IBBT Friday food talk, October 3, 2008 Image and video restoration 49
50. Video denoising: motion estimation…
center of
the motion block
motion direction
(smaller amplitude)
motion direction
(larger amplitude)
Accurate motion estimation is essential for video denoising.
Also important: reliability of the estimated motion at each point
IBBT Friday food talk, October 3, 2008 Image and video restoration 50
51. …Motion compensated video denoising
Further development currently within IBBT project ISYSS
[V. Zlokolica, A. Pizurica, W. Philips; IEEE TCSVT 2006]
IBBT Friday food talk, October 3, 2008 Image and video restoration 51
52. Reusing motion estimator from video codecs
• Motion estimators from video codecs tolerate errors cannot be directly
used in denoising
• Can we still use them with some postprocessing? The core of our approach:
• Motion field refinement step
• Reliability to motion estimates controls the recursive filter
• Competitive with state-of-the art video denoisers
[LJ. Jovanov et al; IEEE TCSVT 2008, in press]
IBBT Friday food talk, October 3, 2008 Image and video restoration 52
53. Reusing motion estimator from video codecs
noise-free input
[Balster; TCSVT 2006] [Jovanov; TCSVT 2008]
IBBT Friday food talk, October 3, 2008 Image and video restoration 53
54. Denoising and outlier removal in 3D video
Time-of-flight camera
records simultaneously
luminance and depth information
Degradations in the depth image:
noise, and outliers (similar to
impulse noise but in bursts)
3D reconstructions using
“surf” in Matlab
The biggest errors in the
depth measurement are
induced by strong
ambient light
The measured
distance is much smaller
than the true distance)
IBBT Friday food talk, October 3, 2008 Image and video restoration 54
55. Noisy 3D video sequence (luminance and depth)
Luminance image contains much less noise
Luminance and depth images are correlated
Use the luminance information for denoising depth data
IBBT Friday food talk, October 3, 2008 Image and video restoration 55
56. Denoised luminance and depth
IBBT Friday food talk, October 3, 2008 Image and video restoration 56
57. Acknowledgements
Thanks to my colleagues for their contributions
• Vladimir Zlokolica (video denoising)
• Bart Goossens (removal of correlated noise)
• Ljubomir Jovanov (video, 3D video, OCT)
• Linda Tessens (curvelets)
• Jan Aelterman (MRI denoising)
• Filip Rooms (deblurring)
• Ewout Vansteenkiste (quality evaluation CT)
Related material available at: http://telin.ugent.be/~sanja
IBBT Friday food talk, October 3, 2008 Image and video restoration 57