1. Defocus Techniques for
Camera Dynamic Range
Expansion
Matthew Trentacoste, Cheryl Lau, Mushfiqur Rouf,
Rafal Mantiuk, Wolfgang Heidrich
University of British Columbia

2. Defocus DR expansion
• Sensorsexpanded, dynamic rangeexist
Can be
limited in
but tradeoffs
• Evaluate the scene incident onthe dynamic
range of
the opposite, reduce
the sensor
by optical blurring, restore in software
1/9 1/9 1/9 5/9 5/9 5/9
5 1/9 1/9 1/9
= 5/9 5/9 5/9
1/9 1/9 1/9 5/9 5/9 5/9

3. Approach
• Use 2 techniques to aid:
coded aperture + deconvolution
• Aperture filtermore information
PSF preserves
to improve deconvolution quality
[Rashkar 2006][Levin 2007][Veeraraghavan 2007]
• Deconvolution tousing natural image statistics
Recent advances
restore original image
[Bando 2007][Levin 2007]

4. Physical setup
• Rays from focused onto sensoraperture
plane and
scene pass through
• Cone of rays fromforming the shape of the
intersects sensor,
out-of-focus points
aperture
• A patternsensor aperture plane ispoints
onto the
in the
for out-of-focus
projected

5. Coded Aperture
• Originally from x-ray 1989]
[Fenimore 1978][Gottesman
astronomy
• Structured of pinhole, but better SNR with
resolution
arrays + decoding algorithm
• Employed in visible light photography
[Rashkar 2006][Levin 2007][Veeraraghavan 2007]
• Improve frequency properties of filter

6. Aperture filters
• What makes a good filter?
• Frequency response
• Position and spacing of zero frequencies
• Diffraction / transmission

7. Deconvolution
• Restore image distorted by PSF
[Wiener 1964][Richardson 1972][Lucy 1974]
f = f0 ⊗ k + η
• Ill-posed, infinite solutions
• No exact solution due to noise
• Division in FFT, issues with small
values in OTF of filter

8. Deconvolution
• Current state-of-the-art methods rely on natural
image statistics
• Real-worlddistribution of several properties:
Heavy-tail
images share
gradients
• Prior 2007][Levindeconvolution algorithms
[Bando
term in
2007]
• Favors interpretations fewthe image with all the
gradient intensity at a
of
pixels

9. Evaluation
• Goal : determine whether any combo of filterDR
deconvolution yields meaningful reduction in
/
with acceptable final image quality
• Measure DR reduction both in terms of image
local contrast and filter
• Measure image quality as images between
deconvolved and original
difference

10. Source material
Atrium Morning Atrium Night
Figure 3.3: Sample images used in evaluation.
Radius Atrium Morning Atrium Night
min max reduction min max reduction
Original 0.00 11.0 0.00 12.0
1 0.00 10.8 0.200 0.452 12.0 0.452
2 0.00 10.6 0.424 0.622 12.0 0.622
3 0.00 10.3 0.716 1.163 11.8 1.34
4 0.02 10.0 1.00 1.436 11.4 1.99
5 0.08 9.94 1.14 1.589 11.4 2.23
6 0.15 9.92 1.24 1.731 11.2 2.51
8 0.31 9.83 1.48 1.890 10.8 3.13
9 0.40 9.79 1.61 1.950 10.5 3.41
11 0.66 9.71 1.94 2.08 10.3 3.74
13 0.86 9.67 2.19 2.18 10.1 4.13
16 1.04 9.59 2.45 2.26 9.61 4.65
Figure 3.4: Amount of reduction in dynamic range as a function of radius of a standard aperture (disk)
filter in pixels. All units are in terms of powers of two, referred to as exposure value (EV) stops.
Atrium Morning Atrium Night

11. Source material
Atrium Morning Atrium Night
Figure 3.3: Sample images used in evaluation.
Radius Atrium Morning Atrium Night
min max reduction min max reduction
Original 0.00 11.0 0.00 12.0
1 0.00 10.8 0.200 0.452 12.0 0.452
2 0.00 10.6 0.424 0.622 12.0 0.622
3 0.00 10.3 0.716 1.163 11.8 1.34
4 0.02 10.0 1.00 1.436 11.4 1.99
5 0.08 9.94 1.14 1.589 11.4 2.23
6 0.15 9.92 1.24 1.731 11.2 2.51
8 0.31 9.83 1.48 1.890 10.8 3.13
9 0.40 9.79 1.61 1.950 10.5 3.41
11 0.66 9.71 1.94 2.08 10.3 3.74
13 0.86 9.67 2.19 2.18 10.1 4.13
16 1.04 9.59 2.45 2.26 9.61 4.65
2.45 EV
Figure 3.4: Amount of reduction in dynamic range as a function of radius of a standard aperture (disk)
filter in pixels. All units are in terms of powers of two, referred to as exposure value (EV) stops.
Atrium Morning Atrium Night

12. Source material
Atrium Morning Atrium Night
Figure 3.3: Sample images used in evaluation.
Radius Atrium Morning Atrium Night
min max reduction min max reduction
Original 0.00 11.0 0.00 12.0
1 0.00 10.8 0.200 0.452 12.0 0.452
2 0.00 10.6 0.424 0.622 12.0 0.622
3 0.00 10.3 0.716 1.163 11.8 1.34
4 0.02 10.0 1.00 1.436 11.4 1.99
5 0.08 9.94 1.14 1.589 11.4 2.23
6 0.15 9.92 1.24 1.731 11.2 2.51
8 0.31 9.83 1.48 1.890 10.8 3.13
9 0.40 9.79 1.61 1.950 10.5 3.41
11 0.66 9.71 1.94 2.08 10.3 3.74
13 0.86 9.67 2.19 2.18 10.1 4.13
16 1.04 9.59 2.45 2.26 9.61 4.65
2.45 EV 4.56 EV
Figure 3.4: Amount of reduction in dynamic range as a function of radius of a standard aperture (disk)
filter in pixels. All units are in terms of powers of two, referred to as exposure value (EV) stops.
Atrium Morning Atrium Night

13. Tests
• Filters evaluated: • Deconvolution evaluated:
• Normal aperture • Wiener filtering
• Gaussian • Richardson-Lucy
• Veeraraghavan • Bando
• Levin • Levin
• Zhou

14. Evaluation (cont)
• Success criteria:
• Reduction of computational cost of deconv
to justify the
at least 2 stops
• Quality of at least PSNR 35

15. Images
Weiner Richardson-Lucy Bando Levin
filter=Zhou, noise = 0, radius = 1

16. Images
Weiner Richardson-Lucy Bando Levin
filter=Zhou, noise = 0, radius = 5

17. Images
Weiner Richardson-Lucy Bando Levin
filter=Zhou, noise = 0, radius = 16

18. Deconv: no noise
orning deconvolution
Atrium Morning deconvolution
Atrium Morning deconvolution
Atrium Night deconvolution
60 60 60
Weiner
Weiner Richardson−Lucy Weiner Weiner
Richardson−Lucy
55 55
Richardson−Lucy
Bando
Levin
Richardson−Lucy
Bando
Levin
50
55 Bando 50 Bando
Levin Levin
45 45
50
40 40
PSNR (dB)
PSNR (dB)
PSNR
PSNR
35 35
45
30 30
25 25
40
20 20
PSNR (dB)
15 15
35
10 10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Dynamic range reduction (EV stops) Dynamic range reduction (EV stops)
30 DR reduction DR reduction
25

19. Aperture: no noise
Morning aperture filter filter Atrium Morning aperture filter filter
Atrium Morning aperture Atrium Night aperture
60 60 60
Standard Aperture
Standard Aperture
Gaussian
Standard Aperture
Standard Aperture
Gaussian
55
Gaussian Veeraraghavan
Zhou
55
Gaussian
Veeraraghavan
Zhou
55 Veeraraghavan
Levin Veeraraghavan
Levin
50 50
Zhou Zhou
45 Levin 45 Levin
50
40 40
PSNR (dB)
PSNR (dB)
PSNR
PSNR
35 35
45
30 30
25 25
40
PSNR (dB)
20 20
15 35 15
10 10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Dynamic range reduction (EV stops) Dynamic range reduction (EV stops)
30DR reduction DR reduction
25
20

20. Deconv: noise
orning deconvolution
Atrium Morning deconvolution
Atrium Morning deconvolution
Atrium Night deconvolution
60 60 60
Weiner
Weiner Richardson−Lucy Weiner Weiner
Richardson−Lucy
55 55
Richardson−Lucy
Bando
Levin
Richardson−Lucy
Bando
Levin
50
55 Bando 50 Bando
Levin Levin
45 45
50
40 40
PSNR (dB)
PSNR (dB)
PSNR
PSNR
35 35
45
30 30
25 25
40
20 20
PSNR (dB)
15 15
35
10 10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Dynamic range reduction (EV stops) Dynamic range reduction (EV stops)
30 DR reduction DR reduction
25

21. Aperture: noise
Morning aperture filter filter Atrium Morning aperture filter filter
Atrium Morning aperture Atrium Night aperture
60 60 60
Standard Aperture
Standard Aperture
Gaussian
Standard Aperture
Standard Aperture
Gaussian
55
Gaussian Veeraraghavan
Zhou
55
Gaussian
Veeraraghavan
Zhou
55 Veeraraghavan
Levin Veeraraghavan
Levin
50 50
Zhou Zhou
45 Levin 45 Levin
50
40 40
PSNR (dB)
PSNR (dB)
PSNR
PSNR
35 35
45
30 30
25 25
40
PSNR (dB)
20 20
15 35 15
10 10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Dynamic range reduction (EV stops) Dynamic range reduction (EV stops)
30DR reduction DR reduction
25
20

22. Conclusions
• Levin deconv at very low noise levels with
coded filters
the best, obtaining results
• No combination of filter and deconvolution
consistently produced acceptable results
• Efficiency of the approach is scene dependent
Most efficient for small, isolated bright regions