Light Field Camera Technology Allows Refocusing Photos After Capture

Light Field Camera
from scientiﬁc research to a $50-million business

SHI Weili, Tsinghua University
Photo from Lytro.com

With big aperture, DCLRs can achieve shallow focus.

Consumer DCs produce photos with deep focus.

What we’d prefer...
(post-processed with Adobe Photoshop)

But how about
loss of focus?...
(previously irreparable)

Introducing Lytro: refocus afterwards!

Photos from Lytro.com

Technological features

+ Refocusing
+ Extending depth of ﬁeld
+ 3D images
+ Speed
+ Low-light sensitivity
+ Easy sharing
- Low resolution

"Our vision is a product
that allows people to
shoot and share very simply."
Ren Ng, founder and CEO of Lytro

Photo and quote from dpreview.com

Industrial design


Interaction design


Interaction design

Image from NewDealDesign

"We worked really hard to
create an iconic design that
really conveys the idea that
this is ‘camera 3.0’."


Depth of ﬁeld: source of blurriness

Figure from Wikipedia

Depth of ﬁeld: source of blurriness

Some information is missing:
If we know the direction of each ray of light,
we can trace the rays back to their source,
and don’t have to mix them up.

4D light ﬁeld

Figures from Ng, R. 2005. Fourier slice photography.

 2D light field study with a linear camerachapter . light fields and photographs

Figure .: Parameterization for the light field flowing into the camera.
parameterization for the light field flowing into the camera
the photosensor. Let us refer to u as the directional axis, because the u intercept on the lens
determines the direction at which the ray strikes the sensor. In addition, let us refer to x as
Figure from Ng, R. 2006. spatial axis. Of course in general the ray exists in d and we would consider intersections
the Digital light field photography.

2D light field study with a linear camera
.. photograph formation 

Figure .: The set of all rays flowing into the camera.
the set of all rays flowing into the camera
of rays in a sampled light field, and it has become very common in the light field literature.

Figure from Ng, R. 2006. Digital light field photography.
2.3 Photograph Formation

 2D light ﬁeld study with a linear camera
chapter . light fields and photographs

Figure .: The cone of rays summed to produce one pixel in a photograph.
the cone of rays summed to produce one pixel in a photograph
which arrive at the sensor from more oblique angles, contribute less energy to the value of
the pixel. Another example is that the photosensitive portion of a pixel in a cmos sensor is
Figure from Ng, R. 2006. Digital light ﬁeld an overlay of metal wires [Catrysse and Wandell ], so rays from
typically obscured by photography.

A photograph is an integral projection
of the canonical light ﬁeld.

Quote from Ng, R. 2006. Digital light ﬁeld photography.

Refocusing

Figure from Ng, R. 2006. Digital light ﬁeld photography. and Wikipedia
(a) (b)

A photograph is an integral projection
of the canonical light ﬁeld, where the
trajectory of the projection depends on
the depth at which the photograph is
focused.

Quote from Ng, R. 2006. Digital light ﬁeld photography.

Recording the light field
.. a plenoptic camera records the light field 

Figure .: Sampling of a photograph’s light field provided by a plenoptic camera.
sampling of a photograph’s light field provided by a plenoptic camera
camera the width of a grid column is the width of a photosensor pixel. In the plenoptic
camera, on the other hand, the grid cells are shorter and wider. The column width is the
Figure from Ng, R. 2006. Digital microlens, and the column is vertically divided into the number of pixels across
width of a light field photography.

 Recording the light ﬁeld
chapter . recording a photograph’s light field

(z) (z

Figure .: Raw light field photograph read off the photo
array. The figure shows a crop of approximately one qua
crolenses are clearly visible in print.

(z) (z)

Figure .: Raw light field photograph read off the photosensor underneath th
Figure from Ng, R. 2006. Digital light ﬁeld photography. a crop of approximately one quarter the full image so
array. The figure shows

Recording the light ﬁeld

(z) (z

Figure .: Raw light field photograph read off the photo
array. The figure shows a crop of approximately one qua
crolenses are clearly visible in print.

(z) (z)
Integrating on each microlen images gives the conventional photography.
Figure .: Raw light field photograph read off the photosensor underneath th
Figure from Ng, R. 2006. Digital light ﬁeld photography. a crop of approximately one quarter the full image so
array. The figure shows

Refocusing
.. image synthesis algorithms 

Two sub-aperture photographs obtained from a light ﬁeld by
extracting the shown pixel under each microlens (depicted on left).
Note that the images are not the same, but represents different
viewpoints.

(a): No refocus (b): Refocus closer (c): Refocus further

Figure .: Shift-and-add refocus algorithm, illustrated with just two sub-aperture images
for didactic purposes.

. three views of the recorded light field 

Refocusing

 chapter . light fields and photographs

(b)

(b)

(a) (b)
(a) (b)

Figure .: Overview of processing the recorded light field. Figure .: The projection of the light field corresponding to focusing further and closer
than the chosen x parameterization plane for the light field.

.. image synthesis algorithms
Refocusing



Figure .: Shift-and-add refocus algorithm, illustratedillustratedsub-aperture images
Shift-and-add refocus algorithm, with just two with just
for didactic purposes.
two sub-aperture images for didactic purposes.
Figure from Ng,with bothDigital light ﬁeld photography. from the center of the lens (u, v), and the relative
R. 2006. the distance of the sub-aperture


Refocusing
chapter . digital refocusing

(a) (a) (a)

(a) (a) (b)

Examples of refocusing (a1–a5) and extended depth of ﬁeld (b).
Figure .: Examples of refocusing (a–a) and extended depth of field (b).

Extending the depth of ﬁeld
.. image synthesis algorithms 

The sub-aperture photographs themselves have inﬁnite DOF,
resulted from the minor aperture.

Figure .: Shift-and-add refocus algorithm, illustrated with just two sub-aperture images


Extending thedigital refocusing ﬁeld
chapter .
depth of

(a): Unrefocused (b): Sub-aperture image (c): Extended dof

Refocusing each pixel gives extended an image with much digitally ex-
Figure .: Comparison of a sub-aperture image and DOF computed with higher SNR.
tended depth of field.

Figure from Ng, R. 2006. integration of Equation ., we obtain high snr by combining the contributions
numerical Digital light ﬁeld photography.

In the spatial domain, photographs are
integral projections of the light ﬁeld.

In the Fourier domain, photographs are
just 2D slice in the 4D light ﬁeld.

Quote from Ng, R. 2005. Fourier slice photography.

Classical Fourier Slice Theorem projection
Fourier slice vs. integral

Integral
Projection

2D Fourier 1D Fourier
Transform Transform

Slicing

Page from Ng, R. 2006. Digital light ﬁeld photography.

Fourier slice vs. integral projection

Integral
Projection

4D Fourier
Transform Spatial Domain
Inverse
Fourier Domain 2D Fourier
Transform

Slicing


Fourier slice vs. integral projection


In the Fourier domain, photographs are
just 2D slice in the 4D light ﬁeld.

That’s much simpler than in the spatial
domain, where photographs are integral
projections of the light ﬁeld.


Fourier-domain slicing algorithm
Pre-process: O(N4 log N)
Refocusing: O(N2 log N)

Spatial-domain integration algorithm
Refocusing: O(N 4)


Resolution of the light-field sample
.. a plenoptic camera records the light field 

Figure .: Sampling of a photograph’s light field provided by a plenoptic camera.
sampling of a photograph’s light field provided by a plenoptic camera
camera the width of a grid column is the width of a photosensor pixel. In the plenoptic
camera, on the other hand, the grid cells are shorter and wider. The column width is the
Figure from Ng, R. 2006. Digital microlens, and the column is vertically divided into the number of pixels across
width of a light field photography.

Band-Limited Analysis Band-limited analysis

Band-width of
measured light field

Light field shot
with camera


Band-Limited Analysis Band-limited analysis


Ren Ng improved the theory and technology of light ﬁeld camera
during his doctoral program.

"It was a scientiﬁc breakthrough
we were working towards.
The next step we've been working
on has been making a commercial
breakthrough."


Technological evolution:
from research to consumer product

early 2000s

rd Tech Report CTSR 2005-02
Stanford Tech Report CTSR 2005-02

he pinhole
g because
minate the
ssing rays
ays do not
ive image
eriphery. Figure 8: Top: Exploded view of assembly for attaching the microlens array
to the digital back. Bottom: Cross-section through assembled parts.
Figure 7: Technique for ameliorating vignetting. Top: Moving the pinhole
observer beyond the bounds shown in Figure 6 results in vignetting because
Our mi-
art 0125-
some required rays are unavailable (shaded gray). Bottom: To eliminate the
e, square we use the closest available rays, by clamping the missing rays
vignetting,
ctor. The
to the bounds of the aperture (shaded region). Note that these rays do not
number is
pass through the original pinhole, so the resulting multi-perspective image
and used
has a different center of projection for each ray in the corrected periphery. Figure 8: Top: Exploded view of assembly for attaching the microlens array
se lenses to the digital back. Bottom: Cross-section through assembled parts.
on tubes
4 image-
approximately 4000×4000 pixels that are 9 microns wide. Our mi-
ns holder, array was made by Adaptive Optics Associates (part 0125-
crolens mid 2000s
photosen-It has 296×296 lenslets that are 125 field camera in use.
0.5-S). Figure 9: Our light microns wide, square
Figures from Ng,with very close to 100% fill-factor. The
with three and square packed
shaped, R. 2006. Digital light field photography.
focal length of the microlenses is 500 microns, so their f -number is


2011: Lytro
Images from Lytro.com


2011: Lytro
Figure from Lytro.com

The company was founded in 2006.
It has raised approximately $50 million of venture capital.
Its ﬁrst camera went on sale October 19, 2011,
and began shipping on February 29, 2012,
starting with a very affordable price of $399.

"At first we'll be making those decisions for
the user - so that we can make the process as
simple as possible but, further down the line,
we'll provide tools to give more control over
the final output.
It's important to understand that Lytro's
camera will record full light fields at day one."



The Lytro Desktop application is required
to interact with the light field data format
(.lfp). The Lytro Desktop software comes
on the Lytro camera. The install window
will pop up the first time you plug in
your camera into your computer. You can
then start the install process. If it doesn’t,
find the disk image on your desktop to
start the install.

2. Unplug and re-plug in the camera.
After you install the software, you must plug the camera
back in to prompt an import of your first light field pictures.

3. Back up process begins.
A back up process will start after the first time the Lytro Desktop software runs.
This happens only the first time you plug your camera in and takes about 4-5 minutes.

Reminder: The minimum spec is Mac OS X 10.6.6 or higher.

Images from Lytro User Manual and Lytro.com

"We're very keen to see light
ﬁeld images develop through
an ecosystem of software."



Back to its limitation...

+ Refocusing
+ Extending depth of ﬁeld
+ 3D images
+ Speed
+ Low-light sensitivity
+ Easy sharing
- Low resolution Solution:
much higher density of
microlenses and sensors

"It's not technological limitations that
are deﬁning that ﬁgure, it's a
marketing-driven progression.
If you applied the technology being
developed for mobile phone cameras
and applied it to an APS-C sensor, you
could in theory make a sensor with
hundreds of millions of pixels."


Or think about Nokia’s 40-megapixel 808 PureView!
Pictures from Nokia

Revolution led by the crazy one

“Lytro is developing a new type of camera
that dramatically changes photography
for the ﬁrst time since the 1800s.”
–TechCrunch
Photo from NewDealDesign

A tribute to Apple?

Photo by Annie Tao

Light Field Camera Technology Allows Refocusing Photos After Capture

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Light Field Camera Technology Allows Refocusing Photos After Capture

Similar to Light Field Camera Technology Allows Refocusing Photos After Capture (20)

Recently uploaded

Recently uploaded (20)

Light Field Camera Technology Allows Refocusing Photos After Capture