Ramesh Raskar presents his vision for computational photography and discusses several open research problems. He envisions cameras that provide ultimate post-capture control over focus and motion blur. He also discusses capturing the essence of a scene through techniques like multi-flash photography. Raskar outlines a wish list that includes understanding the world through analyzing photos, sharing visual experiences while preserving privacy, and looking around corners using techniques like transient imaging. The talk promotes co-designing optics and digital processing to change the rules of photography.

1. Computational Light Transport and Computational Photography: Inverse problems Camera Culture Ramesh Raskar Ramesh Raskar http://raskar.info MIT Media Lab raskar@mit.edu

3. How to Invent? After X, what is neXt Full Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010 Ramesh Raskar, MIT Media Lab

4. Ramesh Raskar, http://raskar.info X+Y X neXt Xd X X++ X Full Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010

5. Simple Exercise .. Image Compression Save Bandwidth and storage What is neXt

6. Strategy #1: Xd Extend it to next (or some other) dimension ..

7. X = Idea you just heard Concept Patent New Product/Best project/invention award Product feature Design Art Algorithm

8. Ramesh Raskar, http://raskar.info X+Y X neXt Xd X X++ X Full Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010

9. Research .. http://raskar.info How to come up w ideas: Idea Hexagon How to write a paper How to give a talk Open research problems How to decide merit of a project How to attend a conference, brainstorm Facebook.com/ rRaskar Tips Get on Seminar/Talks mailing lists worldwide http://www.cs.virginia.edu/~robins/YouAndYourResearch.html Why do so few scientists make significant contributions and so many are forgotten in the long run? Highly recommended Hamming talk at Bell Labs

10. Is project worthwhile? Heilmeier's Questions http://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechism What What are you trying to do? Articulate your objectives using absolutely no jargon. Related work How is it done today, and what are the limits of current practice? Contribution What's new in your approach and why do you think it will be successful? Motivation Who cares? If you're successful, what difference will it make? Challenges What are the risks and the payoffs? How much will it cost? How long will it take? Evaluation What are the midterm and final "exams" to check for success? Raskar additions: Why now? (why not before, what’s new that makes possible) Why us? (wrong answers: I am smart, I can work harder than others)

11. Great Research: Strive for Five Before Five teams Be first, often let others do details Beyond Five years What no one is thinking about Within Five layers of ‘Human’ Impact Relevance Beyond Five minutes of description Deep, iterative, participatory Fusing Five+ Expertise Multi-disciplinary, proactive Ramesh Raskar, http://raskar.info

13. Tools for Visual Computing Shadow Refractive Reflective Fernald, Science [Sept 2006]

14. Computational Photography Camera Culture Ramesh Raskar

15. Traditional Photography Detector Lens Pixels Mimics Human Eye for a Single Snapshot: Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world Image Courtesy: Shree Nayar

16. Picture Computational Camera + Photography: Optics, Sensors and Computations GeneralizedSensor Generalized Optics Computations Ray Reconstruction 4D Ray Bender Upto 4D Ray Sampler Merged Views, Programmable focus and dynamic range, Closed-loop Controlled Illumination, Coded exposure/apertures

17. Computational Photography Novel Illumination Light Sources Modulators Computational Cameras Generalized Optics GeneralizedSensor Generalized Optics Processing 4D Incident Lighting 4D Ray Bender Ray Reconstruction Upto 4D Ray Sampler 4D Light Field Display Scene: 8D Ray Modulator Recreate 4D Lightfield

18. Computational Photography [Raskar and Tumblin] captures a machine-readable representation of our world to hyper-realistically synthesize the essence of our visual experience. Resources ICCP 2012, Seattle Apr 2012 Papers due Dec 2nd, 2011 http://wikipedia.org/computational_photography http://raskar.info/photo

19. Computational Photography Computational Photography aims to make progress on both axis Phototourism Comprehensive Essence Scene completion from photos Augmented Human Experience Looking Around Corners Priors Capture Human Stereo Vision Metadata Coded Depth fg/bg Non-visual Data, GPS Virtual Object Insertion Spectrum Decomposition problems 8D reflectance field Direct/Global LightFields Relighting Epsilon Angle, spectrum aware Camera Array HDR, FoV Focal stack Resolution Material editing from single photo Digital Motion Magnification Raw Low Level Mid Level HighLevel Hyper realism Synthesis/Analysis

20. Co-designing Optical and Digital Processing Computational Light Transport Optics Displays Sensors Computational Photography Photon Hacking Illumination Signal Processing Computer Vision Machine Learning Bit Hacking

21. Take home points Co-design of hw/sw Avoid computational or optical chauvinism in imaging (Camera flash/Kinect) Hardware cost going to zero, Parallel technology trends Computer vision not just mimicking human vision/perception Borrow ideas from other fields: astronomy, scientific imaging, audio, communications Photons not just Pixels Change the rules of the game Optics, Sensors, Illum, Priors, Sparsity, Transforms Meta-data, Internet collection, Crowdsourcing

22. Computational Photography Wish List: Open Research Problems Camera Culture Ramesh Raskar

23. Wish #1 Ultimate Post-capture Control Camera Culture Ramesh Raskar

24. Digital Refocusing using Light Field Camera 125μ square-sided microlenses [Ng et al 2005]

25. Motion Blur in Low Light

26. Traditional Blurred Photo Deblurred Image

27. Fluttered Shutter Camera Raskar, Agrawal, Tumblin Siggraph2006 Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

28. Preserves High Spatial Frequencies Fourier Transform Sharp Photo Blurred Photo PSF == Broadband Function Flutter Shutter: Shutter is OPEN and CLOSED

29. Coded Exposure Traditional Deblurred Image Deblurred Image Image of Static Object

30. Motion Blur in Low Light

31. Fast periodic phenomena Vocal folds flapping at 40.4 Hz Bottling line 4000 fps hi-speed camera 500 fps hi-speed camera

32. Compressive Sensing Single Pixel Camera image compressive image measurement matrix

33. Periodic signals -fP -2fP -4fP 3fP -3fP 0 fMax - fMax 2fP fP=1/P 4fP Periodic signal x(t) with period P t P = 16ms Periodic signal with period P and band-limited to fMax = 500 Hz. Fourier transform is non-zero only at multiples of fP=1/P ~ 63Hz.

34. High speed camera P = 16ms Ts = 1/(2 fMax) -fP -2fP -4fP -3fP 4fP 3fP 2fP 0 fMax - fMax fP=1/P Nyquist Sampling of x(t) Periodic signal has regularly spaced, sparse Fourier coefficients. Is it necessary to use a high-speed video camera? Why waste bandwidth?

35. Traditional Strobing Use low frame-rate camera and generate beat frequencies. P t Low exposure to avoid blurring. Low light throughput. Period known apriori. Strobing animation credit Wikipedia

37. On an average, light throughput is 50%Coded Strobing Photography. Reddy, D., Veeraraghavan, A., Raskar, R. IEEE PAMI 2011

38. Observation Model x at 2000fps y at 25fps

39. Signal Model x at 2000fps y at 25fps

40. Signal & Observation Model Ais M x N, M<<N x at 2000fps y at 25fps N / M = 2000 / 25 = 80

41. Recovery: Sparsity Very few non-zero elements y = A s Observed values Mixing matrix Structured Sparse Coefficients Basis Pursuit De-noising

42. Simulation on hi-speed toothbrush 25fps normal camera 25fps coded strobing camera Reconstructed frames 2000fps hi-speed camera ~100X speedup

43. Rotating mill tool Mill tool rotating at 50Hz Reconstructed Video at 2000fps Normal Video: 25fps Coded Strobing Video: 25fps Blur increases as rotational velocity increases rotating at 200Hz rotating at 150Hz rotating at 100Hz increasing blur

44. Compressive Sensing for Images .. A good idea? Single Pixel Camera image compressive image measurement matrix

45. Is Randomized Projection-based Captureapt for Natural Images ? Periodic Signals Progressive Projections Randomized Projections Compression Ratio [Pandharkar, Veeraraghavan, Raskar 2009]

46. Compact ProgrammableLights ?

48. Emulate studio light from compact flashCamera Culture Ramesh Raskar

51. Wish #3 Understand the World Camera Culture Ramesh Raskar

52. Convert single 2D photo into 3D ? Snavely, Seitz, Szeliski U of Washington/Microsoft: Photosynth

53. Exploit Community Photo Collections U of Washington/Microsoft: Photosynth

55. 3D Awareness

56. Interact with informationCamera Culture Ramesh Raskar

58. Privacy in public and authentication

59. Hyper-real Photo Frames

60. Print ‘material’ Camera Culture Ramesh Raskar

61. Wish #5 Capturing Essence Camera Culture Ramesh Raskar

62. What are the problems with ‘real’ photo in conveying information ? Why do we hire artists to draw what can be photographed ?

63. Shadows Clutter Many Colors Highlight Shape Edges Mark moving parts Basic colors

64. Depth Edges with MultiFlash Raskar, Tan, Feris, Jingyi Yu, Turk – ACM SIGGRAPH 2004

69. Depth Discontinuities Internal and externalShape boundaries, Occluding contour, Silhouettes

70. Depth Edges

71. Our Method Canny

72. Result Photo Canny Intensity Edge Detection Our Method

73. Questions What will a camera look like in 10,20 years? How will a billion networked and portable cameras change the social culture? How will online photo collections transform visual social computing? How will movie making/new reporting change?

74. Photos of tomorrow: computed not recorded http://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/

76. Emulate studio lights with compact flash

77. Focus and motion blur

78. New forms

79. Flat camera, large LCDs as cameras

80. Image destabilization for larger aperture

81. Understand the world

82. Real or fake

83. Place 2D photo into 3D

84. Look around corner

85. Bokode: long distance barcode

86. Sharing

87. Lifelogs auto summary

88. Privacy/Verification

89. 6D photoframes

90. Essence

91. New visual arts

92. Multi-flash camera

95. Every Photon has a Story

96. What isaround the corner ?

97. Can you look around the corner ?

98. Multi-path Analysis 2nd Bounce 1st Bounce 3rd Bounce

100. 2009: Marr PrizeHonorable Mention (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)

102. Multi-Dimensional Light Transport 5-D Transport Gigapan

103. Collision avoidance, robot navigation, …

104. z x S L R s Occluder Streak-camera 3rd bounce C Laser beam B Echoes of Light

105. Steady State 4D Impulse Response, 5D

106. Scene with Ultra fast illumination and camera hidden elements Raw 5D Capture Time profiles Signal Proc. Photo, geometry, reflectance beyond line of sight Novel light transport models and inference algorithms ® t 3D Time images Femto-PhotographyTime Resolved Multi-path Imaging

107. Team Moungi G. Bawendi, Professor, Dept of Chemistry, MITJames Davis, UC Santa CruzAndreas Velten, Postdoctoral Associate, MIT Media LabRohitPandharkar, RA, MIT Media Lab Otkrist Gupta, RA, MIT Media LabAndrew Matthew Bardagjy, RA, MIT Media Lab Nikhil Naik, RA, MIT Media LabTyler Hutchison, RA, MIT Media LabEverett Lawson, MIT Media Lab Ramesh Raskar, Asso. Prof., MIT Media Lab Camera Culture Ramesh Raskar

108. Photos from Streak Camera Capture Setup Hidden Scene

109. Photos from Streak Camera Capture Setup Hidden Scene Overlay Reconstruction

110. Motion beyond line of sight Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011

111. …, bronchoscopies, … Participating Media

112. Photo First Bounce Later Bounces + Direct Global [Nayar, Krishnan, Grossberg, Raskar 2006]

114. Each frame = ~2ps = 0.6 mm of Light Travel

115. Ripples of Waves

120. View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale

121. Two Layer Displays barrier lenslet sensor/display sensor/display PB = dim displays Lenslets = fixed spatial and angular resolution Dynamic Masks = Brighter, High spatial resolution

122. Limitations of 3D Display Parallaxbarrier LCD display Front Back Lanman, Hirsch, Kim, RaskarSiggraph Asia 2010

123. Light Field Analysis of Barriers k L[i,k] i ` k g[k] i L[i,k] f[i] light box

124. Content-Adaptive Parallax Barriers L[i,k] ` k g[k] i f[i] light box

126. Content-Adaptive Parallax Barriers ` =

127. Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010 Rank-Constrained Displays and LF Adaptation ` Content-Adaptive Parallax Barriers = All dual layer display = rank-1 constraint Light field display is a matrix approximation problem Exploit content-adaptive parallax barriers

128. Optimization: Iteration 1 rear mask: f1[i,j] front mask: g1[k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

129. Optimization: Iteration 10 rear mask: f1[i,j] front mask: g1[k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

130. Optimization: Iteration 20 rear mask: f1[i,j] front mask: g1[k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

131. Optimization: Iteration 30 rear mask: f1[i,j] front mask: g1[k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

132. Optimization: Iteration 40 rear mask: f1[i,j] front mask: g1[k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

133. Optimization: Iteration 50 rear mask: f1[i,j] front mask: g1[k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

134. Optimization: Iteration 60 rear mask: f1[i,j] front mask: g1[k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

135. Optimization: Iteration 70 rear mask: f1[i,j] front mask: g1[k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

136. Optimization: Iteration 80 rear mask: f1[i,j] front mask: g1[k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

137. Optimization: Iteration 90 rear mask: f1[i,j] front mask: g1[k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

138. Content-Adaptive Front Mask (1 of 9)

139. Content-Adaptive Rear Mask (1 of 9)

140. Emitted 4D Light Field

142. With a fixed pair of masks, emitted light field is rank-1

143. Achieved higher-rank approximation using temporal multiplexing

144. With T time-multiplexed masks, emitted light field is rank-T

145. Constructed a prototype using off-the-shelf panels

146. Demonstrated light field display is a matrix approximation problem

147. Introduced content-adaptive parallax barriers

148. Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

149. Parallax Barrier: Np=103 pix. Hologram: NH=105 pix. ϕP∝w/d ϕH∝λ/tH θp=10 pix θH =1000 pix Fourier Patch xH =100 patches w xp=100slits Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009

150. Augmented Light Field 118 wave optics based rigorous but cumbersome Wigner Distribution Function WDF Augmented LF Traditional Light Field Traditional Light Field ray optics based simple and powerful Interference & Diffraction Interaction w/ optical elements Oh, Raskar, Barbastathis 2009: Augmented Light Field

151. position light field transformer LF LF LF LF (diffractive) optical element Reference plane LF propagation LF propagation Light Fields Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters angle

152. Augmented Lightfield for Wave Optics Effects WDF Wigner Distribution Function Augmented Light Field Light Field LF LF < WDF Lacks phase properties Ignores diffraction, interferrence Radiance = Positive ALF ~ WDF Supports coherent/incoherent Radiance = Positive/Negative Virtual light sources

153. Free-space propagation Light field transformer Virtual light projector Possibly negative radiance 121

154. Lightfieldvs Hologram Displays

155. Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?

156. Zooming into the Light Field Rays: No Bending 1 Fresnel HG Patch p Wm * * p d(θ) q d(θ) * q q p p * q Wm L(x,θ) W(x,u) Wm= sinc d = delta u θ

157. Algebraic Rank Constraint Rank-3 Rank-1 Rank-1 s1* s1 m2 s1* m2 s1 (a) Parallax Barrier (c) Hybrid (b) Hologram s1 s1

158. (a) Two Slits, Coherent Interference xʹ Rank-1 -1 Transform u -Transform R45, D x <t(x+xʹ/2)t*(x-xʹ/2)> t(x1)t*(x2) t(x+xʹ/2)t*(x-xʹ/2) W(x,u)

159. L1(x,θ) (a) L1 L2(x,θ) L2 L3(x,θ) s1 m2 L3 hH ϕ1 ϕ1 z2 ϕ1 ϕ1 z1 r d L3(x,θ) L1(x,θ) L2(x,θ)

160. Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?

161. Three Questions What are the benefits of higher dimensional imaging? Why is the algebraic rank of a Light Field not full? What makes looking around the corner possible?

162. How to do Research in Imaging http://raskar.info How to come up w ideas: Idea Hexagon How to write a paper How to give a talk Open research problems How to decide merit of a project How to attend a conference, brainstorm Facebook.com/ rRaskar Tips Get on Seminar/Talks mailing lists worldwide http://www.cs.virginia.edu/~robins/YouAndYourResearch.html Why do so few scientists make significant contributions and so many are forgotten in the long run? Highly recommended Hamming talk at Bell Labs

163. Take home points Co-design of hw/sw Avoid computational or optical chauvinism in imaging (Camera flash/Kinect) Hardware cost going to zero, Parallel technology trends Computer vision not just mimicking human vision/perception Borrow ideas from other fields: astronomy, scientific imaging, audio, communications Photons not just Pixels Change the rules of the game Optics, Sensors, Illum, Priors, Sparsity, Transforms Meta-data, Internet collection, Crowdsourcing