1. Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab

3. Where are the ‘camera’s?

4. Where are the ‘camera’s?

7. Cameras of Tomorrow

9. Computational Illumination Curved Planar Non-planar Single Projector Multiple Projectors Pocket-Proj Objects 1998 1998 2002 2002 1999 2002 2003 1998 1997 Computational Camera and Photography Projector j User : T ?

10. Motion Blurred Photo

12. Flutter Shutter Camera Raskar, Agrawal, Tumblin [Siggraph2006] LCD opacity switched in coded sequence

13. Short Exposure Traditional MURA Coded Deblurred Result Captured Single Photo Shutter Banding Artifacts and some spatial frequencies are lost Dark and noisy

14. Blurring == Convolution Traditional Camera: Shutter is OPEN: Box Filter PSF == Sinc Function ω Sharp Photo Blurred Photo Fourier Transform

15. Flutter Shutter: Shutter is OPEN and CLOSED Sharp Photo Blurred Photo PSF == Broadband Function Fourier Transform Preserves High Spatial Frequencies

16. Traditional Coded Exposure Image of Static Object Deblurred Image Deblurred Image

18. Coded Exposure Temporal 1-D broadband code: Motion Deblurring Coded Aperture Spatial 2-D broadband mask: Focus Deblurring

19. Coded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern

20. In Focus Photo LED

21. Out of Focus Photo: Open Aperture

22. Out of Focus Photo: Coded Aperture

23. Captured Blurred Photo

24. Refocused on Person

25. Less is More Blocking Light == More Information Coding in Time Coding in Space

26. Larval Trematode Worm Coded Aperture Camera “ Photography is the Art of Exclusion”

27. Shielding Light … Larval Trematode Worm Turbellarian Worm

37. Mask? Sensor 4D Light Field from 2D Photo: Heterodyne Light Field Camera Full Resolution Digital Refocusing: Coded Aperture Camera Mask? Sensor Mask Sensor Mask? Sensor Mask Sensor

38. Light Field Inside a Camera

39. Lenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al. 2005 ] Light Field Inside a Camera

41. Digital Refocusing [Ng et al 2005] Can we achieve this with a Mask alone?

42. Mask based Light Field Camera [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor

43. How to Capture 4D Light Field with 2D Sensor ? What should be the pattern of the mask ?

44. Optical Heterodyning Photographic Signal (Light Field) Carrier Incident Modulated Signal Reference Carrier Main Lens Object Mask Sensor Recovered Light Field Software Demodulation Baseband Audio Signal Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz

45. 1/f 0 Mask Tile Cosine Mask Used

46. Captured 2D Photo Encoding due to Mask

47. 2D FFT Traditional Camera Photo Heterodyne Camera Photo Magnitude of 2D FFT 2D FFT Magnitude of 2D FFT

48. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*9 4D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9

49. How to Capture 2D Light Field with 1D Sensor ? f θ f x f θ 0 f x0 Band-limited Light Field Sensor Slice Fourier Light Field Space

50. Extra sensor bandwidth cannot capture extra dimension of the light field f θ f x f θ 0 f x0 Sensor Slice Extra sensor bandwidth Fourier Light Field Space

51. Solution: Modulation Theorem Make spectral copies of 2D light field f θ f x f θ 0 f x0 Modulation Function

52. f θ Modulated Light Field f x f θ 0 f x0 Modulation Function Sensor Slice captures entire Light Field

53. 1/f 0 Mask Tile Cosine Mask Used

54. Demodulation to recover Light Field f θ f x Reshape 1D Fourier Transform into 2D 1D Fourier Transform of Sensor Signal

55. Mask Sensor Where to place the Mask? Mask Modulation Function f x f θ

56. Full resolution 2D image of Focused Scene Parts Captured 2D Photo Image of White Lambertian Plane divide

57. Coding and Modulation in Camera Using Masks Coded Aperture for Full Resolution Digital Refocusing Heterodyne Light Field Camera Mask? Sensor Mask Sensor Mask Sensor

58. Agile Spectrum Imaging Programmable Color Gamut for Sensor With Ankit Mohan, Jack Tumblin [Eurographics 2008]

59. R ≈ 0.0 G ≈ 0.2 B ≈ 0.8 Traditional Fixed Color Gamut B G R

60. Adaptive Color Primaries

61. C B A A’ B’ C’ Pinhole Lens L 1 Prism or Diffraction Grating Lens L 2 Sensor Rainbow Plane C’’ B’’ A’’ Scene Rainbow Plane inside Camera

62. Lens Flare Reduction/Enhancement using 4D Ray Sampling Captured Glare Reduced Glare Enhanced

64. Computational Camera and Photography

65. Dependence on incident angle

66. Dependence on incident angle

67. Towards a 6D Display Passive Reflectance Field Display Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch Siggraph 2008 1 2 1 1 1 MPI Informatik, Germany 2 MIT 2D 2D 2D

68. View dependent 4D Display

71. 6D = light sensitive 4D display

72. One Pixel of a 6D Display = 4D Display

73. Single shot visual hull Lanman, Raskar, Agrawal, Taubin [Siggraph Asia 2008]

74. Single shot 3D reconstruction: Simultaneous Projections using Masks

77. Vicon Motion Capture High-speed IR Camera Body-worn markers Medical Rehabilitation Athlete Analysis Performance Capture Biomechanical Analysis

79. Inside of Projector The Gray code pattern Focusing Optics Gray code Slide Condensing Optics Light Source

80. Imperceptible Tags under clothing, tracked under ambient light

82. Coded Imaging

83. Forerunners .. Mask Sensor Mask Sensor

84. Fernald, Science [Sept 2006] Shadow Refractive Reflective Tools for Visual Computing

86. Blind Camera Sascha Pohflepp, U of the Art, Berlin, 2006

87. Cameras of Tomorrow