Though revolutionary in many ways, digital photography is essentially electronically implemented film photography. By contrast, computational photography exploits plentiful low-cost computing and memory, new kinds of digitally enabled sensors, optics, probes, smart lighting, and communication to capture information far beyond just a simple set of pixels. It promises a richer, even a multilayered, visual experience that may include depth, fused photo-video representations, or multispectral imagery. Professor Raskar will discuss and demonstrate advances he is working on in the areas of generalized optics, sensors, illumination methods, processing, and display, and describe how computational photography will enable us to create images that break from traditional constraints to retain more fully our fondest and most important memories, to keep personalized records of our lives, and to extend both the archival and the artistic possibilities of photography.

1. Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab Photography Wishlist http://raskar.info

2. Film-like Digital Photography

3. Fernald, Science [Sept 2006] Shadow Refractive Reflective

4. ‘ film-like’ Photography Lens Detector Pixels Image Slide by Shree Nayar Reproduce for the eye

6. Computational Photography Computational Illumination Computational Camera Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing Ray Bender Ray Sampler Ray Reconstruction Generalized Optics Recreate 4D Lightfield Light Sources Modulators 4D Incident Lighting 4D Light Field

7. Can you look around a corner ?

10. Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab Computational Photography Wish List http://raskar.info

11. Synthesis Low Level Mid Level High Level Hyper realism Raw Angle, spectrum aware Non-visual Data, GPS Metadata Priors Comprehensive 8D reflectance field Computational Photography Digital Epsilon Coded Essence Computational Photography aims to make progress on both axis Camera Array HDR, FoV Focal stack Decomposition problems Depth Spectrum LightFields Human Stereo Vision Transient Imaging Virtual Object Insertion Relighting Augmented Human Experience Material editing from single photo Scene completion from photos Motion Magnification Phototourism Resolution

12. Camera Culture Ramesh Raskar Wish #1 Ultimate Post-capture Control

13. Digital Refocusing using Light Field Camera 125 μ square-sided microlenses [Ng et al 2005]

14. Zooming into the raw photo

15. Mask based Light Field Camera [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor

16. Captured 2D Photo

17. x y u v 121 Sub-aperture Views

19. Figure 2 results Input Image Motion Blur in Low Light

20. Motion Blur in Low Light

21. Fluttered Shutter Camera Raskar, Agrawal, Tumblin Siggraph2006 Ferroelectric shutter in front of the lens is turned opaque or transparent in a rapid binary sequence

22. Image Deblurred by solving a linear system. No post-processing Blurred Taxi

23. Motion Blur in Low Light

24. Compact Programmable Lights ?

25. Image-Based Re-lighting Measure incoming light in Milan, Light the actress in LA Matte the background Matched LA and Milan lighting. Debevec et al., SIGG2001

27. Camera Culture Ramesh Raskar Wish #2 Freedom from Form

28. Convert LCD into a big flat camera ? Beyond Multi-touch: 3D Gestures

29. Large Virtual Camera for 3D Interactive HCI and Video Conferencing Matthew Hirsch, Henry Holtzman Doug Lanman, Ramesh Raskar Siggraph Asia 2009 BiDi Screen

30. Touch + Hover using Thin, Depth Sensing LCD Sensor

31. Sensing Depth from Array of Virtual Cameras in LCD

32. Shallow DoF with Simple Lens Lots of glass; Heavy; Bulky; Expensive

33. Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh Raskar MIT Media Lab MIT Media Lab Camera Culture

34. Image De stabilization Lens Sensor Camera Static Scene

35. Image Destabilization Static Scene Lens Motion Sensor Motion Camera Mohan, Lanman,Hiura, Raskar ICCP 2009

36. Shifting Pinhole and Sensor A B A’ B’ Pinhole v p Sensor v s d a d b d s Focus Here

37. Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here

38. Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here

39. Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here

40. “ Time Lens” Ratio of speeds Lens Equation: Virtual Focal Length: Virtual F-Number:

41. Time Lens:

42. Small Aperture Photo all-in-focus

43. Destabilized Small Aperture focused in the front using destabilization

44. Adjusting the Focus Plane focused in the middle using destabilization

45. Adjusting the Focus Plane focused in the back using destabilization

46. Camera Culture Ramesh Raskar Wish #3 Understand the World

47. Real or Fake ?

48. Direct Reflection

49. Fake Real Internal Reflection

50. High Frequency Pattern surface Camera Flash Nayar, Krishnan, Grossberg, Raskar [Siggraph 2006] i

51. Convert single 2D photo into 3D ? Snavely, Seitz, Szeliski U of Washington/Microsoft: Photosynth

52. Exploit Community Photo Collections U of Washington/Microsoft: Photosynth

53. Can you look around a corner ?

54. Can you ‘see’ around a corner ?

55. Femto-Photography: Higher Dimensional Capture FemtoFlash UltraFast Detector Computational Optics Serious Sync

57. Bokode

59. Defocus blur of Bokode

60. Coding in Angle Mohan, Woo, Smithwick, Hiura, Raskar [Siggraph 2009]

63. capturing Bokodes cell-phone camera close to the Bokode (10,000+ bytes of data)

65. Camera Culture Ramesh Raskar Wish #4 Sharing Visual Experience

67. 6D Photo Frames One Pixel of a 6D Display = 4D Display Single Pixel of 6D Frame 1 2 1 1 2D 2D 2D

68. 6D Photo Frames: Respond to Viewpoint + Ambient Light

70. Camera Culture Ramesh Raskar Wish #5 Capturing Essence

72. What are the problems with ‘real’ photo in conveying information ? Why do we hire artists to draw what can be photographed ?

73. Shadows Clutter Many Colors Highlight Shape Edges Mark moving parts Basic colors

74. Depth Edges with MultiFlash Raskar, Tan, Feris, Jingyi Yu, Turk – ACM SIGGRAPH 2004

79. Depth Discontinuities Internal and external Shape boundaries, Occluding contour, Silhouettes

80. Depth Edges

81. Our Method Canny

82. Canny Intensity Edge Detection Our Method Photo Result

86. Blind Camera Sascha Pohflepp, U of the Art, Berlin, 2006

87. Scene Completion Using Millions of Photographs Hays and Efros, Siggraph 2007

88. Camera Culture Ramesh Raskar Wish #5 …………………

89. Photos of tomorrow: computed not recorded http://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/

91. 2 nd International Conference on Computational Photography Papers due November 2, 2009 http://cameraculture.media.mit.edu/iccp10