Keywords: Signal processing, Applied optics, Computer graphics and vision, Electronics, Art, and Online photo collections A computational camera attempts to digitally capture the essence of visual information by exploiting the synergistic combination of task-specific optics, illumination, sensors and processing. We will discuss and play with thermal cameras, multi-spectral cameras, high-speed, and 3D range-sensing cameras and camera arrays. We will learn about opportunities in scientific and medical imaging, mobile-phone based photography, camera for HCI and sensors mimicking animal eyes. We will learn about the complete camera pipeline. In several hands-on projects we will build several physical imaging prototypes and understand how each stage of the imaging process can be manipulated. We will learn about modern methods for capturing and sharing visual information. If novel cameras can be designed to sample light in radically new ways, then rich and useful forms of visual information may be recorded -- beyond those present in traditional protographs. Furthermore, if computational process can be made aware of these novel imaging models, them the scene can be analyzed in higher dimensions and novel aesthetic renderings of the visual information can be synthesized. In this couse we will study this emerging multi-disciplinary field -- one which is at the intersection of signal processing, applied optics, computer graphics and vision, electronics, art, and online sharing through social networks. We will examine whether such innovative camera-like sensors can overcome the tough problems in scene understanding and generate insightful awareness. In addition, we will develop new algorithms to exploit unusual optics, programmable wavelength control, and femto-second accurate photon counting to decompose the sensed values into perceptually critical elements.

1. MIT Media LabMIT Media Lab
Camera CultureCamera Culture
Ramesh RaskarRamesh Raskar
MIT Media LabMIT Media Lab
http:// CameraCulture . info/http:// CameraCulture . info/
MAS 131/ 531MAS 131/ 531
Computational Camera &Computational Camera &
Photography:Photography:
MAS 131/ 531MAS 131/ 531
Computational Camera &Computational Camera &
Photography:Photography:

2. MIT Media LabMIT Media Lab
http://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/

3. Direct Global
Shower Curtain: Diffuser

4. Direct Global
Shower Curtain: Diffuser

5. A Teaser: Dual PhotographyA Teaser: Dual Photography
Scene
PhotocellProjector

6. A Teaser: Dual PhotographyA Teaser: Dual Photography
Scene
PhotocellProjector

7. A Teaser: Dual PhotographyA Teaser: Dual Photography
Scene
PhotocellProjector

8. A Teaser: Dual PhotographyA Teaser: Dual Photography
Scene
PhotocellProjector Camera

9. camera
The 4D transport matrix:The 4D transport matrix:
Contribution of each projector pixel to each camera pixelContribution of each projector pixel to each camera pixel
scene
projector

10. camera
The 4D transport matrix:The 4D transport matrix:
Contribution of each projector pixel to each camera pixelContribution of each projector pixel to each camera pixel
scene
projector
Sen et al, Siggraph 2005Sen et al, Siggraph 2005

11. camera
The 4D transport matrix:The 4D transport matrix:
Which projector pixel contribute to each camera pixelWhich projector pixel contribute to each camera pixel
scene
projector
Sen et al, Siggraph 2005Sen et al, Siggraph 2005
??

12. Dual photography from diffuse reflections:Dual photography from diffuse reflections:
Homework Assignment 2Homework Assignment 2
the camera’s view
Sen et al, Siggraph 2005Sen et al, Siggraph 2005

13. Digital cameras are boring:Digital cameras are boring:
Film-like PhotographyFilm-like Photography
• Roughly the same features and controls as film camerasRoughly the same features and controls as film cameras
– zoom and focuszoom and focus
– aperture and exposureaperture and exposure
– shutter release and advanceshutter release and advance
– one shutter press = one snapshotone shutter press = one snapshot

14. Improving FILM-LIKEImproving FILM-LIKE
Camera PerformanceCamera Performance
What would make it ‘perfect’ ?What would make it ‘perfect’ ?
• Dynamic RangeDynamic Range
• Vary Focus Point-by-PointVary Focus Point-by-Point
• Field of view vs. ResolutionField of view vs. Resolution
• Exposure time and Frame rateExposure time and Frame rate

15. MIT Media Lab
• What type of ‘Cameras’ will we study?
• Not just film-mimicking 2D sensors
– 0D sensors
• Motion detector
• Bar code scanner
• Time-of-flight range detector
– 1D sensors
• Line scan camera (photofinish)
• Flatbed scanner
• Fax machine
– 2D sensors
– 2-1/2D sensors
– ‘3D’ sensors
– 4D and 6D tomography machines and displays

16. Can you look around a
corner ?

17. Convert LCD into a big flat camera?
Beyond Multi-touch

18. 18
Medical Applications of Tomography
bone reconstruction segmented vessels
Slides by Doug Lanman

19. MIT Media Lab
Camera Culture
Ramesh Raskar
Camera Culture

20. Mitsubishi Electric Research Laboratories Raskar 2006Spatial Augmented Reality
CurvedPlanar Non-planar
Single
Projector
Multiple
Projectors
Projector
j
User
: T
?
Pocket-
Proj
Objects
Computational IlluminationComputational Illumination
1998
1998 2002
2002
1999
2002
20031998
1997
Computational PhotographyComputational Photography
My Background

21. MIT Media Lab
Questions
• What will a camera look like in 10,20 years?
• How will the next billion cameras change the social culture?
• How can we augment the camera to support best ‘image search’?
• What are the opportunities in pervasive recording?
– e.g. GoogleEarth Live
• How will ultra-high-speed/resolution imaging change us?
• How should we change cameras for
movie-making, news reporting?

22. MIT Media Lab
Approach
• Not just USE but CHANGE camera
– Optics, illumination, sensor, movement
– Exploit wavelength, speed, depth, polarization etc
– Probes, actuators, Network
• We have exhausted bits in pixels
– Scene understanding is challenging
– Build feature-revealing cameras
– Process photons

23. PlanPlan
• What is Computational Camera?What is Computational Camera?
• IntroductionsIntroductions
• Class formatClass format
• Fast Forward PreviewFast Forward Preview
– Sample topicsSample topics
• First warmup assignmentFirst warmup assignment

24. Fernald, Science [Sept 2006]
Shadow Refractive Reflective
Tools
for
Visual
Computing

25. Traditional ‘film-like’ PhotographyTraditional ‘film-like’ Photography
Lens
Detector
Pixels
Image
Slide by Shree Nayar

26. Computational CameraComputational Camera::
Optics, Sensors and ComputationsOptics, Sensors and Computations
Generalized
Sensor
Generalized
Optics
Computations
Picture
4D Ray Bender
Upto 4D
Ray Sampler
Ray Reconstruction
Raskar and Tumblin

27. Novel Cameras
Generalized
Sensor
Generalized
Optics
Processing

28. Programmable Lighting
Novel Cameras
Scene
Generalized
Sensor
Generalized
Optics
Processing
Generalized
Optics
Light Sources
Modulators

29. Cameras EverywhereCameras Everywhere

30. Where are the ‘cameras’?Where are the ‘cameras’?

31. Where are the ‘cameras’?Where are the ‘cameras’?

32. DIY Green Screen Effects $160
Panasonic Beauty Appliance

33. Fujifilm FinePix 3D (stereo)
GigaPan Epic 100

34. Simply getting depth is challenging !
Godin et al. An Assessment of Laser Range Measurement on Marble Surfaces. Intl.
M. Levoy. Why is 3D scanning hard? 3DPVT, 2002
 Must be simultaneously illuminated and imaged (occlusion problems)
 Non-Lambertian BRDFs (transparency, reflections, subsurface scattering)
 Acquisition time (dynamic scenes), large (or small) features, etc.
Lanman and Taubin’09

35. Contact
Non-Contact
Active
Passive
Transmissive
Reflective
Shape-from-X
(stereo/multi-view, silhouettes, focus/defocus, motion, texture, etc.)
Active Variants of Passive Methods
(stereo/focus/defocus using projected patterns)
Time-of-Flight
Triangulation
(laser striping and structured lighting)
Computed Tomography (CT)
Transmissive Ultrasound
Direct Measurements
(rulers, calipers, pantographs, coordinate measuring machines (CMM), AFM)
Non-optical Methods
(reflective ultrasound, radar, sonar, MRI)
Taxonomy of 3D Scanning:
Lanman and Taubin’09

36. DARPA Grand ChallengeDARPA Grand Challenge

37. Do-It-Yourself (DIY) 3D Scanners
Lanman and Taubin’09

38. What is ‘interesting’ here?
Social voting in the real world = ‘popular’

39. Pantheon

40. How do we move through a space?

41. Computational Photography
[Raskar and Tumblin]
1. Epsilon Photography
– Low-level vision: Pixels
– Multi-photos by perturbing camera parameters
– HDR, panorama, …
– ‘Ultimate camera’
1. Coded Photography
– Mid-Level Cues:
• Regions, Edges, Motion, Direct/global
– Single/few snapshot
• Reversible encoding of data
– Additional sensors/optics/illum
– ‘Scene analysis’
1. Essence Photography
– High-level understanding
• Not mimic human eye
• Beyond single view/illum
– ‘New artform’
captures a machine-readable representation of our world to
hyper-realistically synthesize the essence of our visual experience.

42. Goal and Experience
Low Level Mid Level High
Level
Hyper
realism
Raw
Angle,
spectrum
aware
Non-visual
Data, GPS
Metadata
Priors
Comprehensive
8D reflectance
field
Digital
Epsilon
Coded
Essence
CP aims to
make progress
on both axis
Camera Array
HDR, FoV Focal stack
Decompositio
n problems
Depth
Spectrum
LightFields
Human Stereo
Vision
Transient
Imaging
Virtual Object
Insertion
Relighting
Augmented
Human
Experience
Material
editing from
single photo
Scene
completion
from photos
Motion
Magnification
Phototourism

44. • Ramesh Raskar and
Jack Tumblin
• Book Publishers: A K Peters
• ComputationalPhotography.org

45. GoalsGoals
• Change the rules of the gameChange the rules of the game
– Emerging optics, illumination, novel sensorsEmerging optics, illumination, novel sensors
– Exploit priors and online collectionsExploit priors and online collections
• ApplicationsApplications
– Better scene understanding/analysisBetter scene understanding/analysis
– Capture visual essenceCapture visual essence
– Superior Metadata tagging for effective sharingSuperior Metadata tagging for effective sharing
– Fuse non-visual dataFuse non-visual data
• Sensors for disabled, new art forms, crowdsourcing,Sensors for disabled, new art forms, crowdsourcing,
bridging culturesbridging cultures

46. Vein ViewerVein Viewer (Luminetx)(Luminetx)
Locate subcutaneous veinsLocate subcutaneous veins

47. Vein ViewerVein Viewer (Luminetx)(Luminetx)
Near-IR camera locates subcutaneous veins and projectNear-IR camera locates subcutaneous veins and project
their location onto the surface of the skin.their location onto the surface of the skin.
Coaxial IR cameraCoaxial IR camera
+ Projector+ Projector
Coaxial IR cameraCoaxial IR camera
+ Projector+ Projector

49. Beyond Visible SpectrumBeyond Visible Spectrum
CedipRedShift

50. • FormatFormat
– 4 (3) Assignments4 (3) Assignments
• Hands on with optics,Hands on with optics,
illumination, sensors, masksillumination, sensors, masks
• Rolling schedule for overlapRolling schedule for overlap
• We have cameras, lenses,We have cameras, lenses,
electronics, projectors etcelectronics, projectors etc
• Vote on best projectVote on best project
– Mid term examMid term exam
• Test conceptsTest concepts
– 1 Final project1 Final project
• Should be a Novel and CoolShould be a Novel and Cool
• Conference quality paperConference quality paper
• Award for best projectAward for best project
– Take 1 class notesTake 1 class notes
– Lectures (and guestLectures (and guest
talks)talks)
– In-class + onlineIn-class + online
discussiondiscussion
• If you are a listenerIf you are a listener
– Participate in online discussion, digParticipate in online discussion, dig
new recent worknew recent work
– Present one short 15 minute idea orPresent one short 15 minute idea or
new worknew work
• CreditCredit
• Assignments: 40%Assignments: 40%
• Project: 30%Project: 30%
• Mid-term: 20%Mid-term: 20%
• Class participation: 10%Class participation: 10%
• Pre-reqsPre-reqs
• Helpful: Linear algebra, imageHelpful: Linear algebra, image
processing, think in 3Dprocessing, think in 3D
• We will try to keep math toWe will try to keep math to
essentials, but complex conceptsessentials, but complex concepts

51. What is the emphasis?What is the emphasis?
• Learn fundamental techniques in imagingLearn fundamental techniques in imaging
– In class and in homeworksIn class and in homeworks
– Signal processing, Applied optics, Computer graphics and vision,Signal processing, Applied optics, Computer graphics and vision,
Electronics, Art, and Online photo collectionsElectronics, Art, and Online photo collections
– This is not a discussion classThis is not a discussion class
• Three Applications areasThree Applications areas
– PhotographyPhotography
• Think in higher dimensions 3D, 4D, 6D, 8D, thermal IR, range cam,Think in higher dimensions 3D, 4D, 6D, 8D, thermal IR, range cam,
lightfields, applied opticslightfields, applied optics
– Active Computer Vision (real-time)Active Computer Vision (real-time)
• HCI, Robotics, Tracking/Segmentation etcHCI, Robotics, Tracking/Segmentation etc
– Scientific ImagingScientific Imaging
• Compressive sensing, wavefront coding, tomography, deconvolution, psfCompressive sensing, wavefront coding, tomography, deconvolution, psf
– But the 3 areas are merging and use similar principlesBut the 3 areas are merging and use similar principles

52. Pre-reqsPre-reqs
• Two tracks:Two tracks:
– Supporting students with varying backgroundsSupporting students with varying backgrounds
– A. software-intensive (Photoshop/HDRshop maybe ok)A. software-intensive (Photoshop/HDRshop maybe ok)
• But you will actually take longer to do assignmentsBut you will actually take longer to do assignments
– B. software-hardware (electronics/optics) emphasis.B. software-hardware (electronics/optics) emphasis.
• Helpful:Helpful:
– Watch all videos on http://raskar.info/photo/Watch all videos on http://raskar.info/photo/
– Linear algebra, image processing, think in 3DLinear algebra, image processing, think in 3D
– Signal processing, Applied optics, Computer graphics and vision, Electronics,Signal processing, Applied optics, Computer graphics and vision, Electronics,
Art, and Online photo collectionsArt, and Online photo collections
• We will try to keep math to essentials, but introduce complex concepts at rapidWe will try to keep math to essentials, but introduce complex concepts at rapid
pacepace
• Assignments versus Class materialAssignments versus Class material
– Class material will present material with varying degree of complexityClass material will present material with varying degree of complexity
– Each assignments has sub-elements with increasing sophisticationEach assignments has sub-elements with increasing sophistication
– You can pick your levelYou can pick your level

53. Assignments:Assignments:
You are encouraged to program in Matlab for image analysisYou are encouraged to program in Matlab for image analysis
You may need to use C++/OpenGL/Visual programming for some hardware assignmentsYou may need to use C++/OpenGL/Visual programming for some hardware assignments
Each student is expected to prepare notes for one lectureEach student is expected to prepare notes for one lecture
These notes should be prepared and emailed to the instructor no later than the followingThese notes should be prepared and emailed to the instructor no later than the following
Monday night (midnight EST). Revisions and corrections will be exchanged by email andMonday night (midnight EST). Revisions and corrections will be exchanged by email and
after changes the notes will be posted to the website before class the following week.after changes the notes will be posted to the website before class the following week.
5 points5 points
Course mailing listCourse mailing list: Please make sure that your emailid is on the course mailing list: Please make sure that your emailid is on the course mailing list
Send email to raskar (at) media.mit.eduSend email to raskar (at) media.mit.edu
Please fill in the email/credit/dept sheetPlease fill in the email/credit/dept sheet
Office hoursOffice hours::
Email is the best way to get in touchEmail is the best way to get in touch
Ramesh:. raskar (at) media.mit.eduRamesh:. raskar (at) media.mit.edu
Ankit:Ankit: ankit (at) media.mit.eduankit (at) media.mit.edu
Other mentors: Prof Mukaigawa, Dr Ashok VeeraraghavanOther mentors: Prof Mukaigawa, Dr Ashok Veeraraghavan
After class:After class:
Muddy Charles PubMuddy Charles Pub
(Walker Memorial next to tennis courts, only with valid id)(Walker Memorial next to tennis courts, only with valid id)

54. 2 Sept 18th Modern Optics and Lenses, Ray-matrix operations
3 Sept 25th Virtual Optical Bench, Lightfield Photography, Fourier Optics, Wavefront Coding
4 Oct 2nd Digital Illumination, Hadamard Coded and Multispectral Illumination
5 Oct 9th
Emerging Sensors: High speed imaging, 3D range sensors, Femto-second concepts, Front/back
illumination, Diffraction issues
6
Oct 16th
Beyond Visible Spectrum: Multispectral imaging and Thermal sensors, Fluorescent imaging, 'Audio
camera'
7 Oct 23rd
Image Reconstruction Techniques, Deconvolution, Motion and Defocus Deblurring, Tomography,
Heterodyned Photography, Compressive Sensing
8
Oct 30th
Cameras for Human Computer Interaction (HCI): 0-D and 1-D sensors, Spatio-temporal coding,
Frustrated TIR, Camera-display fusion
9
Nov 6th
Useful techniques in Scientific and Medical Imaging: CT-scans, Strobing, Endoscopes, Astronomy
and Long range imaging
10
Nov 13th Mid-term Exam, Mobile Photography, Video Blogging, Life logs and Online Photo collections
11
Nov 20th Optics and Sensing in Animal Eyes. What can we learn from successful biological vision systems?
12
Nov 27th Thanksgiving Holiday (No Class)
13
Dec 4th Final Projects

55. Topics not coveredTopics not covered
• Only a bit of topics belowOnly a bit of topics below
• Art and AestheticsArt and Aesthetics
• 4.343 Photography and Related Media4.343 Photography and Related Media
• Software Image ManipulationSoftware Image Manipulation
– Traditional computer vision,Traditional computer vision,
– Camera fundamentals, Image processing, Learning,Camera fundamentals, Image processing, Learning,
• 6.815/6.865 Digital and Computational Photography6.815/6.865 Digital and Computational Photography
• OpticsOptics
• 2.71/2.710 Optics2.71/2.710 Optics
• PhotoshopPhotoshop
– Tricks, toolsTricks, tools
• Camera OperationCamera Operation
– Whatever is in the instruction manualWhatever is in the instruction manual

56. Courses related to CompCameraCourses related to CompCamera
• Spring 2010:Spring 2010:
– Camera Culture Seminar [Raskar, Media Lab]Camera Culture Seminar [Raskar, Media Lab]
• Graduate seminarGraduate seminar
• Guest lectures + in class discussionGuest lectures + in class discussion
• Homework question each weekHomework question each week
• Final survey paper (or project)Final survey paper (or project)
• CompCamera class: hands on projects, technical detailsCompCamera class: hands on projects, technical details
– Digital and Computational Photography [Durand, CSAIL]Digital and Computational Photography [Durand, CSAIL]
• Emphasis on software methods, Graphics and image processingEmphasis on software methods, Graphics and image processing
• CompCamera class: hardware projects, devices, beyond visibleCompCamera class: hardware projects, devices, beyond visible
spectrum/next gen camerasspectrum/next gen cameras
– Optics [George Barbastathis, MechE]Optics [George Barbastathis, MechE]
• Fourier optics, coherent imagingFourier optics, coherent imaging
• CompCamera class: Photography, time-domain, sensors, illuminationCompCamera class: Photography, time-domain, sensors, illumination
– Computational Imaging (Horn, Spring 2006)Computational Imaging (Horn, Spring 2006)
• Coding, Nuclear/Astronomical imaging, emphasis on theoryCoding, Nuclear/Astronomical imaging, emphasis on theory

57. Questions ..Questions ..

58. • Brief IntroductionsBrief Introductions
• Are you a photographer ?Are you a photographer ?
• Do you use camera for vision/imageDo you use camera for vision/image
processing? Real-time processing?processing? Real-time processing?
• Do you have background inDo you have background in
optics/sensors?optics/sensors?
• Name, Dept, Year, Why you are hereName, Dept, Year, Why you are here

59. 2nd
International
Conference on
Computational
Photography
Papers due
November 2,
2009
http://cameraculture.media.mit.edu/iccp10

60. Writing a Conference Quality PaperWriting a Conference Quality Paper
• How to come up with new ideasHow to come up with new ideas
– See slideshow on Stellar siteSee slideshow on Stellar site
• Developing your ideaDeveloping your idea
– Deciding if it is worth persuingDeciding if it is worth persuing
– http://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechismhttp://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechism
– What are you trying to do? How is it done today, and what are the limits of currentWhat are you trying to do? How is it done today, and what are the limits of current
practice? What's new in your approach and why do you think it will be successful? Whopractice? What's new in your approach and why do you think it will be successful? Who
cares? If you're successful, what difference will it make? What are the risks and thecares? If you're successful, what difference will it make? What are the risks and the
payoffs? How much will it cost? How long will it take?payoffs? How much will it cost? How long will it take?
• Last year outcomeLast year outcome
– 3 Siggraph/ICCV submissions, SRC award, 2 major research themes3 Siggraph/ICCV submissions, SRC award, 2 major research themes

61. How to quickly get started writing a paperHow to quickly get started writing a paper
• AbstractAbstract
• 1. Introduction1. Introduction
• MotivationMotivation
• Contributions** (For the first time, we have shown that xyz)Contributions** (For the first time, we have shown that xyz)
• Related WorkRelated Work
• Limitations and BenefitsLimitations and Benefits
• 2. Method2. Method
• (For every section as well as paragraph, first sentence should be the 'conlusion'(For every section as well as paragraph, first sentence should be the 'conlusion'
of what that section or paragraph is going to show)of what that section or paragraph is going to show)
• 3. More Second Order details (Section title will change)3. More Second Order details (Section title will change)
• 4. Implementation4. Implementation
• 5. Results5. Results
• Performance EvaluationPerformance Evaluation
• DemonstrationDemonstration
• 6. Discussion and Issues6. Discussion and Issues
• Future DirectionsFuture Directions
• 7. Conclusion7. Conclusion

62. Casio EX F1Casio EX F1
• What can it do?What can it do?
– Mostly high speed imagingMostly high speed imaging
– 1200 fps1200 fps
– Burst modeBurst mode
• Déjà vuDéjà vu (Media Lab 1998) and(Media Lab 1998) and Moment CameraMoment Camera (Michael(Michael
Cohen 2005)Cohen 2005)
• HDRHDR
• MovieMovie

64. Cameras and PhotographyCameras and Photography
Art, Magic, MiracleArt, Magic, Miracle

65. TopicsTopics
• Smart LightingSmart Lighting
– Light stages, Domes, Light waving, Towards 8DLight stages, Domes, Light waving, Towards 8D
• Computational Imaging outside PhotographyComputational Imaging outside Photography
– Tomography, Coded Aperture ImagingTomography, Coded Aperture Imaging
• Smart OpticsSmart Optics
– Handheld Light field camera, ProgrammableHandheld Light field camera, Programmable
imaging/apertureimaging/aperture
• Smart SensorsSmart Sensors
– HDR Cameras, Gradient Sensing, Line-scan Cameras,HDR Cameras, Gradient Sensing, Line-scan Cameras,
DemodulatorsDemodulators
• SpeculationsSpeculations

66. Debevec et al. 2002: ‘Light Stage 3’

67. Image-Based Actual Re-lightingImage-Based Actual Re-lighting
Film the background in Milan,Film the background in Milan,
Measure incoming light,Measure incoming light,
Light the actress in Los AngelesLight the actress in Los Angeles
Matte the backgroundMatte the background
Matched LA and Milan lighting.Matched LA and Milan lighting.
Debevec et al., SIGG2001

68. Can you look around a
corner ?

69. Can you look around a corner ?
Kirmani, Hutchinson, Davis, Raskar 2009
Accepted for ICCV’2009, Oct 2009 in Kyoto
Impulse Response of a Scene
cameraculture.media.mit.edu/femtotransientimaging

70. Femtosecond Laser as Light Source
Pico-second detector array as Camera

71. Rollout Photographs © Justin Kerr:Rollout Photographs © Justin Kerr:
Slide idea: Steve SeitzSlide idea: Steve Seitz
http://research.famsi.org/kerrmaya.html
AreAre BOTHBOTH a ‘photograph’?a ‘photograph’?

72. Part 2:Part 2:
Fast ForwardFast Forward
PreviewPreview

73. Synthetic LightingSynthetic Lighting
Paul Haeberli, Jan 1992Paul Haeberli, Jan 1992

74. HomeworkHomework
• Take multiple photos by changing lightingTake multiple photos by changing lighting
• Mix and match color channels to relightMix and match color channels to relight
• Due Sept 19thDue Sept 19th

75. Debevec et al. 2002: ‘Light Stage 3’

76. Image-Based Actual Re-lightingImage-Based Actual Re-lighting
Film the background in Milan,Film the background in Milan,
Measure incoming light,Measure incoming light,
Light the actress in Los AngelesLight the actress in Los Angeles
Matte the backgroundMatte the background
Matched LA and Milan lighting.Matched LA and Milan lighting.
Debevec et al., SIGG2001

77. Depth Edge CameraDepth Edge Camera

78. Ramesh Raskar, Karhan Tan, Rogerio Feris,Ramesh Raskar, Karhan Tan, Rogerio Feris,
Jingyi Yu, Matthew TurkJingyi Yu, Matthew Turk
Mitsubishi Electric Research Labs (MERL), Cambridge, MAMitsubishi Electric Research Labs (MERL), Cambridge, MA
U of California at Santa BarbaraU of California at Santa Barbara
U of North Carolina at Chapel HillU of North Carolina at Chapel Hill
Non-photorealistic Camera:Non-photorealistic Camera:
Depth Edge DetectionDepth Edge Detection andand Stylized RenderingStylized Rendering
usingusing
Multi-Flash ImagingMulti-Flash Imaging

83. Depth Discontinuities
Internal and external
Shape boundaries, Occluding contour, Silhouettes

84. Depth
Edges

85. Our MethodCanny

86. Participatory Urban SensingParticipatory Urban Sensing
Deborah Estrin talk yesterday
Static/semi-dynamic/dynamic data
A. City Maintenance
-Side Walks
B. Pollution
-Sensor network
C. Diet, Offenders
-Graffiti
-Bicycle on sidewalk
Future ..
Citizen Surveillance
Health Monitoring
http://research.cens.ucla.edu/areas/2007/Urban_Sensing/
(Erin Brockovich)
n

87. CrowdsourcingCrowdsourcing
http://www.wired.com/wired/archive/14.06/crowds.html
Object Recognition
Fakes
Template matching
Amazon Mechanical Turk:
Steve Fossett search
ReCAPTCHA=OCR

89. Community Photo CollectionsCommunity Photo Collections U of Washington/Microsoft: Photosynth

90. GigaPixel ImagesGigaPixel Images
Microsoft HDView
http://www.xrez.com/owens_giga.html
http://www.gigapxl.org/

91. OpticsOptics
• It is all about rays not pixelsIt is all about rays not pixels
• Study using lightfieldsStudy using lightfields

92. Assignment 2Assignment 2
• Andrew Adam’s Virtual Optical BenchAndrew Adam’s Virtual Optical Bench

93. Light Field Inside a CameraLight Field Inside a Camera

94. Lenslet-based Light Field cameraLenslet-based Light Field camera
[Adelson and Wang, 1992, Ng et al. 2005 ]
Light Field Inside a CameraLight Field Inside a Camera

95. Stanford Plenoptic CameraStanford Plenoptic Camera [Ng et al 2005][Ng et al 2005]
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses

96. Digital RefocusingDigital Refocusing
[Ng et al 2005][Ng et al 2005]
Can we achieve this with aCan we achieve this with a MaskMask alone?alone?

97. Mask based Light Field Camera
Mask
Sensor
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

98. How to Capture
4D Light Field with
2D Sensor ?
What should be the
pattern of the mask ?

99. Radio Frequency HeterodyningRadio Frequency Heterodyning
Baseband Audio
Signal
Receiver: DemodulationHigh Freq Carrier
100 MHz
Reference
Carrier
Incoming
Signal
99 MHz

100. Optical HeterodyningOptical Heterodyning
Photographic
Signal
(Light Field)
Carrier Incident
Modulated
Signal
Reference
Carrier
Main LensObject Mask Sensor
Recovered
Light
Field
Software Demodulation
Baseband Audio
Signal
Receiver: DemodulationHigh Freq Carrier
100 MHz
Reference
Carrier
Incoming
Signal
99 MHz

101. Captured 2D Photo
Encoding due to
Mask

102. 1/f0
Mask Tile
Cosine Mask Used

103. fθ
Modulated Light Field
fx
fθ0
fx0
Modulation
Function
Sensor Slice captures entire Light Field

104. 2D
FFT
Traditional Camera Photo
Heterodyne Camera Photo
Magnitude of 2D FFT
2D
FFT
Magnitude of 2D FFT

105. Computing 4D Light Field
2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800
2D
FFT
Rearrange 2D tiles into 4D planes
200*200*9*94D IFFT
4D Light Field
9*9=81 spectral copies
200*200*9*9

106. Agile Spectrum Imaging
With Ankit Mohan, Jack Tumblin [Eurographics 2008]

107. Lens Glare Reduction
[Raskar, Agrawal, Wilson, Veeraraghavan SIGGRAPH 2008]
Glare/Flare due to camera lenses reduces contrast

108. Glare Reduction/Enhancement usingGlare Reduction/Enhancement using
4D Ray Sampling4D Ray Sampling
Captured Glare
Reduced
Glare
Enhanced

109. i
j
x
Sensor
u
Glare = low frequency noise in 2D
•But is high frequency noise in 4D
•Remove via simple outlier rejection

110. Long-range
synthetic aperture photography
Levoy et al., SIGG2005

111. Synthetic aperture videography

112. Focus Adjustment: Sum of Bundles

113. Synthetic aperture photography
Smaller aperture   less blur, smaller circle of confusion

114. Synthetic aperture photography
Merge MANY cameras to act as ONE BIG LENS
Small items are so blurry
they seem to disappear..

115. Light field photography using a
handheld plenoptic camera
Ren Ng, Marc Levoy, Mathieu Brédif,
Gene Duval, Mark Horowitz and Pat Hanrahan

116. Prototype camera
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses

118. Example of digital refocusing

119. Extending the depth of field
conventional photograph,
main lens at f / 22
conventional photograph,
main lens at f / 4
light field, main lens at f / 4,
after all-focus algorithm
[Agarwala 2004]

120. Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Imaging in Sciences:Imaging in Sciences:
Computer TomographyComputer Tomography
• http://info.med.yale.edu/intmed/cardio/imaging/techniques/ct_imhttp://info.med.yale.edu/intmed/cardio/imaging/techniques/ct_im
aging/aging/

121. © 2004 Marc Levoy
Borehole tomography
• receivers measure end-to-end travel time
• reconstruct to find velocities in intervening cells
• must use limited-angle reconstruction method (like ART)
(from Reynolds)

122. © 2004 Marc Levoy
Deconvolution microscopy
• competitive with confocal imaging, and much faster
• assumes emission or attenuation, but not scattering
• therefore cannot be applied to opaque objects
• begins with less information than a light field (3D vrs 4D)
ordinary microscope image deconvolved from focus stack

123. Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Coded-Aperture ImagingCoded-Aperture Imaging
• Lens-free imaging!Lens-free imaging!
• Pinhole-cameraPinhole-camera
sharpness,sharpness,
without massive lightwithout massive light
loss.loss.
• No ray bending (OK forNo ray bending (OK for
X-ray, gamma ray, etc.)X-ray, gamma ray, etc.)
• Two elementsTwo elements
– Code Mask: binaryCode Mask: binary
(opaque/transparent)(opaque/transparent)
– Sensor gridSensor grid
• Mask autocorrelation isMask autocorrelation is
delta function (impulse)delta function (impulse)
• Similar to MotionSensorSimilar to MotionSensor

124. Mask in a Camera
Mask
Aperture
Canon EF 100 mm 1:1.28 Lens,
Canon SLR Rebel XT camera

125. Digital Refocusing
Captured Blurred Image

126. Digital Refocusing
Refocused Image on Person

127. Digital Refocusing

128. Larval Trematode WormLarval Trematode Worm

129. Mask? Sensor
Mask
SensorMask? Sensor
Mask
Sensor
Mask? Sensor
4D Light Field from 2D
Photo:
Heterodyne Light Field
Camera
Full Resolution Digital
Refocusing:
Coded Aperture Camera

130. Coding and Modulation in Camera Using MasksCoding and Modulation in Camera Using Masks
Mask? Sensor
Mask
Sensor
Mask
Sensor
Coded Aperture for
Full Resolution
Digital Refocusing
Heterodyne Light
Field Camera

131. Slides by Todor Georgiev

132. Slides by Todor Georgiev

133. Slides by Todor Georgiev

134. Conventional Lens: Limited Depth of FieldConventional Lens: Limited Depth of Field
Smaller
Aperture
Open
Aperture
Slides by Shree Nayar

135. Wavefront Coding using Cubic Phase PlateWavefront Coding using Cubic Phase Plate
"Wavefront Coding: jointly optimized optical and digital imaging systems“,
E. Dowski, R. H. Cormack and S. D. Sarama ,
Aerosense Conference, April 25, 2000
Slides by Shree Nayar

136. Depth Invariant BlurDepth Invariant Blur
Conventional System Wavefront Coded System
Slides by Shree Nayar

137. Typical PSF changes slowly Designed PSF changes fast
Decoding depth via defocus blur
• Design PSF that changes quickly through focus so
that defocus can be easily estimated
• Implementation using phase diffractive mask
(Sig 2008, Levin et al used amplitude mask)
Phase mask
R. Piestun, Y. Schechner, J. Shamir, “Propagation-Invariant Wave Fields with Finite Energy,” JOSA A 17, 294-303 (2000)
R. Piestun, J. Shamir, “Generalized propagation invariant wave-fields,” JOSA A 15, 3039 (1998)

138. Rotational PSFRotational PSF
R. Piestun, Y. Schechner, J. Shamir, “Propagation-Invariant Wave Fields with Finite Energy,” JOSA A 17, 294-303 (2000)
R. Piestun, J. Shamir, “Generalized propagation invariant wave-fields,” JOSA A 15, 3039 (1998)

139. Can we deal with particle-wave duality of light
with modern Lightfield theory ?
15
Young’s Double Slit
Expt
first null
(OPD = /2)λ
Diffraction and Interferences modeled using Ray representation

140. Light Fields
• Radiance per ray
• Ray parameterization:
• Position : x
• Direction : θ Reference
plane
position
direction
Goal: Representing propagation, interaction and image formation of light using
purely position and angle parameters

141. Light Fields for Wave Optics EffectsLight Fields for Wave Optics Effects
Wigner
Distribution
Function
Light
Field
LF < WDF
Lacks phase properties
Ignores diffraction, phase masks
Radiance = Positive
Light
Field
Augmente
d Light
Field
WDF
ALF ~ WDF
Supports coherent/incoherent
Radiance = Positive/Negative
Virtual light sources

142. Limitations of Traditional Lightfields
Wigner
Distribution
Function
TraditionalTraditional
Light FieldLight Field
TraditionalTraditional
Light FieldLight Field
ray optics based
simple and powerful
rigorous but cumbersome
wave optics based
limited in diffraction & interference
holograms beam shaping
rotational PSF

143. Example: New Representations Augmented Lightfields
Wigner
Distribution
Function
TraditionalTraditional
Light FieldLight Field
TraditionalTraditional
Light FieldLight Field
WDF
TraditionalTraditional
Light FieldLight Field
TraditionalTraditional
Light FieldLight Field
Augmented LF
Interference & Diffraction
Interaction w/ optical elements
ray optics based
simple and powerful
limited in diffraction & interference
rigorous but cumbersome
wave optics based
Non-paraxial propagation
http://raskar.scripts.mit.edu/~raskar/lightfields/

144. (ii) Augmented Light Field with LF
Transformer
15
WDF
LightLight
FieldField
LightLight
FieldField
Augmented LF
Interaction at the optical elements
LF
propagation
(diffractive)
optical
element
LF LF LF LF
LF
propagation
light field
transformer
negative
radiance
Augmenting Light Field to Model Wave Optics Effects , [Oh, Barbastathis, Raskar]

145. Virtual light projector with real valued (possibly
negative radiance) along a ray
15
real projector
real projector
first null
(OPD = /2)λ
virtual light projector
Augmenting Light Field to Model Wave Optics Effects , [Oh, Barbastathis, Raskar]

146. (ii) ALF with LF Transformer
15

147. “Origami Lens”: Thin Folded Optics (2007)
“Ultrathin Cameras Using Annular Folded Optics, “
E. J. Tremblay, R. A. Stack, R. L. Morrison, J. E. Ford
Applied Optics, 2007 - OSA
Slides by Shree Nayar

148. Gradient Index (GRIN) Optics
Conventional Convex LensGradient Index ‘Lens’
Continuous change of the refractive index
within the optical material
Constant refractive index but carefully
designed geometric shape
Refractive Index
along width
n
x
Change in RI is very small, 0.1 or 0.2

149. Photonic Crystals
• ‘Routers’ for photons instead of electrons
• Photonic Crystal
– Nanostructure material with ordered array of holes
– A lattice of high-RI material embedded within a lower RI
– High index contrast
– 2D or 3D periodic structure
• Photonic band gap
– Highly periodic structures that blocks certain wavelengths
– (creates a ‘gap’ or notch in wavelength)
• Applications
– ‘Semiconductors for light’: mimics silicon band gap for electrons
– Highly selective/rejecting narrow wavelength filters (Bayer Mosaic?)
– Light efficient LEDs
– Optical fibers with extreme bandwidth (wavelength multiplexing)
– Hype: future terahertz CPUs via optical communication on chip

150. Schlieren
Photography
• Image of small index of refraction gradients in a gas
• Invisible to human eye (subtle mirage effect)
Knife edge blocks half the light
unless
distorted beam focuses imperfectly
Collimated
Light
Camera

151. http://www.mne.psu.edu/psgdl/FSSPhotoalbum/index1.htm

152. Varying PolarizationVarying Polarization
Yoav Y. Schechner, Nir Karpel 2005Yoav Y. Schechner, Nir Karpel 2005
Best polarization state
Worst polarization state
Best polarization
state
Recovered
image
[Left] The raw images taken through a polarizer. [Right] White-balanced results:
The recovered image is much clearer, especially at distant objects, than the raw image

153. Varying PolarizationVarying Polarization
• Schechner, Narasimhan, NayarSchechner, Narasimhan, Nayar
• Instant dehazingInstant dehazing
of images usingof images using
polarizationpolarization

154. Photon-x:
Polarization Bayer Mosaic for
Surface normals

155. Novel SensorsNovel Sensors
• Gradient sensingGradient sensing
• HDR Camera, Log sensingHDR Camera, Log sensing
• Line-scan CameraLine-scan Camera
• DemodulatingDemodulating
• Motion CaptureMotion Capture
• 3D3D

156. MIT Media Lab
• Camera =
– 0D sensors
• Motion detector
• Bar code scanner
• Time-of-flight range detector (Darpa Grand Challenge)
– 1D sensors
• Line scan camera (photofinish)
• Flatbed scanner
• Fax machine
– 2D sensors
– 2-1/2D sensors
– ‘3D’ sensors

157. Single Pixel Camera

158. Compressed Imaging
∫ =
Scene
X Aggregate
Brightness
Y
“A New Compressive Imaging Camera Architecture”
D. Takhar et al., Proc. SPIE Symp. on Electronic Imaging, Vol. 6065, 2006.
Sparsity of Image: θΨ=X
sparse basis coefficients
XY Φ=
measurement basis
Measurements:

159. Single Pixel Camera
Image on the
DMD

160. Example
Original Compressed Imaging
4096 Pixels
1600 Measurements
(40%)
65536 Pixels
6600 Measurements
(10%)

161. Example
Original Compressed Imaging
4096 Pixels
800 Measurements
(20%)
4096 Pixels
1600 Measurements
(40%)

162. Line Scan Camera: PhotoFinish 2000 Hz

164. © 2004 Marc Levoy
The CityBlock Project
Precursor to Google Streetview Maps

165. Figure 2 results
Input Image
Problem: Motion Deblurring

166. Image Deblurred by solving a linear system. No post-processing
Blurred Taxi

167. Application: Aerial Imaging
Time = 0Time = T
Long Exposure:
The moving camera creates smear
Time
Shutter Open
Shutter Closed
Short Explosure:
Avoids blur. But the image is dark
Time
Time
Shutter Open
Shutter Closed
Goal: Capture sharp image with
sufficient brightness using a
camera on a fast moving aircraft
Sharpness versus Image Pixel Brightness
Time
Shutter Open
Shutter Closed
Solution:
Flutter Shutter

168. Application: Electronic Toll
Booths
Time
Goal: Automatic number
plate recognition from
sharp image
Monitoring Camera for detecting license plates
Time
Shutter Open
Shutter Closed
Solution:
Sufficiently long exposure
duration with fluttered shutter
Ideal exposure duration
depends on car speed
which is difficult to
determine a-priory.
Longer exposure duration
blurs the license plate
image making character
recognition difficult

169. Fluttered Shutter Camera
Raskar, Agrawal, Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turned
opaque or transparent in a rapid binary sequence

170. Short Exposure Traditional MURA Coded
Coded Exposure Photography:
Assisting Motion Deblurring using Fluttered Shutter
Raskar, Agrawal, Tumblin (Siggraph2006)
Deblurred
Results
Captured
Photos
Shutter
Result has Banding
Artifacts and some spatial
frequencies are lost
Decoded image is as
good as image of a
static scene
Image is dark
and noisy

172. Compound Lens of Dragonfly

173. TOMBO: Thin Camera (2001)
“Thin observation module by bound optics (TOMBO),”
J. Tanida, T. Kumagai, K. Yamada, S. Miyatake
Applied Optics, 2001

174. TOMBO: Thin Camera

175. ZCam (3Dvsystems),
Shuttered Light Pulse
Resolution :Resolution :
1cm for 2-7 meters1cm for 2-7 meters

176. Graphics can inserted behind and between characters

177. Cameras for HCICameras for HCI
• Frustrated total internal reflectionFrustrated total internal reflection
Han, J. Y. 2005. Low-Cost Multi-Touch
Sensing through Frustrated Total Internal
Reflection. In Proceedings of the 18th
Annual ACM Symposium on User Interface
Software and Technology

178. Converting LCD Screen = large Camera for 3D
Interactive HCI and Video Conferencing
Matthew Hirsch, Henry Holtzman
Doug Lanman, Ramesh Raskar
Siggraph Asia 2009
Class Project in CompCam 2008
SRC Winner
BiDi Screen*

179. Beyond Multi-touch: Mobile
Laptops
Mobile

180. Light Sensing Pixels in LCD
Displaywithembeddedopticalsensors
Sharp Microelectronics Optical Multi-touch Prototype

181. Design Overview
Displaywithembeddedopticalsensors
LCD,
displaying mask
Opticalsensorarray
~2.5 cm~50 cm

182. Beyond Multi-touch: Hover Interaction
• Seamless transition of
multitouch to gesture
• Thin package, LCD

183. Design Vision
Object Collocated Capture
and Display
Bare Sensor
SpatialLightModulator

184. Touch + Hover using Depth Sensing LCD Sensor

185. Overview: Sensing Depth from
Array of Virtual Cameras in
LCD

186. • 36 bit code at 0.3mm resolution36 bit code at 0.3mm resolution
• 100 fps camera at 8800 nm100 fps camera at 8800 nm
http://www.acreo.se/upload/Publications/Proceedings/OE00/00-KAURANEN.pdf

187. • Smart Barcode size : 3mm x 3mm
• Ordinary Camera: Distance 3 meter
Computational Probes:Computational Probes:
Long Distance Bar-codesLong Distance Bar-codes
Mohan, Woo,Smithwick, Hiura, Raskar
Accepted as Siggraph 2009 paper

188. MIT Media Lab Camera Culture
Bokode

189. MIT media lab camera culture
Barcodes
markers that assist machines in
understanding the real world

190. MIT media lab camera culture
Bokode:
ankit mohan, grace woo, shinsaku hiura,
quinn smithwick, ramesh raskar
camera culture group, MIT media lab
imperceptible visual tags for camera
based interaction from a distance

191. MIT Media Lab Camera Culture
Defocus
blur of
Bokode

192. MIT Media Lab Camera Culture
Image greatly magnified.
Simplified Ray Diagram

193. MIT Media Lab Camera Culture
Our Prototypes

194. MIT media lab camera culture
street-view tagging

195. Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Vicon
Motion Capture
High-speed
IR Camera
Medical Rehabilitation Athlete Analysis
Performance Capture Biomechanical Analysis

196. Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
R Raskar, H Nii, B de Decker, Y Hashimoto, J Summet, D
Moore, Y Zhao, J Westhues, P Dietz, M Inami, S Nayar, J
Barnwell, M Noland, P Bekaert, V Branzoi, E Bruns
Siggraph 2007
Prakash: Lighting-Aware Motion Capture Using
Photosensing Markers and Multiplexed Illuminators

197. Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Imperceptible Tags under clothing, tracked under ambient light
Hidden
Marker Tags
Outdoors
Unique Id
http://raskar.info/prakash

198. Camera-based HCICamera-based HCI
• Many projects hereMany projects here
– Robotics, Speechome, Spinner, Sixth SenseRobotics, Speechome, Spinner, Sixth Sense
• Sony EyeToySony EyeToy
• WiiWii
• Xbox/NatalXbox/Natal
• Microsoft SurfaceMicrosoft Surface
– Shahram Izadi (Microsoft Surface/SecondLight)Shahram Izadi (Microsoft Surface/SecondLight)
– Talk at Media Lab, Tuesday Sept 22Talk at Media Lab, Tuesday Sept 22ndnd
, 3pm, 3pm

199. 213
Computational Imaging in the Sciences
Driving Factors:
 new instruments lead to new discoveries
(e.g., Leeuwenhoek + microscopy  microbiology)
 Q: most important instrument in last century?
A: the digital computer
What is Computational Imagining?
 according to B.K. Horn:
“…imaging methods in which computation is
inherent in image formation.”
 digital processing has led to a revolution in medical
and scientific data collection
(e.g., CT, MRI, PET, remote sensing, etc.)
Slides by Doug Lanman

200. 214
Computational Imaging in the Sciences
Medical Imaging:
 transmission tomography (CT)
 reflection tomography (ultrasound)
Geophysics:
 borehole tomography
 seismic reflection surveying
Applied Physics:
 diffuse optical tomography
 diffraction tomography
 scattering and inverse scattering
Biology:
 confocal microscopy
 deconvolution microscopy
Astronomy:
 coded-aperture imaging
 interferometric imaging
Remote Sensing:
 multi-perspective panoramas
 synthetic aperture radar
Optics:
 wavefront coding
 light field photography
 holography
Slides by Doug Lanman

201. 215
What is Tomography?
Definition:
 imaging by sectioning (from Greek tomos: “a section” or “cutting”)
 creates a cross-sectional image of an object by transmission or
reflection data collected by illuminating from many directions
Parallel-beam Tomography Fan-beam Tomography
Slides by Doug Lanman

202. 216
Reconstruction: Filtered Backprojection
x
y
fy
fx
Fourier Projection-Slice Theorem:
 F-1
{Gθ(ω)} = Pθ(t)
 add slices Gθ(ω) into {u,v} at all angles θ and
inverse transform to yield g(x,y)
 add 2D backprojections Pθ(t) into {x,y} at all
angles θ
Pθ(t)
Pθ(t,s)
Gθ(ω)
g(x,y)
Slides by Doug Lanman

203. 217
Medical Applications of Tomography
bone reconstruction segmented vessels
Slides by Doug Lanman

204. 218
Biology: Confocal Microscopy
pinhole
light source
photocell
pinhole
Slides by Doug Lanman

205. 219
Confocal Microscopy Examples
Slides by Doug Lanman

206. Forerunners ..Forerunners ..
Mask
Sensor
Mask
Sensor

207. Fernald, Science [Sept 2006]
Shadow Refractive Reflective
Tools
for
Visual
Computing

208. Project AssignmentsProject Assignments
• RelightingRelighting
• Dual PhotographyDual Photography
• Virtual Optical BenchVirtual Optical Bench
• Lightfield captureLightfield capture
– Mask or LCD with programmable apertureMask or LCD with programmable aperture
• One ofOne of
– High speed imagingHigh speed imaging
– Thermal imagingThermal imaging
– 3D range sensing3D range sensing
• Final ProjectFinal Project

209. Synthetic LightingSynthetic Lighting
Paul Haeberli, Jan 1992Paul Haeberli, Jan 1992

210. Image-Based Actual Re-lightingImage-Based Actual Re-lighting
Film the background in Milan,Film the background in Milan,
Measure incoming light,Measure incoming light,
Light the actress in Los AngelesLight the actress in Los Angeles
Matte the backgroundMatte the background
Matched LA and Milan lighting.Matched LA and Milan lighting.
Debevec et al., SIGG2001

211. Dual photography from diffuse reflections:Dual photography from diffuse reflections:
Homework Assignment 2Homework Assignment 2
the camera’s view
Sen et al, Siggraph 2005Sen et al, Siggraph 2005

212. Direct Global
Shower Curtain: Diffuser

213. • Andrew Adam’s Virtual Optical BenchAndrew Adam’s Virtual Optical Bench

214. Beyond Visible SpectrumBeyond Visible Spectrum
CedipRedShift

215. GoalsGoals
• Change the rules of the gameChange the rules of the game
– Emerging optics, illumination, novel sensorsEmerging optics, illumination, novel sensors
– Exploit priors and online collectionsExploit priors and online collections
• ApplicationsApplications
– Better scene understanding/analysisBetter scene understanding/analysis
– Capture visual essenceCapture visual essence
– Superior Metadata tagging for effective sharingSuperior Metadata tagging for effective sharing
– Fuse non-visual dataFuse non-visual data
• Sensors for disabled, new art forms, crowdsourcing,Sensors for disabled, new art forms, crowdsourcing,
bridging culturesbridging cultures

216. First Assignment: Synthetic LightingFirst Assignment: Synthetic Lighting
Paul Haeberli, Jan 1992Paul Haeberli, Jan 1992

217. • FormatFormat
– 4 (3) Assignments4 (3) Assignments
• Hands on with optics,Hands on with optics,
illumination, sensors, masksillumination, sensors, masks
• Rolling schedule for overlapRolling schedule for overlap
• We have cameras, lenses,We have cameras, lenses,
electronics, projectors etcelectronics, projectors etc
• Vote on best projectVote on best project
– Mid term examMid term exam
• Test conceptsTest concepts
– 1 Final project1 Final project
• Should be a Novel and CoolShould be a Novel and Cool
• Conference quality paperConference quality paper
• Award for best projectAward for best project
– Take 1 class notesTake 1 class notes
– Lectures (and guestLectures (and guest
talks)talks)
– In-class + onlineIn-class + online
discussiondiscussion
• If you are a listenerIf you are a listener
– Participate in online discussion, digParticipate in online discussion, dig
new recent worknew recent work
– Present one short 15 minute idea orPresent one short 15 minute idea or
new worknew work
• CreditCredit
• Assignments: 40%Assignments: 40%
• Project: 30%Project: 30%
• Mid-term: 20%Mid-term: 20%
• Class participation: 10%Class participation: 10%
• Pre-reqsPre-reqs
• Helpful: Linear algebra, imageHelpful: Linear algebra, image
processing, think in 3Dprocessing, think in 3D
• We will try to keep math toWe will try to keep math to
essentials, but complex conceptsessentials, but complex concepts

218. Assignments:Assignments:
You are encouraged to program in Matlab for image analysisYou are encouraged to program in Matlab for image analysis
You may need to use C++/OpenGL/Visual programming for some hardware assignmentsYou may need to use C++/OpenGL/Visual programming for some hardware assignments
Each student is expected to prepare notes for one lectureEach student is expected to prepare notes for one lecture
These notes should be prepared and emailed to the instructor no later than the followingThese notes should be prepared and emailed to the instructor no later than the following
Monday night (midnight EST). Revisions and corrections will be exchanged by email andMonday night (midnight EST). Revisions and corrections will be exchanged by email and
after changes the notes will be posted to the website before class the following week.after changes the notes will be posted to the website before class the following week.
5 points5 points
Course mailing listCourse mailing list: Please make sure that your emailid is on the course mailing list: Please make sure that your emailid is on the course mailing list
Send email to raskar (at) media.mit.eduSend email to raskar (at) media.mit.edu
Please fill in the email/credit/dept sheetPlease fill in the email/credit/dept sheet
Office hoursOffice hours::
Email is the best way to get in touchEmail is the best way to get in touch
Ramesh:. raskar (at) media.mit.eduRamesh:. raskar (at) media.mit.edu
Ankit:Ankit: ankit (at) media.mit.eduankit (at) media.mit.edu
After class:After class:
Muddy Charles PubMuddy Charles Pub
(Walker Memorial next to tennis courts)(Walker Memorial next to tennis courts)

219. 2 Sept 18th Modern Optics and Lenses, Ray-matrix operations
3 Sept 25th Virtual Optical Bench, Lightfield Photography, Fourier Optics, Wavefront Coding
4 Oct 2nd Digital Illumination, Hadamard Coded and Multispectral Illumination
5 Oct 9th
Emerging Sensors: High speed imaging, 3D range sensors, Femto-second concepts, Front/back
illumination, Diffraction issues
6
Oct 16th
Beyond Visible Spectrum: Multispectral imaging and Thermal sensors, Fluorescent imaging, 'Audio
camera'
7 Oct 23rd
Image Reconstruction Techniques, Deconvolution, Motion and Defocus Deblurring, Tomography,
Heterodyned Photography, Compressive Sensing
8
Oct 30th
Cameras for Human Computer Interaction (HCI): 0-D and 1-D sensors, Spatio-temporal coding,
Frustrated TIR, Camera-display fusion
9
Nov 6th
Useful techniques in Scientific and Medical Imaging: CT-scans, Strobing, Endoscopes, Astronomy
and Long range imaging
10
Nov 13th Mid-term Exam, Mobile Photography, Video Blogging, Life logs and Online Photo collections
11
Nov 20th Optics and Sensing in Animal Eyes. What can we learn from successful biological vision systems?
12
Nov 27th Thanksgiving Holiday (No Class)
13
Dec 4th Final Projects

220. What is the emphasis?What is the emphasis?
• Learn fundamental techniques in imagingLearn fundamental techniques in imaging
– In class and in homeworksIn class and in homeworks
– Signal processing, Applied optics, Computer graphics and vision,Signal processing, Applied optics, Computer graphics and vision,
Electronics, Art, and Online photo collectionsElectronics, Art, and Online photo collections
– This is not a discussion classThis is not a discussion class
• Three Applications areasThree Applications areas
– PhotographyPhotography
• Think in higher dimensions 4D, 6D, 8D, thermal, range cam, lightfields,Think in higher dimensions 4D, 6D, 8D, thermal, range cam, lightfields,
applied opticsapplied optics
– Active Computer Vision (real-time)Active Computer Vision (real-time)
• HCI, Robotics, Tracking/Segmentation etcHCI, Robotics, Tracking/Segmentation etc
– Scientific ImagingScientific Imaging
• Compressive sensing, wavefront coding, tomography, deconvolution, psfCompressive sensing, wavefront coding, tomography, deconvolution, psf
– But the 3 areas are merging and use similar principlesBut the 3 areas are merging and use similar principles

221. First Homework AssignmentFirst Homework Assignment
• Take multiple photos by changing lightingTake multiple photos by changing lighting
• Mix and match color channels to relightMix and match color channels to relight
• Due Sept 25Due Sept 25thth
• Need Volunteer: taking notes for next classNeed Volunteer: taking notes for next class
– Sept 18: Sam PerliSept 18: Sam Perli
– Sept 25: ?Sept 25: ?

222. Goal and Experience
Low Level Mid Level High
Level
Hyper
realism
Raw
Angle,
spectrum
aware
Non-visual
Data, GPS
Metadata
Priors
Comprehensive
8D reflectance
field
Digital
Epsilon
Coded
Essence
CP aims to
make progress
on both axis
Camera Array
HDR, FoV Focal stack
Decompositio
n problems
Depth
Spectrum
LightFields
Human Stereo
Vision
Transient
Imaging
Virtual Object
Insertion
Relighting
Augmented
Human
Experience
Material
editing from
single photo
Scene
completion
from photos
Motion
Magnification
Phototourism

223. Capture
• Overcome Limitations of Cameras
• Capture Richer Data
Multispectral
• New Classes of Visual Signals
Lightfields, Depth, Direct/Global, Fg/Bg separation
Hyperrealistic Synthesis
• Post-capture Control
• Impossible Photos
• Exploit Scientific Imaging
Computational Photography
http://raskar.info/photo/

225. Blind CameraBlind Camera
Sascha Pohflepp,Sascha Pohflepp,
U of the Art, Berlin, 2006U of the Art, Berlin, 2006

226. ENDEND