This document discusses digital hologram image processing techniques. It begins with an introduction to digital holography and why image processing is needed to extract 3D information from digital holograms. Key topics covered include reconstructing digital holograms, focusing and segmentation techniques, and removing unwanted twin images and other artifacts from reconstructions. The document provides an overview of recording digital holograms and sources of error, as well as outlining various image processing approaches that can be applied.
16. Why digital holography? Using digital holography we can record a scene in a complex valued data structure which retains some of the scene's 3D information. A standard image obtained with a camera records a 2D focused image of the scene from one perspective. Contact: conormce@cs.nuim.ie
18. Why digital holography? Using digital holography we can record a scene in a complex valued data structure which retains some of the scene's 3D information. A standard image obtained with a camera records a 2D focused image of the scene from one perspective. Reconstructions Why do we need image processing? However reconstructing a digital hologram returns a 2D image of the scene at a specific depth (300mm from the camera) from an individual perspective (along the optical axis). Algorithms and processing techniques need to be developed to extract the 3D information from digital holograms by processing multiple (volumes of) reconstructions. Image Processing Depth Map Contact: conormce@cs.nuim.ie
19. Why not standard 2D image processing? 2D Standard 2D image processing techniques can be applied to individual digital holographic reconstructions with varying success. 2D Image Processing DHIP Digital Holographic Image Processing Holograms have a shallow depth-of-field, which means that we need to multiple images (reconstructions) over a range of depths and then process this volume of reconstructions to get accurate information back out. Reconstructions Contact: conormce@cs.nuim.ie
28. Black box - reconstructing a digital hologram Digital Hologram Digital Reconstruction Reconstruct Distance d Contact: conormce@cs.nuim.ie
29. Black box - Reconstructing a digital hologram Digital Hologram Digital Reconstruction d1 Reconstruct d2 d3 d4 d5 d6 Set of distances {d1,d2,d3,d4,d5,d6} Contact: conormce@cs.nuim.ie
30. Shallow depth-of-field Reconstructions from digital holograms have a shallow depth of field, sometimes as small as 1mm. This means that processing an individual reconstruction is rarely a good idea. 188mm 178mm Contact: conormce@cs.nuim.ie
31. Shallow depth-of-field Reconstructions from digital holograms have a shallow depth of field, sometimes as small as 1mm. This means that processing an individual reconstruction is rarely a good idea. 188mm 178mm Contact: conormce@cs.nuim.ie
35. What can we do with image processing? Extract shape information Background Segmentation Object Segmentation Create Focused Images DHIP DHIP DHIP DHIP Contact: conormce@cs.nuim.ie
57. Recording a hologram Recording a hologram is (generally) achieved through illuminating an object with a coherent light source and recording the interference between this beam and a reference beam. It is analogous to recording a photograph. Contact: conormce@cs.nuim.ie
58. Recording a hologram Recording a hologram is (generally) achieved through illuminating an object with a coherent light source and recording the interference between this beam and a reference beam. It is analogous to recording a photograph. Photography Sun Lens Photo Film Contact: conormce@cs.nuim.ie
59. Recording a hologram Recording a hologram is (generally) achieved through illuminating an object with a coherent light source and recording the interference between this beam and a reference beam. It is analogous to recording a photograph. Photography Sun Lens Photo Film Contact: conormce@cs.nuim.ie
60. Recording a hologram Recording a hologram is (generally) achieved through illuminating an object with a coherent light source and recording the interference between this beam and a reference beam. It is analogous to recording a photograph. Photography Sun Lens Photo Film Contact: conormce@cs.nuim.ie
61. Recording a hologram Recording a hologram is (generally) achieved through illuminating an object with a coherent light source and recording the interference between this beam and a reference beam. It is analogous to recording a photograph. Holography Laser Contact: conormce@cs.nuim.ie
62. Recording a hologram Recording a hologram is (generally) achieved through illuminating an object with a coherent light source and recording the interference between this beam and a reference beam. It is analogous to recording a photograph. Holography Laser Contact: conormce@cs.nuim.ie
63. Recording a hologram Recording a hologram is (generally) achieved through illuminating an object with a coherent light source and recording the interference between this beam and a reference beam. It is analogous to recording a photograph. Holography Laser Contact: conormce@cs.nuim.ie
64. Recording a hologram Recording a hologram is (generally) achieved through illuminating an object with a coherent light source and recording the interference between this beam and a reference beam. It is analogous to recording a photograph. Holography Laser Object Beam Holo Film Contact: conormce@cs.nuim.ie
65. Recording a hologram Recording a hologram is (generally) achieved through illuminating an object with a coherent light source and recording the interference between this beam and a reference beam. It is analogous to recording a photograph. Holography Laser Object Beam Holo Film Contact: conormce@cs.nuim.ie
66. Reference Beam Recording a hologram Recording a hologram is (generally) achieved through illuminating an object with a coherent light source and recording the interference between this beam and a reference beam. It is analogous to recording a photograph. Holography Laser Object Beam Holo Film Contact: conormce@cs.nuim.ie
107. Digital hologram recording options There are (at least) three questions that need to be answered when choosing the setup you want, with the third question the toughest to answer! Contact: conormce@cs.nuim.ie
108. Digital hologram recording options There are (at least) three questions that need to be answered when choosing the setup you want, with the third question the toughest to answer! 1) Is your object large or small? Contact: conormce@cs.nuim.ie
109. Digital hologram recording options There are (at least) three questions that need to be answered when choosing the setup you want, with the third question the toughest to answer! 1) Is your object large or small? This will allow you to choose either a: Macroscopic setup Microscopic setup Contact: conormce@cs.nuim.ie
110. Digital hologram recording options There are (at least) three questions that need to be answered when choosing the setup you want, with the third question the toughest to answer! 1) Is your object large or small? This will allow you to choose either a: 2) Does your object let light through or reflect light? Macroscopic setup Microscopic setup Contact: conormce@cs.nuim.ie
111. Digital hologram recording options There are (at least) three questions that need to be answered when choosing the setup you want, with the third question the toughest to answer! 1) Is your object large or small? This will allow you to choose either a: 2) Does your object let light through or reflect light? This will allow you to make your setup: Macroscopic setup Microscopic setup Reflective mode Transmissive mode Contact: conormce@cs.nuim.ie
112. Digital hologram recording options There are (at least) three questions that need to be answered when choosing the setup you want, with the third question the toughest to answer! 1) Is your object large or small? This will allow you to choose either a: 2) Does your object let light through or reflect light? This will allow you to make your setup: 3) What are recording architecture do you want? Macroscopic setup Microscopic setup Reflective mode Transmissive mode Contact: conormce@cs.nuim.ie
113. Digital hologram recording options There are (at least) three questions that need to be answered when choosing the setup you want, with the third question the toughest to answer! 1) Is your object large or small? This will allow you to choose either a: 2) Does your object let light through or reflect light? This will allow you to make your setup: 3) What are recording architecture do you want? The options are: Macroscopic setup Microscopic setup Reflective mode Transmissive mode In-line Off-axis Contact: conormce@cs.nuim.ie
114. Digital hologram recording options There are (at least) three questions that need to be answered when choosing the setup you want, with the third question the toughest to answer! 1) Is your object large or small? This will allow you to choose either a: 2) Does your object let light through or reflect light? This will allow you to make your setup: 3) What are recording architecture do you want? The options are: I will try and cover some of the differences between these two options but that is a topic in itself. Macroscopic setup Microscopic setup Reflective mode Transmissive mode In-line Off-axis Contact: conormce@cs.nuim.ie
115. In-Line setup This form of holography gets its name from the fact that the reference beam and the object beam are centred on the optical axis. They both are at the same angle in relation to the CCD. Contact: conormce@cs.nuim.ie
116. Off-axis setup In off-axis holography the object beam and reference beam are at a different angle in relation to the CCD. Contact: conormce@cs.nuim.ie
117. Digital hologram recording There are numerous ways of implementing a digital hologram recording setup. There are also numerous advantages and disadvantages of using an in-line or off-axis setup. This topic in itself would take a whole talk. For the purposes of this talk all digital holograms were recorded with in-line digital holography. Only Macroscopic Reflective In-line Digital Holograms in this talk Contact: conormce@cs.nuim.ie
140. Unwanted terms in a digital hologram The interference pattern recorded is described by: Contact: conormce@cs.nuim.ie
141. Unwanted terms in a digital hologram The interference pattern recorded is described by: Contact: conormce@cs.nuim.ie
142. Unwanted terms in a digital hologram The interference pattern recorded is described by: Contact: conormce@cs.nuim.ie
143. Unwanted terms in a digital hologram The interference pattern recorded is described by: which contains three terms Contact: conormce@cs.nuim.ie
144. Unwanted terms in a digital hologram The interference pattern recorded is described by: which contains three terms The DC-term Contact: conormce@cs.nuim.ie
145. Unwanted terms in a digital hologram The interference pattern recorded is described by: which contains three terms The DC-term The virtual image Contact: conormce@cs.nuim.ie
146. Unwanted terms in a digital hologram The interference pattern recorded is described by: which contains three terms The DC-term The virtual image The real image Contact: conormce@cs.nuim.ie
147. Unwanted terms in a digital hologram The interference pattern recorded is described by: which contains three terms The DC-term The virtual image The real image The DC-term is a term which corrupts any reconstructions of the hologram. It can be removed through spatially filtering the Fourier transform of the hologram. Contact: conormce@cs.nuim.ie
148. Unwanted terms in a digital hologram The interference pattern recorded is described by: which contains three terms The DC-term The virtual image The real image The DC-term is a term which corrupts any reconstructions of the hologram. It can be removed through spatially filtering the Fourier transform of the hologram. The virtual image and real image are known as the twin-images and are viewed as noise in reconstructions of digital holograms. Contact: conormce@cs.nuim.ie
162. DC-term suppression Taking a single hologram prior to any processing, we reconstruct this to demonstrate the corruptive effect of the DC-term 365 mm Reconstruction plane Hologram Reconstruct Contact: conormce@cs.nuim.ie
163. DC-term suppression Taking a single hologram prior to any processing, we reconstruct this to demonstrate the corruptive effect of the DC-term 365 mm Reconstruction plane Hologram Reconstruct There are many methods for suppressing the DC-term, our twin-image removal algorithm takes as input a DC-term suppressed hologram. For our experiments we apply a high-pass filter in the Fourier domain to suppress the DC-term. Contact: conormce@cs.nuim.ie
185. Twin-image With the DC-term suppressed we can now view the twin-images. DC-Free Hologram Virtual Image Real Image 355 mm -355 mm Contact: conormce@cs.nuim.ie
186. Twin-image removal There are different approaches for suppressing or removing the unwanted twin. These are dependent on the architecture. We have developed an approach based on image processing for In-Line digital holography. We will discuss this in detail later in the talk Contact: conormce@cs.nuim.ie or tomn@cs.nuim.ie
200. Phase-shifting digital holography Phase shifting digital holography is a technique which calculates in-line digital holograms which are free of the dc-term and twin-image. This is achieved through recording multiple different holograms where the phase of the reference beam has been modified. By modifying the phase of the reference beam the constant terms in the resulting holograms are the objects amplitude and phase. This allows for the extraction of the object wavefront from multiple holograms. Contact: conormce@cs.nuim.ie
201. Phase-shifting digital holography Notice that the intensity of the virtual image is much lower than the real image. Phase shifting has suppressed the twin. Virtual Image Real Image Contact: conormce@cs.nuim.ie
224. Reconstructing a digital hologram Reconstructing a hologram is achieved through illuminating the recorded object by the original source, much like in photography. Contact: conormce@cs.nuim.ie
225. Reconstructing a digital hologram Reconstructing a hologram is achieved through illuminating the recorded object by the original source, much like in photography. Photography Sun Viewer PhotoFilm Contact: conormce@cs.nuim.ie
226. Reconstructing a digital hologram Reconstructing a hologram is achieved through illuminating the recorded object by the original source, much like in photography. Holography Laser Viewer Holo Film Contact: conormce@cs.nuim.ie
227.
228. The eye can focus on different parts of the scene.Holography Photography Sun Laser Viewer Viewer Holo Film PhotoFilm Contact: conormce@cs.nuim.ie
242. Reconstructing a digital hologram Reconstructing a hologram is achieved through illuminating the recorded object by a reference beam. In digital holography, numerical reconstruction is achieved through simulating the illumination of the hologram by the reference beam using discrete Fresnel approximations. Contact: conormce@cs.nuim.ie
243. Reconstructing a digital hologram Reconstructing a hologram is achieved through illuminating the recorded object by a reference beam. In digital holography, numerical reconstruction is achieved through simulating the illumination of the hologram by the reference beam using discrete Fresnel approximations. In this work we employ two approximations Propagation transfer function Discrete Fresnel approximation. Contact: conormce@cs.nuim.ie
244. The propagation transfer function The propagation transfer function illuminates the digital hologram with what can be thought of as a planar reference beam. Contact: conormce@cs.nuim.ie
245. The propagation transfer function The propagation transfer function illuminates the digital hologram with what can be thought of as a planar reference beam. In this Fresnel approximation the pixel size at the reconstruction plane is constant for all reconstruction distances. Contact: conormce@cs.nuim.ie
246. The propagation transfer function The propagation transfer function illuminates the digital hologram with what can be thought of as a planar reference beam. In this Fresnel approximation the pixel size at the reconstruction plane is constant for all reconstruction distances. y x z Hologram Plane Image Plane Contact: conormce@cs.nuim.ie
247. The propagation transfer function The propagation transfer function illuminates the digital hologram with what can be thought of as a planar reference beam. In this Fresnel approximation the pixel size at the reconstruction plane is constant for all reconstruction distances. This means that field of view is constant irrespective of the propagation distance. Contact: conormce@cs.nuim.ie
248. The propagation transfer function The propagation transfer function illuminates the digital hologram with what can be thought of as a planar reference beam. In this Fresnel approximation the pixel size at the reconstruction plane is constant for all reconstruction distances. This means that field of view is constant irrespective of the propagation distance. This makes the function reciprocal but also means that if the object recorded is larger than the CCD, to ensure that there is no wrapping in the reconstruction plane sufficient padding needs to be applied at the hologram plane. Contact: conormce@cs.nuim.ie
249. The propagation transfer function The propagation transfer function illuminates the digital hologram with what can be thought of as a planar reference beam. In this Fresnel approximation the pixel size at the reconstruction plane is constant for all reconstruction distances. This means that field of view is constant irrespective of the propagation distance. This makes the function reciprocal but also means that if the object recorded is larger than the CCD, to ensure that there is no wrapping in the reconstruction plane sufficient padding needs to be applied at the hologram plane. This padding operation can increase computation time. Contact: conormce@cs.nuim.ie
250. The propagation transfer function Reconstruction plane Hologram plane 355 mm 2048 samples 2048 samples propagation transfer function Contact: conormce@cs.nuim.ie
251. The propagation transfer function Reconstruction plane Hologram plane 355 mm 2048 samples 2048 samples propagation transfer function Aliasing (wrapping of object signal) Contact: conormce@cs.nuim.ie
252. The propagation transfer function Reconstruction plane Hologram plane 355 mm 2048 samples 2048 samples propagation transfer function Padding of the hologram plane is required to ensure that the reconstruction is free of wrapping and aliasing. Contact: conormce@cs.nuim.ie
253. The propagation transfer function Reconstruction plane Hologram plane 355 mm 2048 samples 2048 samples propagation transfer function 4096 samples 4096 samples Contact: conormce@cs.nuim.ie
254. Discrete Fresnel transform The discrete Fresnel transform illuminates the digital hologram with what can be thought of as a spherical reference beam. Contact: conormce@cs.nuim.ie
255. Discrete Fresnel transform The discrete Fresnel transform illuminates the digital hologram with what can be thought of as a spherical reference beam. In this Fresnel approximation the pixel size at the reconstruction plane is a function of (primarily) the distance. Contact: conormce@cs.nuim.ie
256. y x z Discrete Fresnel transform The discrete Fresnel transform illuminates the digital hologram with what can be thought of as a spherical reference beam. In this Fresnel approximation the pixel size at the reconstruction plane is a function of (primarily) the distance. Hologram Plane Image Plane Contact: conormce@cs.nuim.ie
257. Discrete Fresnel transform The discrete Fresnel transform illuminates the digital hologram with what can be thought of as a spherical reference beam. In this Fresnel approximation the pixel size at the reconstruction plane is a function of (primarily) the distance. This means that as the distance from the digital hologram increases the field of view increases although the number of samples used in the reconstruction process stays the same. Contact: conormce@cs.nuim.ie
258. Discrete Fresnel transform The discrete Fresnel transform illuminates the digital hologram with what can be thought of as a spherical reference beam. In this Fresnel approximation the pixel size at the reconstruction plane is a function of (primarily) the distance. This means that as the distance from the digital hologram increases the field of view increases although the number of samples used in the reconstruction process stays the same. This also means that the function is not reciprocal. Contact: conormce@cs.nuim.ie
259. Discrete Fresnel transform The discrete Fresnel transform illuminates the digital hologram with what can be thought of as a spherical reference beam. In this Fresnel approximation the pixel size at the reconstruction plane is a function of (primarily) the distance. This means that as the distance from the digital hologram increases the field of view increases although the number of samples used in the reconstruction process stays the same. This also means that the function is not reciprocal. This allows us to reconstruct objects larger than the CCD with a small number of samples which saves computational time. Contact: conormce@cs.nuim.ie
260. Discrete Fresnel transform 365 mm Reconstruction plane Hologram plane discrete Fresnel transform 2048 samples 2048 samples With the discrete Fresnel transform, no padding is required to view this object. Contact: conormce@cs.nuim.ie
274. How do we reconstruct a perspective We select a window size from within the hologram. Win Size Contact: conormce@cs.nuim.ie
275. How do we reconstruct a perspective We select a window size from within the hologram. There is a trade-off between window size and visual quality. Win Size Contact: conormce@cs.nuim.ie
276. How do we reconstruct a perspective We select a window size from within the hologram. There is a trade-off between window size and visual quality. We then move the window from the centre of the hologram window. Offset Contact: conormce@cs.nuim.ie
277. What perspective are we reconstructing Nx y Nx’ Ny ay Ny’ x ax z d and Contact: conormce@cs.nuim.ie
278. Perspectives, what we want in the future Nx y Nx’ Ny ay Ny’ x ax z d We want to move the object closer to the camera We want the size of the camera in mm to stay the same but the pixel size to decrease. Both of these lead to an increase in the range of angles. Contact: conormce@cs.nuim.ie
292. Speckle reduction All coherent imaging systems contain speckle, this is because speckle is created when coherent light illuminates an optically rough surface. The size of speckle increases with the reconstruction distance and exhibits itself as noise in the reconstruction. This is why it is a problem in macroscopic digital holography. The large distance the object needs to be placed away from the camera gives speckle time to evolve and become a corruptive noise in the reconstruction. It is a multiplicative source of noise so standard image processing noise reduction approaches are not always the best option. We apply speckle reduction to reduce noise and improve our image processing algorithms. Contact: conormce@cs.nuim.ie
293. Speckle reduction Before Speckle reduction After Speckle reduction Contact: conormce@cs.nuim.ie
294. Speckle reduction Before Speckle reduction After Speckle reduction Contact: conormce@cs.nuim.ie
310. Focus Detection Focus detection is based on the application of a focus measure to a set of images or to a region within a set of images. Focus measures are functions which attempt to determine the relative level of focus of sets of images, or regions within sets of images. The accepted image property maximized by these functions is the high spatial frequency energy of the image. Contact: conormce@cs.nuim.ie
311. High spatial frequency High spatial frequency in images is equivalent to edges, so the more defined an edge is in an image the higher the spatial frequency. Contact: conormce@cs.nuim.ie
312. High spatial frequency The most accurate way to obtain the spatial frequencies in an image is to use the Fourier transform. The high spatial frequency energy in the Fourier transform is contained in the outer parts of the Fourier transform. Fourier Transform Contact: conormce@cs.nuim.ie
313. High spatial frequency The most accurate way to obtain the spatial frequencies in an image is to use the Fourier transform. The high spatial frequency energy in the Fourier transform is contained in the outer parts of the Fourier transform. Low Spatial Frequency energy Fourier Transform Contact: conormce@cs.nuim.ie
314. High spatial frequency The most accurate way to obtain the spatial frequencies in an image is to use the Fourier transform. The high spatial frequency energy in the Fourier transform is contained in the outer parts of the Fourier transform. High Spatial Frequency energy Fourier Transform Contact: conormce@cs.nuim.ie
315. High spatial frequency So to view the high spatial frequency energy of an image we “block” out the centre pixels from the Fourier transform Fourier Transform X Contact: conormce@cs.nuim.ie
316. High spatial frequency We then inverse Fourier transform the result and we get a rough edge detected image. Fourier Transform Inverse Fourier Transform X Contact: conormce@cs.nuim.ie
317. High spatial frequency We then inverse Fourier transform the result and we get a rough edge detected image. Fourier Transform Inverse Fourier Transform X Contact: conormce@cs.nuim.ie
318. Variance as a focus measure One function which has been shown to be both a sound focus measure and successfully applicable to reconstructions from digital holograms is variance. Image 2 Image 4 Image 6 Image 7 Image 10 variance Contact: conormce@cs.nuim.ie Image Number
320. Focus Detection and digital holography We can also reduce the size of the block being focused to detect the focus of smaller image regions. 2 1 4 3 Contact: conormce@cs.nuim.ie
330. Autofocus - Fibonacci search The Fibonacci search is a modification to the binary search which uses the golden ratio and Fibonacci numbers to speed up the search. Contact: conormce@cs.nuim.ie
331. Autofocus - Fibonacci search The Fibonacci search is a modification to the binary search which uses the golden ratio and Fibonacci numbers to speed up the search. We use the Fibonacci search and variance, as a focus measure, to create an autofocus algorithm for digital holography. It takes a hologram and a start and end search depth in as input. Contact: conormce@cs.nuim.ie
332. Autofocus - Fibonacci search The Fibonacci search is a modification to the binary search which uses the golden ratio and Fibonacci numbers to speed up the search. We use the Fibonacci search and variance, as a focus measure, to create an autofocus algorithm for digital holography. It takes a hologram and a start and end search depth in as input. In the first iteration it calculates two depths, reconstructs the hologram and calculates variance on the intensity of the reconstructions. The reconstruction with the highest variance is taken as the best depth estimate. Contact: conormce@cs.nuim.ie
333. Autofocus - Fibonacci search The Fibonacci search is a modification to the binary search which uses the golden ratio and Fibonacci numbers to speed up the search. We use the Fibonacci search and variance, as a focus measure, to create an autofocus algorithm for digital holography. It takes a hologram and a start and end search depth in as input. In the first iteration it calculates two depths, reconstructs the hologram and calculates variance on the intensity of the reconstructions. The reconstruction with the highest variance is taken as the best depth estimate. At the next iteration a new depth is selected and the focus measure is calculated on the intensity of its reconstruction, it is compared to the previous best focus value and if it is higher, this becomes the new best depth estimate. Contact: conormce@cs.nuim.ie
334. Autofocus - Fibonacci search The Fibonacci search is a modification to the binary search which uses the golden ratio and Fibonacci numbers to speed up the search. We use the Fibonacci search and variance, as a focus measure, to create an autofocus algorithm for digital holography. It takes a hologram and a start and end search depth in as input. In the first iteration it calculates two depths, reconstructs the hologram and calculates variance on the intensity of the reconstructions. The reconstruction with the highest variance is taken as the best depth estimate. At the next iteration a new depth is selected and the focus measure is calculated on the intensity of its reconstruction, it is compared to the previous best focus value and if it is higher, this becomes the new best depth estimate. This process is continued until the termination condition is met. Contact: conormce@cs.nuim.ie
335. Fibonacci search – termination condition Typically the Fibonacci search requires a relative definition of accuracy. This is the termination condition as it determines the maximum number of iterations in the Fibonacci search. We have optimized this termination condition for digital holography to take account of the depth-of-focus of the reconstructions. Contact: conormce@cs.nuim.ie
336. Fibonacci search – termination condition Typically the Fibonacci search requires a relative definition of accuracy. This is the termination condition as it determines the maximum number of iterations in the Fibonacci search. We have optimized this termination condition for digital holography to take account of the depth-of-focus of the reconstructions. Once the depth-of-focus of the two reconstructions being compared by the Fibonacci search overlaps, we stop the search and return the estimated depth. This is because when the depth-of-focus of two reconstructions overlaps we can no longer accurately determine focus. Contact: conormce@cs.nuim.ie
337. Fibonacci search – termination condition We use the discrete Fresnel transform in our Fibonacci search this is to allow us to compute reconstructions without aliasing faster than if we were using the propagation transfer function. Contact: conormce@cs.nuim.ie
338. Fibonacci search – termination condition We use the discrete Fresnel transform in our Fibonacci search this is to allow us to compute reconstructions without aliasing faster than if we were using the propagation transfer function. We can calculate the depth-of-focus range [zi,zj] of a reconstruction at distance d using this reconstruction method with: where ΔX is the size of the CCD Contact: conormce@cs.nuim.ie
339. Fibonacci search – termination condition We use the discrete Fresnel transform in our Fibonacci search this is to allow us to compute reconstructions without aliasing faster than if we were using the propagation transfer function. We can calculate the depth-of-focus range [zi,zj] of a reconstruction at distance d using this reconstruction method with: where ΔX is the size of the CCD The depth-of-focus for reconstructions ranging from 170mm to 1000mm is displayed in this plot. Contact: conormce@cs.nuim.ie
342. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Focus value for that distance Contact: conormce@cs.nuim.ie
343. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Red means current best estimate. Contact: conormce@cs.nuim.ie
351. Autofocus - Fibonacci search example In this example we compare our optimized Fibonacci search with a fixed step size search with a step size of 1mm over a search range between [170mm,1000mm]. We took a Two bolts hologram and computed the fixed step size search and the Fibonacci search. Displayed in the plot is the first 8 estimates output from Fibonacci (both correct and incorrect) overlayed on the focus plot from the fixed step size search. Contact: conormce@cs.nuim.ie
362. What is Depth-from focus? Depth-From-Focus is an image processing technique which is used to determine the depth of a scene or a region within a scene through processing images taken at different focal depths. Why is this applicable to digital holography? Digital Holograms can be numerically reconstructed at an arbitrary depth. These numerical reconstructions are each at a different focal plane, which make them a good input to a Depth-From-Focus algorithm. We can then create depth maps of the scene, segment the scene and create extended focused images of the scene. Contact: conormce@cs.nuim.ie
363. What is Depth-from focus? Depth-From-Focus is an image processing technique which is used to determine the depth of a scene or a region within a scene through processing images taken at different focal depths. Why is this applicable to digital holography? Digital Holograms can be numerically reconstructed at an arbitrary depth. These numerical reconstructions are each at a different focal plane, which make them a good input to a Depth-From-Focus algorithm. We can then create depth maps of the scene, segment the scene and create extended focused images of the scene. Contact: conormce@cs.nuim.ie
364. What is Depth-from focus? Depth-From-Focus is an image processing technique which is used to determine the depth of a scene or a region within a scene through processing images taken at different focal depths. Why is this applicable to digital holography? Digital Holograms can be numerically reconstructed at an arbitrary depth. These numerical reconstructions are each at a different focal plane, which make them a good input to a Depth-From-Focus algorithm. We can then create depth maps of the scene, segment the scene and create extended focused images of the scene. Contact: conormce@cs.nuim.ie
365. How to compute a depth map The depth-from-focus process is a three stage process. In the first stage we create a volume of reconstructions over a range of depths. We then apply speckle reduction Stage 1: At each depth, reconstruct. Focus depthzmax Digital hologram Focus depthzmin Iz(k,l) Contact: conormce@cs.nuim.ie
366. How to compute a depth map In the second stage we calculate what we call focus maps. These are created by computing a focus measure on each overlapping n x n pixel block in a reconstruction. Stage 1: At each depth, reconstruct and apply speckle reduction. Stage 2: Calculate focus on each overlapping n x n pixel block of the reconstructions. Focus depthzmax Digital hologram Focus depthzmin Vz(k,l) Iz(k,l) Contact: conormce@cs.nuim.ie
367. n n Focus Maps To compute a focus map, we first take a reconstruction and a block size of [n x n]. We then calculate our focus measure on the first block in the top left corner of the reconstruction We then process every block in the reconstruction by raster scanning the reconstruction and processing every block with our focus measure. We store the output value from each block in its corresponding position in a focus map. Contact: conormce@cs.nuim.ie
368. How to compute a depth map Once we have our focus maps we can continue to our third and final stage which is the computation of the maximum focus map and the depth map. Stage 1: At each depth, reconstruct and apply speckle reduction. Stage 3: Calculate the maximum focus map and depth map. Stage 2: Calculate focus on each overlapping n x n pixel block of the reconstructions. Focus depthzmax Max values For each (k,l) find maximum over z Vmax(k,l) Digital hologram Depths of max values Focus depthzmin Vz(k,l) Iz(k,l) D(k,l) Contact: conormce@cs.nuim.ie
371. and the distance between reconstructionsContact: conormce@cs.nuim.ie
372. Block Size To determine the depth of a block in an image using a focus measure there needs to be enough object information in the block. Smaller block sizes: finer object features but high error in the estimate of the general shape. Larger block sizes: low error but fine object features lost. Object Contact: conormce@cs.nuim.ie 7x7 43x43 81x81 121x121 151x151
373. Speckle reduction Before Speckle reduction After Speckle reduction Contact: conormce@cs.nuim.ie
374. Speckle reduction Before Speckle reduction After Speckle reduction Contact: conormce@cs.nuim.ie
375. Distance between reconstructions By changing the distance between reconstructions we affect the quality of the depth maps. The smaller the distance the more features we can detect but at the expense of speed. Contact: conormce@cs.nuim.ie or tomn@cs.nuim.ie
376. Distance between reconstructions By changing the distance between reconstructions we affect the quality of the depth maps. The smaller the distance the more features we can detect but at the expense of speed. Contact: conormce@cs.nuim.ie or tomn@cs.nuim.ie
377. Depth map examples Depth map of some hairs recorded by digital holography Reconstruction Depth Map Contact: conormce@cs.nuim.ie
378. Depth map examples Depth map of a lego block recorded by digital holography Reconstruction Depth Map Contact: conormce@cs.nuim.ie
393. What is background segmentation? Background segmentation is the partitioning of a scene into object and background. We have observed that background regions have a lower focus value than object blocks for all depths. We can use this to threshold our maximum focus map into object and background Threshold Line Contact: conormce@cs.nuim.ie
394. Background segmentation Background segmentation is a simple process, where we take the maximum focus map threshold it and then binarise the result. This is achieved by taking a value A, where everything above A in the maximum focus map is assigned the value 1 and everything below is assigned the value 0. Threshold, binarise and erode Maximum focus map Segmentation mask Contact: conormce@cs.nuim.ie
395. Thresholding The threshold is manually selected and the effects are shown below. A threshold that is too high does not segment all of the background. A threshold that is too low does not segment all of the object. Reconstruction Segmentation mask Contact: conormce@cs.nuim.ie
405. What is depth segmentation Depth segmentation is the partitioning of a scene into individual objects after the background has been segmented. Why do we want to perform depth segmentation? Again using the example of object recognition, if a scene is complex (containing multiple occluding objects), using depth segmentation we can partition the scene into independent objects for analysis. Reconstruction Depth Map Depth Maps Histogram 1 2 2 2 1 1 Contact: conormce@cs.nuim.ie
406. How do we segment based on depth Select N largest modes from histogram Contact: conormce@cs.nuim.ie
407. How do we segment based on depth Select N largest modes from histogram 1 2 Contact: conormce@cs.nuim.ie
408. How do we segment based on depth Select N largest modes from histogram Label all pixels belonging to each mode with the mode index (i.e 1 or 2) 1 2 Contact: conormce@cs.nuim.ie
409. How do we segment based on depth Extract unlabelled object pixels Select N largest modes from histogram Label all pixels belonging to each mode with the mode index (i.e 1 or 2) 1 2 Contact: conormce@cs.nuim.ie
410. How do we segment based on depth Extract unlabelled object pixels Select N largest modes from histogram + Label all pixels belonging to each mode with the mode index (i.e 1 or 2) 1 2 Contact: conormce@cs.nuim.ie
411. How do we segment based on depth Segmentation Image Extract unlabelled object pixels Select N largest modes from histogram + Label all pixels belonging to each mode with the mode index (i.e 1 or 2) 1 2 Contact: conormce@cs.nuim.ie
412. Processing the segmentation image We have one final step which we use to improve the depth segmentation images. We calculate the area of each of the distinct regions in the segmentation image and remove ones with a relatively small area. We then reassign these pixels to the object with the nearest boundary. Remove small objects from the segmentation image and resassign these pixels to the nearest object Contact: conormce@cs.nuim.ie
422. What is an Extended Focused image? A disadvantage of holographic reconstructions is the limited depth of field. For a reconstruction at depth d only object points that are located at distance d from the camera are in focus. Why do we want to create an extended focused image? This means that reconstructions can contain large blurry regions. Using our depth maps and the volume of reconstructions used to create them we can create an extended focused image. = + Contact: conormce@cs.nuim.ie Volume of Reconstructions Extended Focused Image Depth Map
435. Twin-image removal algorithm Our twin-image removal algorithm is a three stage process which takes a DC-term suppressed hologram as input. 1 2 3 Twin-Image Reconstruction Twin-Image Segmentation Propagation Contact: conormce@cs.nuim.ie
436. Stage 1: Twin-image Reconstruction The first step of this stage is to determine the focal plane. We have developed a novel fast autofocus algorithm for digital holography. a Automatically detect focal plane Fibonascci focus search d Contact: conormce@cs.nuim.ie
437. Stage 1: Twin-image Reconstruction a Automatically detect focal plane Fibonascci focus search d Once we have the correct depth we need to calculate the amount of padding required to ensure no aliasing of the wanted object signal. Reconstruct Twin-Image b Twin-Image Hologram Plane Reconstruction Fresnel Transform Fresnel Transform -d d Contact: conormce@cs.nuim.ie
438. Twin-image Reconstruction We need to pad the hologram to ensure that there is no aliasing. If there is aliasing it means we could remove some of the wanted object in the segmentation process. d mm Twin-Image plane Hologram plane propagation transfer function NH samples NH samples 2048 samples Contact: conormce@cs.nuim.ie
439. Stage 2: Twin-image Segmentation The first step in Stage 2 is to calculate a focus map from the intensity of a holograms reconstruction, which will then be used to create a segmentation mask. a Calculate Segmentation Mask Twin-Image Intensity Segmentation Mask Focus Map Calculate Focus Map Threshold and Binarize Contact: conormce@cs.nuim.ie
443. Stage 2: Twin-image Segmentation Now that we have our reconstruction and our segmentation mask, we apply elementwise multiplication to calculate a wavefront free of the twin-image. a Calculate Segmentation Mask Twin-Image Intensity Segmentation Mask Focus Map Calculate Focus Map Threshold and Binarize b Segment Twin-Image Segmented Twin-Image Segmentation Mask Twin-Image X = Contact: conormce@cs.nuim.ie
444. Stage 3: Propagation In the third stage we then propagate the twin-image free wavefront back to the hologram plane and remove the centre pixels which relate to the original size of the hologram before padding. Segmented Twin-Image Hologram Plane Fresnel Transform d H(x,y) We now have a hologram (H(x,y)) which is free of the twin-image. Contact: conormce@cs.nuim.ie
445. Implementation We have implemented our algorithm using our framework for digital hologram processing on programmable graphics hardware which has been shown to render images from DHs far more efficiently than traditional CPU-based methods. This stage uses the discrete Fresnel transform and uses holograms of size 2048 x2048 Contact: conormce@cs.nuim.ie
459. d Correct object placement To fulfill the sampling theorem an object of size sizeΔO must be placed at least a distance d from the CCD Object plane CCD θ ΔO ΔX d1 d2 Contact: conormce@cs.nuim.ie
460. d Correct object placement To fulfill the sampling theorem an object of size sizeΔO must be placed at least a distance d from the CCD This distance d can be calculated with: where δx is the spatial resolution of a pixel on the CCD. Object plane CCD θ ΔO ΔX d1 d2 Contact: conormce@cs.nuim.ie
461. d Correct object placement To fulfill the sampling theorem an object of size sizeΔO must be placed at least a distance d from the CCD This distance d can be calculated with: where δx is the spatial resolution of a pixel on the CCD. We can also express the maximum interference angle (theta) as: Object plane CCD θ ΔO ΔX d1 d2 Contact: conormce@cs.nuim.ie
462. d Calculating padding If you were to propagate the object signal from the object plane to the Twin-image plane, the object signals spatial extent would have increased significantly. We have worked this out as: Twin-image plane Object plane Δq CCD θ θ ΔT ΔO ΔX d1 d2 θ Δq Contact: conormce@cs.nuim.ie
463. d Calculating padding To calculate the spatial extent of the object signal (ΔT) we need to be able to calculate Δq. Twin-image plane Object plane Δq CCD θ θ ΔT ΔO ΔX d1 d2 θ Δq Contact: conormce@cs.nuim.ie
464. d Calculating padding To calculate the spatial extent of the object signal (ΔT) we need to be able to calculate Δq. It can be seen from the diagram below that Δq can be represented as Twin-image plane Object plane Δq CCD θ θ ΔT ΔO ΔX d1 d2 θ Δq Contact: conormce@cs.nuim.ie
465. d Calculating padding To calculate the spatial extent of the object signal (ΔT) we need to be able to calculate Δq. It can be seen from the diagram below that Δq can be represented as And that d can be expressed as: Twin-image plane Object plane Δq CCD θ θ ΔT ΔO ΔX d1 d2 θ Δq Contact: conormce@cs.nuim.ie
466. Calculating padding To calculate the spatial extent of the object signal (ΔT) we need to be able to calculate Δq. It can be seen from the diagram below that Δq can be represented as And that d can be expressed as: We can then express d1,2 in terms of theta: Contact: conormce@cs.nuim.ie
467. Calculating padding To calculate the spatial extent of the object signal (ΔT) we need to be able to calculate Δq. It can be seen from the diagram below that Δq can be represented as And that d can be expressed as: We can then express d1,2 in terms of theta: and simplify these to: Contact: conormce@cs.nuim.ie
468. Calculating padding To calculate the spatial extent of the object signal (ΔT) we need to be able to calculate Δq. It can be seen from the diagram below that Δq can be represented as And that d can be expressed as: and simplify these to: We can then express d as: Contact: conormce@cs.nuim.ie
469. Calculating padding We can now represent Δq in terms of the object size and CCD size. Contact: conormce@cs.nuim.ie
470. Calculating padding We can now represent Δq in terms of the object size and CCD size. And ΔT (the spatial extent of the object signal in the twin plane) as: Contact: conormce@cs.nuim.ie
471. Calculating padding Now that we have the spatial extent of the object signal we can calculate the number of samples required in the hologram plane (NH) with Contact: conormce@cs.nuim.ie
472. Calculating padding Now that we have the spatial extent of the object signal we can calculate the number of samples required in the hologram plane (NH) with Padding up to this amount guarantees that after propagation to the unwanted twin- image plane the object signal will not be wrapped within the reconstruction window. This equation relies on knowing Δ O which is unknown. Contact: conormce@cs.nuim.ie
473. Calculating padding Now that we have the spatial extent of the object signal we can calculate the number of samples required in the hologram plane (NH) with Padding up to this amount guarantees that after propagation to the unwanted twin- image plane the object signal will not be wrapped within the reconstruction window. This equation relies on knowing Δ O which is unknown. We take the nearest power of 2 that the is equal to or greater than NH, this is due to the current algorithms use of the fast Fourier transform which is more efficiently computed when a matrix whose size is a power of 2 is input Contact: conormce@cs.nuim.ie
474. Calculating ΔO But assuming that the hologram was recorded in adherence to the sampling theorem, if we can calculate the distance d the object was positioned away from the CCD we can calculate the maximum size of Δ O at d using (where Nx is the number of samples in the CCD) Contact: conormce@cs.nuim.ie
475. Calculating ΔO But assuming that the hologram was recorded in adherence to the sampling theorem, if we can calculate the distance d the object was positioned away from the CCD we can calculate the maximum size of ΔO at d using (where Nx is the number of samples in the CCD) Given that we have developed a fast autofocus algorithm, we now have an equation for calculating the number of samples required in the hologram plane to avoid aliasing with no unknowns. Contact: conormce@cs.nuim.ie
476. Twin-image Reconstruction Now that we have the distance to the in-focus plane of the twin-image and the required amount of padding to ensure no aliasing in the reconstruction plane, we can reconstruct the twin-image with the propagation transfer function. Contact: conormce@cs.nuim.ie