Presentation at my viva defense 2009

1. Digital Hologram Image Processing (DHIP) Conor Mc Elhinney Wednesday 13th May Contact: conormce@cs.nuim.ie

2. Why digital holography? Using digital holography we can record a scene in a complex valued data structure which retains some of the scene's 3D information. A standard image obtained with a camera records a 2D focused image of the scene from one perspective. Contact: conormce@cs.nuim.ie

3. Why digital holography? Yves Gevant Ultimate Hologram http://www.ultimate-holography.com Contact: conormce@cs.nuim.ie

4. Image Processing Image processing attempts to “understand” a scene through the analysis of one recorded image or a sequence of recorded images of the scene. Typical questions it tries to answer are: What is important in the scene? How many relevant objects are in the scene? Where are they in the scene? What do they look like? What are they? In standard image processing each recorded image is a 2D focused image of the scene. We wanted to apply image processing to digital holograms where each reconstruction is a 2D focused image of the scene. Contact: conormce@cs.nuim.ie

5. How can we use this extra information in a hologram In the early days we took a look around to see what was being attempted in digital holography and what had and hadn’t been solved Microscopic Macroscopic Autofocus – Some Proposed solutions Shape Estimation – Phase unwrapping Segmentation - ? Autofocus – Some Proposed solutions Shape Estimation – Ma depth extraction just proposed Segmentation - ? Contact: conormce@cs.nuim.ie

6. How can we use this extra information in a hologram In the early days we took a look around to see what was being attempted in digital holography and what had and hadn’t been solved Microscopic Macroscopic In-Line In-Line Autofocus – Some Proposed solutions Shape Estimation – Phase unwrapping Segmentation - ? Twin-Image - ? Autofocus – Some Proposed solutions Shape Estimation – Ma depth extraction just proposed Segmentation - ? Twin-Image - ? Contact: conormce@cs.nuim.ie

7. How can we use this extra information in a hologram In the early days we took a look around to see what was being attempted in digital holography and what had and hadn’t been solved We saw a lack of research in the processing of single capture in-line macroscopic digital holograms Microscopic Macroscopic In-Line In-Line Autofocus – Some Proposed solutions Shape Estimation – Phase unwrapping Segmentation - ? Twin-Image - ? Autofocus – Some Proposed solutions Shape Estimation – Ma depth extraction just proposed Segmentation - ? Twin-Image - ? Contact: conormce@cs.nuim.ie

8. Our noble goal So we started out wanting to develop ways of “understanding” what is in a hologram..... Contact: conormce@cs.nuim.ie

9. Our noble goal So we started out wanting to develop ways of “understanding” what is in a hologram..... What information is in the scene? Contact: conormce@cs.nuim.ie

10. Our noble goal So we started out wanting to develop ways of “understanding” what is in a hologram..... What information is in the scene? Contact: conormce@cs.nuim.ie

11. Our noble goal So we started out wanting to develop ways of “understanding” what is in a hologram..... What information is in the scene? Where is this information? + Contact: conormce@cs.nuim.ie

12. Our noble goal So we started out wanting to develop ways of “understanding” what is in a hologram..... What information is in the scene? Where is this information? + Contact: conormce@cs.nuim.ie

13. Our noble goal So we started out wanting to develop ways of “understanding” what is in a hologram..... What information is in the scene? Where is this information? What does it look like? + + Contact: conormce@cs.nuim.ie

14. Our noble goal So we started out wanting to develop ways of “understanding” what is in a hologram..... What information is in the scene? Where is this information? What does it look like? What is it? + + + Contact: conormce@cs.nuim.ie

15. Our noble goal So we started out wanting to develop ways of “understanding” what is in a hologram..... What information is in the scene? Where is this information? What does it look like? What is it? + + + Object 1 Object 2 Contact: conormce@cs.nuim.ie

16. Our noble goal So we started out wanting to develop ways of “understanding” what is in a hologram..... Combined this helps us “understand” the scene What information is in the scene? Where is this information? What does it look like? What is it? + + + Object 1 Object 2 Contact: conormce@cs.nuim.ie

17. To be generally applicable.. We also wanted our algorithms to be useful to as many forms of holography as possible. So each function or algorithm should be modular, i.e Depth segmentation requires a depth map as input which could be an unwrapped phase map from digital holographic microscopy. Contact: conormce@cs.nuim.ie

18. What I’ve done What is a digital hologram? Twin-images / dc-term/... How does holography work? What have others done in macroscopic DH? How to focus a DH? How to get depth info? Segmenting based on focus info Create an in-focus image from depth/intensity Segmenting based on depth info How do we focus fast? Segment and remove the twin Contact: conormce@cs.nuim.ie

19. What I’ve done What is a digital hologram? Twin-images / dc-term/... How does holography work? What have others done in macroscopic DH? How to focus a DH? How to get depth info? Segmenting based on focus info Create an in-focus image from depth/intensity Segmenting based on depth info How do we focus fast? Segment and remove the twin Contact: conormce@cs.nuim.ie

20. What I’ve done What is a digital hologram? Twin-images / dc-term/... How does holography work? What have others done in macroscopic DH? How to focus a DH? How to get depth info? Segmenting based on focus info Create an in-focus image from depth/intensity Segmenting based on depth info How do we focus fast? Segment and remove the twin Contact: conormce@cs.nuim.ie

21. What I’ve done What is a digital hologram? Twin-images / dc-term/... How does holography work? What have others done in macroscopic DH? How to focus a DH? How to get depth info? Segmenting based on focus info Create an in-focus image from depth/intensity Segmenting based on depth info How do we focus fast? Segment and remove the twin Contact: conormce@cs.nuim.ie

22. What I’ve done What is a digital hologram? Twin-images / dc-term/... How does holography work? What have others done in macroscopic DH? How to focus a DH? How to get depth info? Segmenting based on focus info Create an in-focus image from depth/intensity Segmenting based on depth info How do we focus fast? Segment and remove the twin Contact: conormce@cs.nuim.ie

23. What I’ve done What is a digital hologram? Twin-images / dc-term/... How does holography work? What have others done in macroscopic DH? How to focus a DH? How to get depth info? Segmenting based on focus info Create an in-focus image from depth/intensity Segmenting based on depth info How do we focus fast? Segment and remove the twin Contact: conormce@cs.nuim.ie

24. What I’ve done What is a digital hologram? Twin-images / dc-term/... How does holography work? What have others done in macroscopic DH? How to focus a DH? How to get depth info? Segmenting based on focus info Create an in-focus image from depth/intensity Segmenting based on depth info How do we focus fast? Segment and remove the twin Contact: conormce@cs.nuim.ie

25. What I’ve done What is a digital hologram? Twin-images / dc-term/... How does holography work? What have others done in macroscopic DH? How to focus a DH? How to get depth info? Segmenting based on focus info Create an in-focus image from depth/intensity Segmenting based on depth info How do we focus fast? Segment and remove the twin Contact: conormce@cs.nuim.ie

26. What I’ve done What is a digital hologram? Twin-images / dc-term/... How does holography work? What have others done in macroscopic DH? How to focus a DH? How to get depth info? Segmenting based on focus info Create an in-focus image from depth/intensity Segmenting based on depth info How do we focus fast? Segment and remove the twin Contact: conormce@cs.nuim.ie

27. What I’ve done What is a digital hologram? Twin-images / dc-term/... How does holography work? What have others done in macroscopic DH? How to focus a DH? How to get depth info? Segmenting based on focus info Create an in-focus image from depth/intensity Segmenting based on depth info How do we focus fast? Segment and remove the twin Contact: conormce@cs.nuim.ie

28. Digital reconstruction Just some quick examples and reminders of what is in a reconstruction. Contact: conormce@cs.nuim.ie

29. DC-term suppression Taking a single hologram prior to any processing, we reconstruct this to demonstrate the corruptive effect of the DC-term 365 mm Reconstruction plane Hologram Reconstruct Contact: conormce@cs.nuim.ie

30. DC-term suppression Taking a single hologram prior to any processing, we reconstruct this to demonstrate the corruptive effect of the DC-term 365 mm Reconstruction plane Hologram Reconstruct There are many methods for suppressing the DC-term, our twin-image removal algorithm takes as input a DC-term suppressed hologram. For our experiments we apply a high-pass filter in the Fourier domain to suppress the DC-term. Contact: conormce@cs.nuim.ie

31. DC-term suppression Hologram Reconstruction Contact: conormce@cs.nuim.ie

32. DC-term suppression Hologram Fourier transform Contact: conormce@cs.nuim.ie

33. DC-term suppression Hologram Fourier transform High-pass Filter Contact: conormce@cs.nuim.ie

34. DC-term suppression Hologram DC-Free Hologram Inverse Fourier transform Fourier transform High-pass Filter Contact: conormce@cs.nuim.ie

35. DC-term suppression Hologram DC-Free Hologram Inverse Fourier transform Fourier transform High-pass Filter Reconstruction Contact: conormce@cs.nuim.ie

36. DC-term suppression Hologram DC-Free Hologram Inverse Fourier transform Fourier transform High-pass Filter Reconstruction Reconstruction Contact: conormce@cs.nuim.ie

37. DC-term suppression example Contact: conormce@cs.nuim.ie

38. Shallow depth-of-field Reconstructions from digital holograms have a shallow depth of field, sometimes as small as 1mm. This means that processing an individual reconstruction is rarely a good idea. 188mm 178mm Contact: conormce@cs.nuim.ie

39. Shallow depth-of-field Reconstructions from digital holograms have a shallow depth of field, sometimes as small as 1mm. This means that processing an individual reconstruction is rarely a good idea. 188mm 178mm Contact: conormce@cs.nuim.ie

40. Focusing a digital hologram Contact: conormce@cs.nuim.ie

41. Using a window to reconstruct Contact: conormce@cs.nuim.ie

42. Perspectives and digital holography So we first select a window from within the hologram plane (we also need to know the distance to the object). Viewer Display Contact: conormce@cs.nuim.ie

43. Perspectives and digital holography So we first select a window from within the hologram plane (we also need to know the distance to the object). Viewer Display Contact: conormce@cs.nuim.ie

44. How to we reconstruct a perspective We select a window size from within the hologram. Win Size Contact: conormce@cs.nuim.ie

45. How to we reconstruct a perspective We select a window size from within the hologram. As we have already seen there is a trade-off between window size and visual quality. Win Size Contact: conormce@cs.nuim.ie

46. How to we reconstruct a perspective We select a window size from within the hologram. There is a trade-off between window size and visual quality. We then move the window from the centre of the hologram window. Offset ax Contact: conormce@cs.nuim.ie

47. What angle are we reconstructing? So we first select a window from within the hologram plane (we also need to know the distance to the object). Hologram Plane Near Object Plane d Optical Axis Contact: conormce@cs.nuim.ie

48. What angle are we reconstructing? So we first select a window from within the hologram plane (we also need to know the distance to the object). Hologram Plane Near Object Plane d Optical Axis Contact: conormce@cs.nuim.ie

49. What angle are we reconstructing? We then work out how far we want to offset this window from the centre of the hologram. Hologram Plane Near Object Plane ax d Optical Axis Contact: conormce@cs.nuim.ie

50. What angle are we reconstructing? The angle we are reconstructing can then be worked out with trigonometry. Hologram Plane Near Object Plane ax d Optical Axis Contact: conormce@cs.nuim.ie

51. What angle are we reconstructing? The angle we are reconstructing can then be worked out with trigonometry. Hologram Plane Near Object Plane ax θx d Optical Axis Contact: conormce@cs.nuim.ie

52. What angle are we reconstructing? The angle we are reconstructing can then be worked out with trigonometry. Hologram Plane Near Object Plane ax θx d Optical Axis Contact: conormce@cs.nuim.ie

53. What perspective are we reconstructing Nx y Nx’ Ny ay Ny’ x ax z d and Contact: conormce@cs.nuim.ie

54. Perspective reconstruction Contact: conormce@cs.nuim.ie

55. How does speckle reduction effect the reconstruction Contact: conormce@cs.nuim.ie

56. Simple linear autofocus Contact: conormce@cs.nuim.ie

57. Examples of what we’ve done Automatically determining the focal plane of a digital hologram using a Fibonacci search Contact: conormce@cs.nuim.ie

58. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Choose our search range Contact: conormce@cs.nuim.ie

59. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Choose our search range start Contact: conormce@cs.nuim.ie

60. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Choose our search range end Contact: conormce@cs.nuim.ie

61. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: Contact: conormce@cs.nuim.ie

62. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm Calculate our first reconstruction distance Contact: conormce@cs.nuim.ie

63. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm 682.96mm Calculate our second reconstruction distance Contact: conormce@cs.nuim.ie

64. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Focus value for that distance Contact: conormce@cs.nuim.ie

65. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Red means current best estimate. Contact: conormce@cs.nuim.ie

66. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Iteration 2: 365.93mm (73.3) Contact: conormce@cs.nuim.ie

67. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Iteration 2: 365.93mm (73.3) Iteration 3: 291.1mm (35.9) Contact: conormce@cs.nuim.ie

68. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Iteration 2: 365.93mm (73.3) Iteration 3: 291.1mm (35.9) 412.20mm (44.8) Iteration 4: Contact: conormce@cs.nuim.ie

69. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Iteration 2: 365.93mm (73.3) Iteration 3: 291.1mm (35.9) 412.20mm (44.8) Iteration 4: Iteration 5: 337.35mm (66.6) Contact: conormce@cs.nuim.ie

70. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Iteration 2: 365.93mm (73.3) Iteration 3: 291.1mm (35.9) 412.20mm (44.8) Iteration 4: Iteration 5: 337.35mm (66.6) Iteration 6: 383.6mm (54.3) Contact: conormce@cs.nuim.ie

71. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Iteration 2: 365.93mm (73.3) Iteration 3: 291.1mm (35.9) 412.20mm (44.8) Iteration 4: Iteration 5: 337.35mm (66.6) Iteration 6: 383.6mm (54.3) Iteration 7: 355.02mm (103.1) Contact: conormce@cs.nuim.ie

72. Autofocus - Fibonacci search example Search Range: [170mm,............................................................................................., 1000mm] Iteration 1: 487.03mm (26.5) 682.96mm (22.3) Iteration 2: 365.93mm (73.3) Iteration 3: 291.1mm (35.9) 412.20mm (44.8) Iteration 4: Iteration 5: 337.35mm (66.6) Iteration 6: 383.6mm (54.3) Iteration 7: 355.02mm (103.1) ............... Iteration 14: 353.42mm (108.0) Contact: conormce@cs.nuim.ie

73. Autofocus - Fibonacci search example Displayed in the plot is the first 8 estimates output from Fibonacci (both correct and incorrect) overlayed on the focus plot from the fixed step size search. Contact: conormce@cs.nuim.ie

74. Depth from focus decisions Two of the primary decisions in depth-from-focus are: What block size to use? What interval between reconstructions to use? Contact: conormce@cs.nuim.ie

75. Block Size To determine the depth of a block in an image using a focus measure there needs to be enough object information in the block. Smaller block sizes: finer object features but high error in the estimate of the general shape. Larger block sizes: low error but fine object features lost. Object Contact: conormce@cs.nuim.ie 7x7 43x43 81x81 121x121 151x151

76. Block Size Contact: conormce@cs.nuim.ie

77. Distance between reconstructions By changing the distance between reconstructions we affect the quality of the depth maps. The smaller the distance the more features we can detect but at the expense of speed. Contact: conormce@cs.nuim.ie

78. Distance between reconstructions By changing the distance between reconstructions we affect the quality of the depth maps. The smaller the distance the more features we can detect but at the expense of speed. Contact: conormce@cs.nuim.ie

79. Depth segmentation and the reconstruction interval First we are going to look at the two bolts object and segmenting it into 4 regions Contact: conormce@cs.nuim.ie

80. Two bolts - 4 segments 0.1mm Contact: conormce@cs.nuim.ie

81. Two bolts - 4 segments 0.2mm Contact: conormce@cs.nuim.ie

82. Two bolts - 4 segments 0.5mm Contact: conormce@cs.nuim.ie

83. Two bolts - 4 segments 1mm Contact: conormce@cs.nuim.ie

84. Two bolts - 4 segments 2mm Contact: conormce@cs.nuim.ie

85. Depth segmentation and the reconstruction interval Now we will see if we can segment it into 8 regions Contact: conormce@cs.nuim.ie

86. Two bolts - 8 segments 0.1mm Contact: conormce@cs.nuim.ie

87. Two bolts - 8 segments 0.1mm Contact: conormce@cs.nuim.ie

88. Two bolts - 8 segments 0.5mm Contact: conormce@cs.nuim.ie

89. Two bolts - 8 segments 1mm Contact: conormce@cs.nuim.ie

90. Two bolts - 8 segments 2mm ?? Contact: conormce@cs.nuim.ie

91. Twin-image A hologram contains a set of twin-images. One at the positive distance and one at the negative. They act as a noise source in each others in-focus plane. DC-Free Hologram Virtual Image Real Image 355 mm -355 mm Contact: conormce@cs.nuim.ie

92. Results – Rotating object Contact: conormce@cs.nuim.ie

93. Results – Rotating object Questions?? Contact: conormce@cs.nuim.ie