Summary about approaches to Dense Image Matching as well as examplary results. Keynote at 3D-ARCH 2015 in Avila, Spain.

1. Dense Image Matching
Challenges and Potentials
Konrad Wenzel
6th 3D-Arch Workshop, 25th of February 2015, Avila, Spain

2. Eiger, 20cm GSD

3. Eiger, 20cm GSD

4. Eiger, 20cm GSD

5. Rottenburg
» Panasonic DMC GX-1
System Camera, 16MP
» 14mm lens, uncalibrated
» 2 images per second
» East façade of tower
• 152 images
• True Orthophoto
GSD 3.5mm

6. Rottenburg
» Orthophoto  drawing
02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 6

7. Gyrocopter & IGI DigiCAM
» IGI DigiCAM 50MP
» 131 images, GSD 6cm
7

8. 02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 8

9. 02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 9

10. Carved stone from the Temple of Heliopolis, Egypt
» 30 images (12MP), Nikon D2X
02.03.2015 10

11. Carved stone from the Temple of Heliopolis, Egypt
02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 11

12. Carved stone from the Temple of Heliopolis, Egypt
Konrad Wenzel, University of Stuttgart, ifp
» 0.5mm GSD

13. Glacier
Result from a single DMC II stereo pair
02.03.2015 13
Glacier, DMCII Stereo Pair

14. Glacier
Result from a single DMC II stereo pair
02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 14
Glacier, DMCII Stereo Pair

15. Trento, 10cm GSD, 80/60

16. Trento, 10cm GSD, 80/60

17. Trento, 10cm GSD, 80/60

18. Trento, 10cm GSD, 80/60

19. Stuttgart, 5cm GSD, DMCII, 80/60, 2 strips

23. True Ortho

24. Purpose & Motivation
» Integration of groups of aligned range images
• Range image: each pixel contains depth
• Depth cameras (TOF, Structured light)
• Dense image matching (e.g. SGM)
• Polar & triangulating Laserscanners
 Extraction of consistent , optimal surface models
24

25. Perth, 5cm GSD

26. Perth, 5cm GSD, 80/70 ov.

27. Image Matching
Pixel correspondence search for depth estimation
 Dense image matching – one 3D sample for each pixel

28. 28
Image Matching

29.  Find corresponding pixels
29
Image Matching

30. 3D Point P
x, x‘: Image correspondence between image 1 and image 2
Projection centers (EO+IO)
Viewing rays
① Image Matching  Correspondence x  x‘
② Exterior + Interior orientation  Viewing rays
③ Intersection of viewing rays  3D Point P
30
Image Matching

31. 31
Image Matching
Depth estimation for each pixel
 Dense image matching (Stereo)

32. 32
Image Matching
More than two views
 Multi-View Stereo

33. Challenges
» Ambiguities
• Repetition of grey values
» Noise and weak texture
• Shadows
» Discontinuities
• Edges and details
» Computational complexity
• Suitability for production

34. Matching for one image pair
Stereo

35. Stereo
Great overview – the Middlebury Stereo Page
» D. Scharstein and R. Szeliski. (2002)
A taxonomy and evaluation of dense
two-frame stereo correspondence
algorithms.
» Datasets
» Overview of methods
» Automatic benchmark
» http://vision.middlebury.edu/stereo/

36. Data fusion
Exploit redundancy
Image space
» Use epipolar relations
» Corresponding measurements
from image matching
+ fast data access
+ balance actual measurement
+ topology is available
-- relation limited to matching
(weak on small baselines) 36
Object space
 Use actual 3D data
 e.g. analysis in local neighborhood
+ local geometry is analyzed
+ indepedent validation
+ no image matching required
-- expensive data access
-- topology is challenging
P
xb
xm1
xm2
d

37. Stereo
Approach: Normalized Cross Correlation (NCC)
» Compare local mask for each pixel (e.g. 9 x 9 pixels)
» NCC: „sliding normalized dot product“
» High correlation  match
Image source: https://siddhantahuja.wordpress.com/tag/normalized-cross-correlation/

38. Stereo
Approach: Scanline Optimization
» Dynamic programming
» Consistency along
epipolar line
 streaking effect
Image source: Behzad Salehian ; Abolghasem A. Raie ; Ali M. Fotouhi ; Meisam Norouzi (2013).
Efficient interscanline consistency enforcing method for dynamic programming-based dense stereo matching algorithms

39. Stereo
Approach: Belief Propagation
» Message passing algorithm
» Usable for global optimization
» Popular:
• Bayesian networks
• Markov random fields
» Similar: graph cuts
 Exact minimum solution
 Computationally rather expensive
Image source: Klaus, A., Sormann, M., & Karner, K. (2006, August).
Segment-based stereo matching using belief propagation and a self-adapting
dissimilarity measure. In Pattern Recognition, 2006. ICPR 2006.

40. Stereo
Approach: Semi-Global Matching
• Matching: dense, intensity-based
• Global: optimization approach using a global model
• Semi: approximation  fast numerical solution
Castle Neuschwanstein, Bavaria, Germany
source: Hirschmüller, Heiko (2005) – Accurate and efficient stereo processing by Semi Global Matching an Mutual Information
Intensity image Disparity image using a
correlation matching method
Disparity image using
Semi Global Matching

41. Stereo
Approach: Semi-Global Matching
» SGM Optimization approach:
disparities similar to neighboring pixels are preferred
Assignment of costs for each possible disparity on each pixel
• Costs for the similarity of the grey value ( similar  low costs )
• Additional costs for disparity jumps  forces smooth surfaces 41
Disparity along a path L in the image
02.03.2015

42. Stereo
Approach: Semi-Global Matching
» Recursive cost aggregation on paths through the image
Base image, pixel
pi
Match image, pixel qi,j
Minimal costs
Costs c(pi ,qi,j)

43. • Problem: SGM cost structures require large
amount of memory
• Solution:
• Reduce disparity search ranges to a tube
around actual surface
• Coarse-to-fine approach: Initialize search
ranges /tubes using low resolution imagery
Fast
Low memory requirements
x [pix]
disparity
[pix]
x [pix]
disparity[pix]
Rothermel, M., Wenzel, K., Fritsch, D., Haala, N. (2012).
SURE: Photogrammetric Surface Reconstruction from Imagery.
Stereo
Approach: Semi-Global Matching – tSGM variation

44. 4402.03.2015
Stereo
Approach: Semi-Global Matching – tSGM variation

45. Multi-View Stereo
Matching on multiple views

46. » Steve Seitz, Brian Curless, James Diebel, Daniel Scharstein, Richard Szeliski
A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, CVPR
2006, vol. 1, pages 519-526.
» http://vision.middlebury.edu/mview/
Stereo
Great overview – the Middlebury Multi-View Stereo Page

47. » Yasutaka Furukawa and Jean Ponce,
Accurate, Dense, and Robust Multi-View
Stereopsis, CVPR 2007
» Steps:
• Match: find features
• Expand: grow patches
• Filter: using visibility constraint
» Mesh using regulation constraints
» Available Open Source as PMVS
Multi-View Stereo
Approach: Grow patches around feature points

48. » Deseilligny, M. P., & Clery, I. (2011). Apero, an open source bundle adjusment
software for automatic calibration and orientation of set of images. 3D Arch 2011
» Multi-Stereo matching for one reference image (available as Open Source: MICMAC)
» Graph cut & dynamic programming optimization
Multi-View Stereo
Approach: Multi-stereo matching

49. Multi-View Stereo
Approach: Depth maps and Volumeteric Range Image Integration
» M. Goesele, N. Snavely, B. Curless, H. Hoppe, S. Seitz (2007).
Multi-view stereo for community photo collections, ICCV 2007
» Grow sparse points from SfM
» Estimate refined depth maps with
photoconsistent normals
» Integration using
Volumetric Range Image Integration
Brian Curless and Marc Levoy,
Stanford University (1996):
A Volumetric Method for Building
Complex Models from Range Images.
http://grail.cs.washington.edu/
software-data/vrip/

50. 50
Multi-View Stereo
Approach: Depth maps and Volumeteric Range Image Integration
Korcz, D. (2011). Volumetric Range Image Integration. Diplomathesis, ifp, University of Stuttgart
Dataset: Middlebury Multi-View Stereo
evaluation, Temple

51. 1. Build volumetric space
entity: voxel, a volumetric pixel
2. Project range image into voxel space
3. Compute Signed Distance Field
4. Extract optimal surface
51
Korcz, D. (2011). Volumetric Range Image Integration. Diplomathesis, ifp, University of Stuttgart
Multi-View Stereo
Approach: Depth maps and Volumeteric Range Image Integration

52. 52
Korcz, D. (2011). Volumetric Range Image Integration. Diplomathesis, ifp, University of Stuttgart
Multi-View Stereo
Approach: Depth maps and Volumeteric Range Image Integration
Signed Distance Field for „Dino“ dataset

53. 53
Korcz, D. (2011). Volumetric Range Image Integration. Diplomathesis, ifp, University of Stuttgart
Multi-View Stereo
Approach: Depth maps and Volumeteric Range Image Integration

54. » Iso-Surface extraction using Marching Cubes algorithm
» Hole filling by space carving method
54
Multi-View Stereo
Approach: Depth maps and Volumeteric Range Image Integration

55. » [Zach et al., 2007]: simple averaging of signed distance fields without further
regularization causes inconsistencies
• due to frequent changes of sign
introduction of additional regularization force
 energy minimization
 uses total variation norm (TV-L1), [Rudin et al, 1992]
Smoothness term allows
• regulatization
• noise suppression
• outlier rejection
55
[Rudin et al., 1992] Rudin, L. I., Osher, S., and Fatemi, E. (1992).
Nonlinear total variation based noise removal algorithms.
[Zach, 2008] Zach, C. (2008). Fast and High Quality Fusion of Depth Maps.
Multi-View Stereo
Depth maps and Volumeteric Range Image Integration + Total Variation

56. 56
Multi-View Stereo
Depth maps and Volumeteric Range Image Integration + Total Variation
Source: Korcz, D. (2011). Volumetric Range Image Integration.

57. » Vu, H.; Keriven, R.; Labatut, P. and
Pons, J.-P (2009). Towards high-resolution
large-scale multi-view stereo. CVPR, 2009
» Rough dense point cloud through
normalized cross correlation (NCC)
» minimum s-t cut global optimization
with visibility filtering
» Mesh refinement with photo consistency
Multi-View Stereo
Approach: rough point cloud and mesh refinement

58. » Rothermel, M., Wenzel, K., Fritsch, D., Haala, N. (2012).
SURE: Photogrammetric Surface Reconstruction from Imagery.
» Approach:
1) Stereo matching using tSGM
2) Multi-ray triangulation
3) Object space fusion, e.g.
• DSM
• Volumetric point cloud filtering
• Meshing
Multi-View Stereo
Approach: stereo matching, multi-ray triangulation, object space fusion
P
xb
xm1
xm2
d
Images
Epipolar images
Disparity images

59. SURE: Multi-View Stereo Triangulation
» Redundant measurements across stereo pairs
allow outlier elimination claiming geometric consistency
Stereo Multi-view Multi-view Multi-view
> 1-fold > 2-fold > 3-fold1-fold

60. SURE: Multi-View Stereo Triangulation
» Improvement of surface noise
Stereo
1-fold
Multi-view
> 1-fold

61. SURE: Point Cloud Fusion for 2.5D Surfaces
» Fusion of 3D point clouds to 2.5D surface models
3D point cloud stereo
matching
3D point clouds multi-view
stereo matching
Fusion of point clouds to
2.5D surface model

62. Munich – DMCII, 10cm GSD [EuroSDR Matching Test]

63. Munich – DMCII, 10cm GSD [EuroSDR Matching Test]

64. True Ortho

65. Traditional Orthophoto
65Courtesy of Sven Briels - burokarto.nl
Low overlap conditions (60/40), 3.5cm GSD

66. SURE True Orthophoto
66Courtesy of Sven Briels - burokarto.nl
Low overlap conditions (60/40), 3.5cm GSD (processed at 7cm GSD, 9cm ortho)

67. SURE: Quadtree based on DSM - Example
02.03.2015 Schwerin_2012 67

68. Munich [EuroSDR Matching Test, DMCII, 10cm GSD, 80/80 ov.]
Textured DSM (coming soon)

69. Munich [EuroSDR Matching Test, DMCII, 10cm GSD, 80/80 ov.]
Textured DSM (coming soon)

70. Braunschweig, 80/60 Nadir DSM
Postprocessing & retexturing by

71. SURE: Out-of-core point cloud filtering
» Retrieve locally densest point cloud
 removal of redundancy
» Validate points
 keep only points, which are
validated by other point clouds
» Adapt resolution locally
» Scalable – out-of-core octree
Wenzel, K., Rothermel, M., Fritsch, D., & Haala, N. (2014).
Filtering of Point Clouds from Photogrammetric Surface Reconstruction

72. SURE: Out-of-core point cloud filtering
(IGI DigiCam Penta)
Imagery courtesy of Aerowest GmbH

75. „Bathymetric Photogrammetry“
02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 75

76. Bathymetric Photogrammetry?
02.03.2015 76
Courtesy of Alfons Krismann,
University of Hohenheim
„Bathymetric Photogrammetry“

77. Bathymetric Photogrammetry?
02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 77
Courtesy of Alfons Krismann,
University of Hohenheim
„Bathymetric Photogrammetry“

78. 02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 78
Courtesy of Alfons Krismann,
University of Hohenheim

79. 02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 79
Courtesy of Alfons Krismann,
University of Hohenheim

80. SURE: Observation of abliation in plasma flow
» Plasma channel
» 2500 degrees
» Mirrors
» Sub-mm accuracy
Loehle, S., Staebler, T., Reimer, T., & Cefalu, A. (2014) Photogrammetric Surface Analysis of Ablation Processes in High Enthalpy Air Plasma Flow.

81. SURE: Depth map fusion (Meshing)
» Rothermel, M.; Haala, N.; Fritsch, D. (2014) Generating oriented pointsets from
redundant depth maps using restricted quadtrees.
» Adapt depth map resolution using Restricted Quadtree
» Choose locally optimal depth map while forcing visibility constraints

82. 3D Meshing Example: Fusion + Poisson
02.03.2015 Schwerin_2012 82
Mesh Patches + Poisson

83. 3D Meshing Example: Fusion + Poisson
02.03.2015 Schwerin_2012 83