Konsep Structure from Motion (SfM) dan Multi-view Stereo (MVS) dalam pemodelan 3D

1. Ekstrak Model Elevasi Digital
dalam bentuk Point Cloud dari
sensor kamera optik
Dany Laksono
January 30th, 2020

2. Extraksi Point Cloud DEM dari Kamera Optik
• Prinsip fotogrametri, SfM-MVS
• Batasan SfM-MVS
• Model Elevasi Digital dan Point Cloud dari SfM
• Beberapa Perangkat Lunak untuk SfM-MVS

3. Prinsip Structure from
Motion dan Multi-view
Stereo

4. “The art, science, and technology of obtaining reliable
information about physical objects and the environment
through processes of recording, measuring, and interpreting
photographic images and patterns of recorded radiant
electromagnetic energy and other phenomena”
Definisi Fotogrametri
--ASPRS

5. Fotogrametri: Kebalikan dari
Fotografi
Tujuan Fotogrametri: mendapatkan ukuran 3D objek dari
serangkaian gambar 2D

6. Dengan perkembangan bidang
computer vision, pengolahan citra
digital memegang peranan penting
dalam rekonstruksi 3D dari foto
untuk keperluan fotogrametri
Ulman (1979) mengusulkan metode
untuk memperoleh informasi 3D
objek tanpa informasi lain selain
gambar objek tersebut
Structure from Motion Photogrammetry
Meskipun demikian, baru pada tahun 2006, Noah Snavely
mendemonstrasikan penggunaan metode sfm untuk rekonstruksi 3D point
cloud dalam ukuran cukup besar

7. Alur Rekonstruksi 3D SfM-MVS
Structure from Motion (SfM) merupakan
metode untuk estimasi posisi dan orientasi
kamera serta gambaran kasar dari objek yang
difoto (≈ AT dan BA)
MVS (Multi-view Stereo) merupakan metode
untuk memperoleh gambaran detil objek
dari tiap piksel yang ada pada foto dengan
menggunakan hasil SfM sebagai masukannya

8. SfM
Photogrammetry?

9. SfM
Photogrammetry?

10. SfM
Photogrammetry?

11. SfM
Photogrammetry?

12. Alur Rekonstruksi 3D SfM-MVS
Image from theiaSFM
1. Feature Detection
2. Feature Matching
3. Pose Estimation and
Adjustment=Sparse
Reconstruction
4. Dense Reconstruction
5. Meshing
6. Texture Mapping
SfM-MVS Algorithm Sequence:

13. SfM-MVS Pipeline
Feature
Detection
Feature
Matching
Bundle
Adjustment
POSE Estimation
Sparse
Reconstruction
Input:
Image Sequences/Multiview Images
Output:
Point Cloud/3D Models
Dense
Reconstruction
Meshing
Texture Mapping

14. Structure from Motion (SfM) Photogrammetry
Image from Noah Snavely
The SfM Problem:
Given a set of
corresponding points in
two or more images,
compute the camera
parameters and the 3D
point coordinates

15. Structure from Motion (SfM) Photogrammetry
Image from Noah Snavely
Estimate:
- Structure xi
From:
- Camera Motion Riti
Iteratively

16. Principles of SfM Reconstruction: Epipolar Geometry
“Given point P in a multiple-view stereo images, the point P, Cl and Cr lies in
an epipolar plane where Cr ≈ Cl * f(R,t)”

17. Principles of SfM Reconstruction: Epipolar Constraint
Titik X dapat berada pada
titik manapun sepanjang
garis OL-X. Dengan
menambahkan tampilan
kedua (OR), lokasi tepat
titik X dibatasi sepanjang
garis epipolar yang
bersesuaian
Rotasi (R) dan translasi (t)
antara kedua view dapat
dinyatakan dengan
Matriks Essensial (E)atau
Matriks Fundamental (F)

18. Principles of SfM Reconstruction: Minimizing Reprojection Error
Setelah x dan R, t awal diketahui, selanjutnya dapat dihitung:
Iterasi dengan meminimalkan reprojection error:
= Bundle Adjustment

19. Alur Rekonstruksi 3D SfM-MVS

20. Alur SfM
Frahm, et. al. (2017)

21. Alur SfM
Frahm, et. al. (2017)

22. Alur SfM
Frahm, et. al. (2017)

23. Alur SfM
Frahm, et. al. (2017)

24. Alur SfM
Frahm, et. al. (2017)

25. Alur SfM
Frahm, et. al. (2017)

26. Alur SfM
Frahm, et. al. (2017)

27. Alur SfM
Frahm, et. al. (2017)

28. Alur SfM
Frahm, et. al. (2017)

29. Alur SfM
Frahm, et. al. (2017)

30. Alur SfM
Frahm, et. al. (2017)

31. Alur SfM
Frahm, et. al. (2017)

32. Alur SfM
Frahm, et. al. (2017)

33. Feature Detection and Matching
“Create a database of keypoints in one image and find the most probable
match in the other image”
Image @OpenCV

34. Feature Detection and Matching: SIFT feature detector
Quinn, 2017

35. Feature Detection and Matching: Beberapa Feature
Descriptor
Fan, 2013
1999, SIFT [Citation: 23819]
2003, Shape Context
2006, SURF [Citation: 4093]
2008, SMD, DAISY
2009, OSID, CS-LBP
2010, BRIEF, HRI-CSLTP, BiCE
2011, ORB, BRISK, LIOP, MROGH
2012, FREAK, KAZE, SYM
2013, Line Context

36. Feature Detection and Matching: SIFT feature detector

37. Feature Detection and Matching: Epipolar Constraint

38. Feature Detection and Matching: Keypoint Matching

39. Feature Detection and Matching: Sebelum RANSAC

40. Feature Detection and Matching: Sesudah RANSAC

41. Eliminasi Outlier dengan RANSAC

42. Sparse Reconstruction
Feature detection, matching and initial bundle adjustment resulted in
estimated camera position and initial 3D geometry of the target
Result of SfM:
Estimation of object’s
3D geometry
(or Sparse Model)
and
Camera Position and
Orientation

43. Alur Rekonstruksi 3D SfM-MVS

44. Alur Rekonstruksi 3D SfM-MVS
Jika seluruh posisi kamera
dan estimasi posisi objek
telah diketahui,
permasalahan yang
tersisa adalah
menentukan kedalaman
titik pada seluruh piksel
Algoritma Multi-view
Stereo (MVS) dapat
digunakan untuk
menyelesaikan
permasalahan tersebut

45. Depth Estimation

46. Depth Estimation

47. MVS: Depth Estimation
error
depth
Slide by Noah Snavely

48. MVS: Depth Map to Point Cloud
Untuk tiap foto, dapat dicari depth map sebagai representasi kedalaman tiap piksel
Dari semua Depth Map di tiap foto kemudian dapat dibentuk point cloud dari objek
(structure) yang dicari

49. MVS: Depth Map to Point Cloud
Untuk tiap foto, dapat dicari depth map sebagai representasi kedalaman tiap piksel
Dari semua Depth Map di tiap foto kemudian dapat dibentuk point cloud dari objek
(structure) yang dicari

50. Dense Reconstruction
Denser Point-cloud,
based on interpolated
(or extrapolated)
sparse point cloud
The color in Colorized
Point cloud are
obtained from image
RGB

51. Dense Reconstruction: Semi Global Matching (SGM)
Semi-Global Matching (SGM) can be
based on high resolution imagery and
each pixel can be processed to render a
3-D Point Cloud of high density (150 to
380 points per square meter)
Up to 9 points per Square Meter
LiDAR
~ 172 points per Square Meter at 3” GSD
SGM
LiDAR derived point clouds are limited by the
sampling frequency of the sweeping beam and
the laser beam width (1 to 9 points per square
meter)
Wilson (2016)

52. Dense Reconstruction: Semi Global Matching (SGM)
A LiDAR derived 3-D Model is lower resolution and thus lacks sufficient resolution to
see surface changes that Semi-Global Matching (SGM) detects
However, current SGM software renders surfaces that have more low-level noise
than LiDAR surfaces
SGM DerivedLiDAR Derived
Images from the paper SEMI-GLOBAL MATCHING: AN ALTERNATIVE TO LIDAR FOR DSM GENERATION? By S. Gehrke, et al,

53. Multiview Stereo: Meshing and Texturing
“Mesh” is
obtained from
further
interpolation of
dense point cloud
Texture are obtained by ‘draping’
photo into the 3D Model based on
model’s Normal Linehttp://www.gris.tu-darmstadt.de/projects/mvs-texturing/

54. Model Elevasi Digital dan
Point Cloud dari SfM

55. Produk SfM-MVS

56. Dengan SfM-MVS, dapat diperoleh
produk, a.l:
Orthophoto Mosaic Point Cloud/3D Models Digital Elevation (DEM)
Yuwono, 2018

57. Deliverables of SfM-MVS
Sparse Reconstruction (SfM)
Depth Map (MVS)
Dense Reconstruction (MVS)
Meshing and Texturing

58. Deliverables of SfM-MVS
Sorted by the order of
processing, the result of SfM
and Multiview Stereo are
(Laksono, 2016):
a) Sparse Point-Cloud
b) Dense Point Cloud
c) Colorized Dense Point
Cloud
d) Mesh Surface
e) Textured Surface

59. DEM or Digital
Elevation
Model are
obtained from
interpolated
Dense Cloud or
3D Mesh
Produk SfM
For Mapping purpose, DEM and Orthophoto Mosaic might be more
desirable

60. DEM or Digital
Elevation
Model are
obtained from
interpolated
Dense Cloud or
3D Mesh
Other Deliverables
An Orthophoto Mosaic is an orthographic projection imagery
(“Top-Down looking” camera)
Orthophoto is
a map ready
production,
showing
elevation-
corrected
photo instead
of just Photo
Mosaic

61. Photogrammetry Produces DSM instead of DTM
General Rule:
“What the
camera (or
pixel) could
see, would be
produced as
3D”
Image from charim.net

62. Photogrammetry Produces DSM instead of DTM
@LMJaelani

63. Ekstraksi DTM dari DSM: Cloth Simulation Filter (CSF)

64. Sparse Reconstruction (MicMac)

65. Dense Reconstruction (MicMac)

66. DTM Extraction (CSF-CloudCompare)

67. Dense Reconstruction
(OpenMVG-MVE)

68. DTM Extract
(CSF-CloudCompare)

69. Aplikasi dan Batasan SfM

71. SfM opens up new possibilities for 3D reconstruction
http://www.cs.cornell.edu/~snavely/bundler/

72. Rekonstruksi 3D dari Flickr Images
http://www.cs.cornell.edu/~snavely/bundler/

73. SfM opens up new possibilities for 3D reconstruction

74. SfM opens up new possibilities for 3D reconstruction
City-Scale Reconstruction using Unordered Images

75. SfM opens up new possibilities for 3D reconstruction
Indoor Mapping
& Localization

76. Trend: Estimasi Depth Map dengan Deep Learning

77. But it also has Limitations..

78. Since SfM relies on Feature
Detection, it is prone to error on
feature with High Similarity

79. Limitations of SfM-MVS
Group of images
capturing eastern wing
of UGM building
Group of images
capturing western wing
of UGM building
Connectivity Graph
showing clustered
images, where it
shouldn’t be connected
at all

80. Effect of Similar Features
https://cvg.ethz.ch/research/symmetries-in-sfm/

81. Limitations of SfM-MVS: Non-Lambertian Surface, Weak
Surface, Thin Structure

82. In Agisoft, SfM
workflow is defined in
one single menu

83. Rolling Shutter Correction
Pengaturan Kamera
Apabila digunakan drone dengan tipe
copter (Mis. DJI Phantom series),
centang 'Enable rolling shutter
compensation' untuk koreksi efek
rolling shutter
rolling shutterglobal shutter

84. Perangkat Lunak SfM Open
Source

85. Perangkat Lunak SfM-MVS
Bemis et al. (2014). Ground-based and UAV-Based photogrammetry: A multi-scale, high resolution mapping tool for structural geology and
paleoseismology. Journal of Structural Geology

86. Perangkat Lunak SfM-MVS
.. Alternative to Commercial Software
• WebODM (https://www.opendronemap.org/webodm/download/)
• OpenMVG (https://github.com/openMVG/openMVG/)
• COLMAP (https://colmap.github.io/)
• Visual SFM/Bundler (http://ccwu.me/vsfm/)
• Apero Micmac (http://logiciels.ign.fr/?Micmac)
• TheiaSFM (http://www.theia-sfm.org/)
• Kitware MapTK (https://github.com/Kitware/maptk)

87. Big Players in 3D Reconstruction
.. Alternative to Commercial Software
• Regard3D (http://www.regard3d.org/)
• AliceVision Meshroom (https://alicevision.github.io/)
• Python Photogrammetry Toolbox (http://184.106.205.13/arcteam/ppt.php)
• Sf3M (http://sf3mapp.csic.es/)
• SFM Toolkit (http://www.visual-experiments.com/demos/sfmtoolkit/)

88. FOSS SfM-MVS
openmvg.readthedocs.org
• Perangkat lunak SfM gratis
• Mengintegrasikan feature detection, feature matching, dan
bundle adjustment
• (optional) menggunakan PMVS/CMVS untuk dense
reconstruction serta SfM Georef untuk Georeferencing
VisualSfM

89. FOSS SfM-MVS
openmvg.readthedocs.org
VisualSfM

90. FOSS SfM-MVS
openmvg.readthedocs.org
OpenMVG
• Perangkat lunak SfM opensource berbasis Linux
• Menggunakan Algoritma Incremental SfM dan Global SfM
• Memiliki fungsi Georeference yang terintegrasi
• Tersedia GUI untuk Windows: Regard3D

91. FOSS SfM-MVS
openmvg.readthedocs.org
Regard3D (OpenMVG GUI for Windows)

92. FOSS SfM-MVS
openmvg.readthedocs.org
COLMAP
Reconstruction of central Rome using 21K photos produced by COLMAP’s SfM pipeline.
Dense reconstruction of several landmarks produced by COLMAP’s MVS pipeline.

93. FOSS SfM-MVS
openmvg.readthedocs.org
SURE Photogrammetry
Point Cloud, 3D Model & orthophoto

94. TERIMA KASIH