Research projects at my TUM group - MVP

1. Research of the MVP Group http://mvp.visual-navigation.com
Visual navigation
Perception for manipulation
Biologically motivated
Machine Vision and Perception Group @ TUM
perception
Rigid and Deformable The Machine
Registration Vision and Perception Group
@TUM works on the aspects of
visual perception and control in
medical, mobile, and HCI
applications
Photogrammetric Visual Action Analysis
Exploration of physical monocular reconstruction
object properties
MVP

2. Visual Localization for hybrid Environmental Modeling
Machine Vision and Perception Group @ TUM
Real-Time Localization
The high accuracy allows direct
stiching of images along the
trajectory without bundle
adjustment
Resulting hybrid (appearance
and geometry) model allows
path planning and prediction of
sensor views for e.g. attention
MVP
research

3. MVP Machine Vision and Perception Group @ TUM
Example of a hybrid model reconstruction

4. Result of a visual localization
6DoF pose of the robot is calculated fom single
camera view
Machine Vision and Perception Group @ TUM
MVP

5. Reconstruction of a hybrid (appearance/geometry) model
Machine Vision and Perception Group @ TUM
MVP

6. Visual Homing
Machine Vision and Perception Group @ TUM
The system is capable of
registration to previously
observed trajectories
MVP

7. Biologically Motivated Navigation
Obstacle avoidance
Machine Vision and Perception Group @ TUM
Decoupling of Rotation
and Translation
Perception-based loop closure
MVP

8. MVP Machine Vision and Perception Group @ TUM
Work on Optimal Sensor Models

9. Knowledge Representation
Human demonstration
Machine Vision and Perception Group @ TUM
• Atlas: Atlas
parametric grasp actions and
– Long-term memory shape description points handling
– Experience of the system
• Working memory:
– Short-term memory
Mapping of
– Experience grounded in a Knowledge
given environment Foreground Experience
segmentation abstraction
• Temporal handling information background foreground Episodic buffer
model model of actions
MVP
Working Memory

10. Indexing of the Atlas information from 3D perception
Real-world scenario
Real-world scenario
Machine Vision and Perception Group @ TUM
scene setup
MVP
input point cloud
recognized models

11. Algorithm Description
Algorithm Description
Machine Vision and Perception Group @ TUM
(Model Preprocessing Phase)
(Model Preprocessing Phase)
n2
• For all pairs of surflets at
 distance d insert the triple
 p2   ,  , ∢ n1 , n 2  
n1 d
p1 plus a pointer to its model in a
hash-table.
• Do this for all models using the
MVP
same hash-table.

12. 3D Object Recognition
3D Object Recognition
Machine Vision and Perception Group @ TUM
• Challenges
―
Incomplete Data
―
Noise and outliers
MVP

13. Online Recognition Phase
Online Recognition Phase
Machine Vision and Perception Group @ TUM
• For each model surflet pair
 p1 , n1  , p2 , n2 
   
in the hash-table cell:
n1
p1 Compute the rigid
transform T that
p2  p ,alignsp , n 
best n  ,  
 
n2 1 1 2 2
to
 p 1 , n1  , p2 , n2 
MVP
model hash-table

14. Online Recognition Phase
Online Recognition Phase
Machine Vision and Perception Group @ TUM
• Check if T(M) matches
the scene.
M is the model of
n1  p1 , n1  , p2 , n2 
   
p1
n1 p1
  p2
p2 
n2
n2

MVP
• If yes, accept T(M).
model hash-table

15. MVP Machine Vision and Perception Group @ TUM
Example of the System in Action

16. How to reconstruct 3D under poor texture
condition?
Machine Vision and Perception Group @ TUM
Problem: texture information is more sparse
MVP
16

17. What can we do if the texture information is
almost non-existent?
Machine Vision and Perception Group @ TUM
→ photogrammetric approach
MVP
17

18. Intensity-Based Methods
Machine Vision and Perception Group @ TUM

Established 3D reconstruction methods:

Rely on presence of many reliable features

May perform very poor if this prerequisite is not met

Medical images are typically problematic

Solution approach: Intensity based methods

Do not exclusively rely on salient image regions

Shaded regions, e.g., also contain information

Use all information that is there
MVP

19. Intensity-Based Bundle Adjustment

Regular Bundle Adjustment equation:
Machine Vision and Perception Group @ TUM
min a , b ∑i=1 ∑m=1 d (Q (a j , b i ) , x i , j )
n
j

Finds camera parameters a and point coordinates b

Minimizes feature reprojection error
min a , b ∑n ∑m d ( I j (Q ( a j , b i )) , I 0 ( p i ))
i=1 j=1

Intensity-Based Bundle Adjustment:

Uses a regularizer: b now describes a smooth surface

Apart from that: Same, but minimizes intensity error
MVP

20. Reconstruction Example
Machine Vision and Perception Group @ TUM
Works well under static lighting conditions and
roughly Lambertian surfaces
MVP
Ruepp and Burschka. Fast recovery of weakly textured surfaces from monocular image
sequences. ACCV 2010

21. Estimation of Physical Properties
in Predict-Act-Perceive loop
Machine Vision and Perception Group @ TUM
Estimation of the Center of mass
Estimation of Mass and Friction Force
Estimation of Mass Distribution
predict act perceive
F
r
τ
MVP
simulator

22. Machine Vision and Perception Group @ TUM
Knowledge Representation in WP5
Mapping of
Knowledge
MVP

23. Defining Foreground: Shape Matching
Indexing to the Atlas database needs to be
extended to object classes
-> deformable shape registration needed
(ACCV2010 - oral)
23

24. Object Container
Machine Vision and Perception Group @ TUM
Orientation
Max. allowed acceleration
Mass
Center of gravity
MVP

25. Functionality Map
Location Areas
Machine Vision and Perception Group @ TUM
Connection Properties:
Pushed vs. lifted object
Arbitrary movement vs.
movement with a goal
Connection relevance
Velocity constraint during
pick-up
MVP
Grasp taxonomy
Approach vector

26. Basic Experiments:
Location Areas (Tracking Data)
Machine Vision and Perception Group @ TUM
MVP

27. Basic Experiments:
Functionality Maps (Tracking Data)
Machine Vision and Perception Group @ TUM
Milk carton
Cup - Handle
MVP
Red = goal, green = push, magenta = arbitrary

28. MVP Machine Vision and Perception Group @ TUM
Trajectories
Entire System: Exemplary