1. Fraunhofer
Image Processing
Heinrich Hertz Institute
Solutions for Immersive 3D Video
Communication
iMinds, Ghent, Belgium, 16th December 2010
Dr.
Dr Oliver Schreer
Nicole Atzpadin
Wolfgang Waizenegger
Ingo F ld
I Feldmann
Fraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587 Berlin www.hhi.fraunhofer.de 16. Dec. 2010 1

2. Image Processing
Outline
 State of the art in telepresence systems and 3D
videoconferencing
 Existing drawbacks of current systems
 Virtual i
Vi t l view rendering
d i
 Conclusion
©
Dr. Oliver Schreer 16. Dec. 2010 2

3. Image Processing
SoA of Telepresence Systems
telepresence
system by
CISCO
HP Halo
telepresence Polycom TPX system
system
©
Dr. Oliver Schreer 16. Dec. 2010 3

4. Image Processing
SoA of 3D Videoconferencing
Coliseum by
HP, Palo Alto, CA, 2003
VIRTUE System by FP5-IST-VIRTUE, 2003
MultiView by Univ. of California, Berkeley, 2004 im.point by Fraunhofer HHI, 2004
©
Dr. Oliver Schreer 16. Dec. 2010 4

5. Image Processing
Outline
State of the art in telepresence systems and 3D
videoconferencing
Existing drawbacks of current systems
Virtual view rendering
Conclusion
©
Dr. Oliver Schreer 16. Dec. 2010 5

6. Image Processing
Existing Drawbacks
Misleading gesture representation in multi-point setups (who is
pointing at whom?)
©
Dr. Oliver Schreer 16. Dec. 2010 6

7. Misleading Viewing Direction and Pointing
Image Processing
Gesture in Commercial Systems
What view of the remote
participant are you expecting
from the position of this local
participant ?
View f
Vi from l ft local t right remote
left l l to i ht t View f
Vi from right l
i ht local t right remote
l to i ht t
©
Dr. Oliver Schreer 16. Dec. 2010 7

8. Image Processing
The Principle of the Shared Table
0 m
1 ,0
R emote C onferees A
0 m
2 ,0
L ocal C onferees 120°
R emote C onferees B
©
Dr. Oliver Schreer 16. Dec. 2010 8

9. Image Processing
Existing Drawbacks
Misleading gesture representation in multi-point setups (who is
pointing at whom?)
shared table principle
correct
t camera mounting
ti
No stereoscopic viewing
©
Dr. Oliver Schreer 16. Dec. 2010 9

10. Image Processing
Which Display Technologies for 3D?
Stereoscopic
S i
displays
Autostereoscopic
flat displays
p y
©
Dr. Oliver Schreer 16. Dec. 2010 10

11. Image Processing
The Answer is Given by Our Chancellor
©
Dr. Oliver Schreer 16. Dec. 2010 11

12. Image Processing
Existing Drawbacks
Misleading gesture representation in multi-point setups (who is
pointing at whom?)
shared table principle
correct
t camera mounting
ti
No stereoscopic viewing
auto-stereoscopic displays
No multi-perspective viewing for multiple users at one site
novel multi-perspective 3D display by Philips (see talk by Harm, Belt
p p p y y p ( y ,
from Philips)
©
Dr. Oliver Schreer 16. Dec. 2010 12

13. Image Processing
Example
©
Dr. Oliver Schreer 16. Dec. 2010 13

14. Image Processing
Existing Drawbacks
Misleading gesture representation in multi-point setups (who is
pointing at whom?)
shared table principle
correct camera mounting
No stereoscopic viewing
auto-stereoscopic displays
No multi-perspective viewing for multiple users at one site
novel multi-perspective 3D display by Philips (see talk by Harm, Belt
from Philips)
Lack of eye contact
©
Dr. Oliver Schreer 16. Dec. 2010 14

15. Image Processing
Problem of Eye Contact
Problem
Displacement
angle
Solution
Virtual camera
Realization
• Di
Disparity/depth
it /d th
estimation
• Virtual rotation of
conferee
©
Dr. Oliver Schreer 16. Dec. 2010 15

16. Image Processing
Summary
Misleading gesture representation in multi-point setups (who is
pointing at whom?)
shared table principle
correct camera mounting
No stereoscopic viewing
auto-stereoscopic displays
No multi-perspective viewing for multiple users at one site
novel multi-perspective 3D display
Lack of eye contact
Real-time 3D video processing is required for
 support of multi-view 3D di l
t f lti i displays
 provision of eye contact by virtual view rendering
©
Dr. Oliver Schreer 16. Dec. 2010 16

17. Image Processing
Outline
State of the art in telepresence systems and 3D
videoconferencing
Existing drawbacks of current systems
Virtual view rendering
Conclusion
©
Dr. Oliver Schreer 16. Dec. 2010 17

18. Image Processing
Virtual View Rendering
top camera
©
Dr. Oliver Schreer 16. Dec. 2010 18

19. Image Processing
Virtual View Rendering
video + depth for
virtual camera
warp
virtual view
©
Dr. Oliver Schreer 16. Dec. 2010 19

20. Image Processing
The approach
horizontal
disparities
horizontal small
baseline DE
trifocal
4 video consistency
robust
streams check
vertical small disparities
baseline DE
vertical
disparities
patch sweeping
refinement video +
depth (n)
 Initial disparity/depth estimation based on hybrid-recursive
matching (g (HRM) )
 Robust disparities after trifocal consistency check
 GPU based patch sweeping for refinement in depth
©
Dr. Oliver Schreer 16. Dec. 2010 20

21. Image Processing
Principle of patch sweeping
 Change of orientation of a surflet
in the reference camera to
match best neighbour cameras
 User d fi d d th resolution
U defined depth l ti
possible
 High p
g parallelization p
possible
 Well suited for graphic card
implementation
©
Dr. Oliver Schreer 16. Dec. 2010 21

22. Image Processing
Depth analysis results
HRM result for pixel accuracy, sub-pel accuracy of factor 2, sub-pel accuracy of factor 4
Patch sweeping result: original view (left), synthesized view (
p g g ( ), y (middle), depth map (right)
), p p( g )
©
Dr. Oliver Schreer 16. Dec. 2010 22

23. Image Processing
Depth-Image-Based Rendering
M
m1 m2
real view virtual view
Cv
C1
©
Dr. Oliver Schreer 16. Dec. 2010 23

24. Simulation Results on Virtual View Image Processing
Synthesis
top camera ground truth camera
©
Dr. Oliver Schreer 16. Dec. 2010 24

25. Simulation Results on Virtual View Image Processing
Synthesis
top camera interpolated view
©
Dr. Oliver Schreer 16. Dec. 2010 25

26. Image Processing
Original Video + Depth
Camera 1:
original video
and depth
Camera 2
C 2:
original video
and depth
©
Dr. Oliver Schreer 16. Dec. 2010 26

27. Original Views and Target Ground Truth Image Processing
Camera
original view target view
©
Dr. Oliver Schreer 16. Dec. 2010 27

28. Image Processing
Target View and Synthesized View
target view synthesized view
©
Dr. Oliver Schreer 16. Dec. 2010 28

29. Video Example: Target View and Image Processing
Synthesized View
target
view
synthesized
view
©
Dr. Oliver Schreer 16. Dec. 2010 29

30. Image Processing
The Virtual Eye Contact Engine
First demonstration to the public at 3D Stereo Media,
Liège, 8.-10. December 2010!!
©
Dr. Oliver Schreer 16. Dec. 2010 30

31. Image Processing
The Virtual Eye Contact Engine
©
Dr. Oliver Schreer 16. Dec. 2010 31

32. Image Processing
Conclusion
 Commercial video communication systems still lack with respect to
true tele-immersion
 Limited gesture awareness
 No eye contact
 Solutions for direct eye contact on prototype level available
 Combined approach of multi-view stereo and patch sweeping
pp p p g
 Exploitation of graphic cards
 High-resolution, high quality rendering achieved
©
Dr. Oliver Schreer 16. Dec. 2010 32

33. Image Processing
Thank you for
your attention!
i !
Contact: Oliver.Schreer@hhi.fraunhofer.de
©
Dr. Oliver Schreer 16. Dec. 2010 33