1. 3D Video: From Stereo to Multi-View
TEWI - Kolloquium,
Universität Klagenfurt
Karsten Müller
Fraunhofer Institute for Telecommunications
Heinrich-Hertz-Institut Berlin
Einsteinufer 37
10587 Berlin +49 30 310 02 – 0
info@hhi.fraunhofer.de
Germany http://www.hhi.fraunhofer.de

2. Overview
• Introduction, scenarios, requirements
• 3D Video: Stereo Video + MVC
• 3D Video: 3D Video Formats & Coding
• International 3D media research and development
• Summary
Karsten Müller
Slide 2
18.03.2010 Universität Klagenfurt

3. 3D Video Introduction
• Provision of left and right eye view
• Added depth sensation
• Resolved 2D viewing
ambiguity
• Additional features
(e.g. free viewpoint,
depth-controlled object
insertion)
3D Video is more than
adding 1 dimension to 2D!
Karsten Müller
Slide 3
18.03.2010 Universität Klagenfurt

4. 3D Video Scenarios
3D cinema
• Increasing number of 3D productions
• First studios start to release all productions also in 3D
• Technology: stereoscopic 3D, glasses based
3D home entertainment
• Different types of displays available: stereoscopic, auto-stereoscopic
with 2 … N views
• Various technologies, input formats and display sizes
• Glasses based systems may not be acceptable
3D mobile
• auto-stereoscopic 2 view display with fixed viewing position
• Good 3D viewing in spite of small display sizes
Karsten Müller
Slide 4
18.03.2010 Universität Klagenfurt

5. Stereo Video Solution
Stereo- • Original Cameras = Views for Display
Capture
Production
Format
• Production format specification (e.g. SMPTE)
Post- • Rectification, color correction, format conversion
processing
Transport • Conventional Stereo Format, e.g. Side-by-side
Format
MVC-
Coding
Transport • Multi-view video coding (MPEG-4 MVC/H.264)
Format
MVC-
Decoding
Display
Format
Karsten Müller
FolieStereo-
5
• High-resolution glasses-based stereo display
Display
18.03.2010
© Fraunhofer HHI

6. 3D format: Conventional Stereo video (CSV)
Original Image Source: Interactive Visual Media Group
Karsten Müller
of Microsoft Research
Slide 6
18.03.2010 Universität Klagenfurt

7. Current Coding Standards for Stereo Video
• Advanced Video Coding (MPEG-4 AVC / H.264)
– Simulcast
– Stereo SEI (Supplementary Enhancement Information)
– Auxiliary Picture Syntax
• MPEG-C part 3 (container format for V+D, e.g. with AVC
coding)
• Multi-view Video Coding (MPEG-4 MVC / H.264)
Karsten Müller
Slide 7
18.03.2010 Universität Klagenfurt

8. Multi-view Video Coding
• Joint Coding of camera views with similar content
Karsten Müller
Slide 8
18.03.2010 Universität Klagenfurt

9. MVC Coding Method
• Coding approach for exploiting temporal and inter-view
dependencies using MPEG4-AVC/H.264
• Usage of hierarchical B pictures in temporal direction
• Usage of P or hierarchical B pictures in inter-view direction
• Frame reordering to optimize memory usage
No H.264 changes required for multi-view coding
• Standardized in July 2008
• Adopted for 3D Blu Ray in Dec. 2009
Karsten Müller
Slide 9
18.03.2010 Universität Klagenfurt

10. MVC Coding Structure- Picture Reordering
Karsten Müller
Slide 10
18.03.2010 Universität Klagenfurt

11. MVC Coding Structure
Karsten Müller
Slide 11
18.03.2010 Universität Klagenfurt

12. MVC Camera Setting Adjustment - Star
Karsten Müller
Slide 12
18.03.2010 Universität Klagenfurt

13. MVC Camera Setting Adjustment – 2D Array
Karsten Müller
Slide 13
18.03.2010 Universität Klagenfurt

14. MVC Coding Results
• Coding structure allows AVC coder, to
select the best inter-view-temporal
neighbors within the MVC sequence
• Coding gain obtained by usage of
hierarchical B-frames and by exploitation of
inter-view dependencies
Karsten Müller
Slide 14
18.03.2010 Universität Klagenfurt

15. MVC Coding Results – delta_PSNR
Karsten Müller
Slide 15
18.03.2010 Universität Klagenfurt

16. Short Term 3DTV will be Stereo TV
anaglyph polarized
shutter
Broadcasters currently concentrate on stereo and will
not invest in new infrastructures
They will adopt a system, which they can feed through
existing pipes: e.g. anaglyph or multiplexed stereo:
checkerboard
side-by-side
above-below
etc.
Will Stereo TV be the long term solution?
Karsten Müller
Slide 16
18.03.2010 Universität Klagenfurt

17. The Answer is Given by Our Chancellor
Karsten Müller
Slide 17
18.03.2010 Universität Klagenfurt

18. Special Requirements for 3DTV
3DTV viewing conditions are more complex than in 3D cinema:
• Glasses are generally not accepted in the living room
• There are many different 3D display technologies
• Varying viewing conditions (distance, display size)
• User preferences have to be considered
• Not all users can see stereo (≈10%)
• Not all users want to see stereo
• Users might not want to see all programmes in stereo
Karsten Müller
Slide 18
18.03.2010 Universität Klagenfurt

19. Multi-view Displays
• The most common form of multi-view displays utilise either lenticular
screens or parallax barriers.
• Lenticular screens consist of a series of vertically aligned or slanted
cylindrical lenses and parallax barriers vertically aligned apertures.
• Light is guided in the appropriate directions by either focussing it or by
blocking unwanted rays.
VIEW 1 VIEW 2 VIEW 3
VIEW 1 VIEW 2 VIEW 3
1 41 41 4
32 1
LENTICULAR PARALLAX BARRIER
Karsten Müller
SCREEN
Slide 19
18.03.2010 Image Source: De Montfort University, UK

20. 3D Video Color only: Assumption 1
• Multi-view Displays require N views (e.g. 5,9,22,50)
• Why not transmitting N color views with MVC?
Karsten Müller
Slide 20
18.03.2010 Universität Klagenfurt

21. MVV and MVC Restrictions
• Pure coding of multi-view video
• Still linear dependency of bit rate from number of
views
• Number and position of views fixed over the whole
3D chain
• No baseline adaptation for different displays
• Number of Views fixed in MVV/MVC
Karsten Müller
Slide 21
18.03.2010 Universität Klagenfurt

22. MVC Restrictions
• Coding experiments on camera density
Original camera distance
Karsten Müller
Slide 22
18.03.2010 Universität Klagenfurt

23. MVC Restrictions (2)
• Coding experiments on camera density with Rena test
sequence
average rate average rate per camera
Karsten Müller
Slide 23
18.03.2010 Universität Klagenfurt

24. 3D Video Color only: Assumption 2
• Multi-view Displays require N views (e.g. 5,9,22,50)
• Why not transmitting 2 or 3 color views with MVC
and reconstructing N views at the receiver?
Karsten Müller
Slide 24
18.03.2010 Universität Klagenfurt

25. Color Only Intermediate View
Karsten Müller
Original Image Source: Interactive
Slide 25
Visual Media Group of Microsoft Research
18.03.2010

26. Depth map
Color information 256 depth layers
Original Image Source: Interactive Visual Media Group
Karsten Müller
of Microsoft Research
Slide 26
18.03.2010 Universität Klagenfurt

27. Example Reconstruction
Pure image blending Depth image based
rendering
Original Image Source: Interactive Visual Media Group
Karsten Müller
of Microsoft Research
Slide 27
18.03.2010 Universität Klagenfurt

28. Advanced 3D Video Solutions Required
• Color-only multi-view video too restricted
– High number of views cannot be efficiently
compressed and transmitted
– View synthesis of new views at the receiver
side from color-only data gives bad
reconstruction quality
• Additional Scene Geometry information required,
e.g. per-pixel depth/disparity data
Karsten Müller
Slide 28
18.03.2010 Universität Klagenfurt

29. Depth-enhanced 3D Video
Capture • Stereo / multiview capturing, depth provision
Production
Format
• Production format specification (e.g. SMPTE)
Post- • Rectification, color correction, format conversion
processing
Transport • 3D format specification
Format
• 3D video coding (e.g. MPEG)
Coding
Transport
Format
View
Synthesis
• High quality intermediate view synthesis
(e.g. 2 view + 2 depth N view)
Display
Format
Karsten Müller
Folie 29
Display
18.03.2010
© Fraunhofer HHI
• High-resolution stereo/multiview display

30. 3D Video Formats
• Conventional stereo video (CSV)
• Mixed resolution stereo (MRS)
• Video plus depth (V+D)
• Multiview video (MVV)
• Multiview video + depth (MVD)
• Layered depth video (LDV)
• Depth-enhanced stereo (DES)
Karsten Müller
Slide 30
18.03.2010 Universität Klagenfurt

31. assuming multiple views with color & depth…
Karsten Müller
Slide 31
18.03.2010 Universität Klagenfurt

32. Stereo video (CSV)
CSV
Karsten Müller
Slide 32
18.03.2010 Universität Klagenfurt

33. Mixed resolution stereo (MRS)
MRS
Karsten Müller
Slide 33
18.03.2010 Universität Klagenfurt

34. Video plus depth (V+D)
V+D
Karsten Müller
Slide 34
18.03.2010 Universität Klagenfurt

35. Multiview video (MVV)
MVV
Karsten Müller
Slide 35
18.03.2010 Universität Klagenfurt

36. Multiview video plus depth (MVD)
MVD
Karsten Müller
Slide 36
18.03.2010 Universität Klagenfurt

37. Layered depth video (LDV)
LDV
Karsten Müller
Slide 37
18.03.2010 Universität Klagenfurt

38. Depth-enhance Stereo (DES)
DES
Karsten Müller
Slide 38
18.03.2010 Universität Klagenfurt

39. Towards a new 3D Video Coding Standard
• MPEG develops a new 3D video coding standard
• Motivation for new standard:
– Decouple production from coding format
– MPEG-4 AVC/H.264 only optimized for 2D color video, but not for
depth information
Karsten Müller
Slide 39
18.03.2010 Universität Klagenfurt

40. Advanced 3DTV concept based on MVD
Karsten Müller
Slide 40
18.03.2010 Universität Klagenfurt

41. Coverage of 3D Video Coding
Capture
Production
Format
Post-
processing
Transport
Format
Coding 3D components under consideration:
– Transport format
Transport
Format – 3D video coding methods
– Intermediate view synthesis
View
Synthesis
Display
Format
Karsten Müller
Folie 41
Display
18.03.2010
© Fraunhofer HHI

42. Challenges for 3DVC
• Consider capturing technology, i.e. maximal 2-3 recorded views
• Break linear dependency of coding bit rate from number of target views
(e.g. MVC)
• Provide scene geometry data in general form, i.e. pixel-wise depth data
• Consider statistical properties of depth (and supplementary) data
• Consider new quality evaluation methods for intermediate views
• Provide high-quality view synthesis for continuous viewing range
• Decrease depth and coding errors
Karsten Müller
Slide 42
18.03.2010 Universität Klagenfurt

43. Coding Experiments MVD with MVC
• Joint color and depth coding for optimal
intermediate view quality
• Evaluation of total bit rate (e.g. of 2 color and 2
depth maps) vs. quality of decoded synthesized
views
• View synthesis is part of coding optimization
• Reduction of coding and interpolation artifacts
through high-quality view synthesis
Karsten Müller
Slide 43
18.03.2010 Universität Klagenfurt

44. Coding Results Intermediate View
• Ballet (JMVM 7.0.1, GOP16, κ=½)
Karsten Müller
Slide 44
18.03.2010 Universität Klagenfurt

45. Coding Results Intermediate View
• Breakdancers (JMVM 7.0.1, GOP16, κ=½)
Karsten Müller
Slide 45
18.03.2010 Universität Klagenfurt

46. Color Depth Bit Rate Distribution
• Objective and subjective examples for optimal and
non-optimal bit rate distribution
• Variation of color and depth quality
• Tradeoff between synthesized views and views at
original positions
Karsten Müller
Slide 46
18.03.2010 Universität Klagenfurt

47. Coding Results Intermediate View
• Ballet (JMVM 7.0.1, GOP16, κ=½)
C30D30
C24D40
Karsten Müller
Slide 47
18.03.2010 Universität Klagenfurt

48. Coding Results across Viewing Range
• Ballet (JMVM 7.0.1, GOP16)
Karsten Müller
Slide 48
18.03.2010 Universität Klagenfurt

49. Coding Results Intermediate View
• Breakdancers (JMVM 7.0.1, GOP16, κ=½)
C30D36
C36D30
Karsten Müller
Slide 49
18.03.2010 Universität Klagenfurt

50. Coding Results across Viewing Range
• Breakdancers (JMVM 7.0.1, GOP16)
Karsten Müller
Slide 50
18.03.2010 Universität Klagenfurt

51. View Synthesis for MVD
– High quality view
interpolation
is essential for 3D video
applications with MVD data.
– Processing steps:
- Layer Extraction
- Layer Projection
- Hole Filling + Filtering
Karsten Müller
Slide 51
18.03.2010 Universität Klagenfurt

52. Synthesis Improvements (Ballet)
simple merging
layer-based
view synthesis
compressed uncompressed
Karsten Müller
Slide 52
18.03.2010 Universität Klagenfurt

53. International R&D in 3D Media
• ISO/IEC MPEG 3DVC (3D video coding)
• SMPTE Task Force of 3D to the home (3D master format)
• ITU-R SG6
• DVB – TM - ES Stereoscopic (3D) TV
• 3D@Home Consortium
• Association in Korea, Japan,…
• EU 3D, Immersive, Interactive Media Cluster
• International Conferences
– 3DTV-CON
– 3DPVT
Karsten Müller
Slide 53
18.03.2010 Universität Klagenfurt

54. 3D Video Coding Group in MPEG
• Nagoya Univ. • Samsung • Zhejiang Univ.
• NTT • LG Electronics • ASTRI
• UPM • Nokia • Qualcomm
• Philips • NICT
• GIST • Kwangwoon Univ.
• ETRI • Sharp
• JVC • Fraunhofer
• Poznan Univ. of • NCTU/ITRI
Technology • Motorola
• Thomson • NXP
• MERL • Ericsson
• Peking Universtity • Logitech
• Tsinghua Univ. • Orange/France • Coordination:
• Huawei Telecom • Karsten Müller (HHI)
• Xidian University • FUB Anthony Vetro (MERL)
• Sony
Karsten Müller
• Telefonica
Slide 54
18.03.2010 Universität Klagenfurt

55. 3D@Home Consortium
• Steering Team 1: Content Creation & Production
Objective: To focus on projects which improve the quality of 3D content for the home audience.
• Steering Team 2: Content Storage, Transmission & Distribution
Objective: To develop useful definitions and guidelines for the successful storage, transmission and
distribution of 3D content.
• Steering Team 3: 3D Promotion
Objective: To promote the adoption of 3D into the home by promoting 3D format and the 3D@Home
Consortium.
• Steering Team 4: 3D Displays
Karsten Müller
Objective: To identify issues around 3D Displays and associated hardware.
Slide 55
18.03.2010 Universität Klagenfurt

56. 3D@Home Consortium
Karsten Müller
Slide 56
18.03.2010 Universität Klagenfurt

57. EU 3D, Immersive, Interactive Media Cluster
• Currently, a number of EU projects develop parts and products of the 3D
media chain for certain application areas, e.g. 3D home entertainment or
mobile services
• Enhanced success of these EC funded projects and subsequent associated
commercialization efforts if simultaneous operations of individual projects can
be further orchestrated towards a strong mutual presence
• Better highlight of Europe‘s good position in worldwide R&D
• 3D, Immersive, Interactive Media Cluster is an umbrella structure for the
projects and provides international contact and information about project
activities
• 3D Media, Immersive, Interactive Cluster is one of the main drivers towards
"3D Media" in the context of the "Future Internet" activities supported by EU
Commission (DG INFSO)
Karsten Müller
Slide 57
18.03.2010 Universität Klagenfurt

58. 3DII Media Cluster – Current Projects
• Leadership: Prof. Levent Onural and Dr. Karsten Müller
• Projects: Status: Framework Program
• 3DTV ended 2008 FP6
• 3DPHONE ongoing FP7 3D for Mobile
• MOBILE3DTV ongoing FP7 Systems
• 3D4YOU ongoing FP7 3D for Broadcast
• 2020 3D Media ongoing FP7 Systems
• 3DPresence ongoing FP7 3D Videoconference
• MUTED ended 2008 FP6
• HELIUM3D ongoing FP7 3D Display
• Real3D ongoing FP7 Technology
• i3DPost ongoing FP7 3D Content
• VICTORY ended 2009 FP7 Description
Karsten Müller
Slide 58
18.03.2010 Universität Klagenfurt

59. 3DII Media Cluster – New Projects
• Projects: Status: Framework Program
• DIOMEDES new 2010 FP7
• MUSCADE new 2010 FP7
• 3D VIVANT new 2010 FP7
• BEAMING new 2010 FP7
• FINE new 2010 FP7
• SKYMEDIA new 2010 FP7
• SALA3D new 2010 FP7
• 3DLife new 2010 FP7
• FascinatE new 2010 FP7
Karsten Müller
Slide 59
18.03.2010 Universität Klagenfurt

60. Summary
• 3D technology is maturing due to world wide development
from capturing to display
• First systems are in use, mainly using stereoscopic
displays and MVC coding
• New challenging research topics in all areas of 3D video:
– Reliable capturing devices with synchronized views
– Improved time-consistent depth estimation
– Geometry-enhanced multi-view coding (e.g. 3DVC)
– Robust view synthesis
– New high-resolution multi-view displays
(e.g. 50 views, each with HDTV resolution!)
Karsten Müller
Slide 60
18.03.2010 Universität Klagenfurt