MPEG for the past, present and future of television

1. MPEG for the
past, present and future
of television
Leonardo Chiariglione
Keynote Speech at ATSC Annual Meeting
Washington, DC – 2012/05/08

2. MPEG for the past of
television
 System
 Transport Stream
 DSM-CC
 Timed metadata
 Video
 MPEG-2 High Profile
 Audio
 MPEG-1 Audio Layer II
 MPEG-2 AAC
2012/05/02
2
MPEG for the past, present and future of TV

3. An assessment
 MPEG has provided the means for the television
business on air, cable and satellite to migrate from the
analogue to the digital age
 It has been a collective efforts
 Representatives from the entire spectrum of the
broadcasting world provided their requirements
 Packages of patent licences were created (outside of
MPEG)
 Today there is virtually no broadcasting system that is
not based, at least partially, on MPEG standards
MPEG for the past, present and future of TV
3
2012/05/08

4. MPEG for the present of
television
 Systems
 Carriage of MPEG-4 on MPEG-2 TS
 Dynamic Adaptive Streaming over HTTP (DASH)
 Video
 MPEG-4 Advanced Video Coding (AVC)
 3D extensions
 Audio
 Various forms of Advanced Audio Coding (AAC)
 Universal Speech and Audio (USAC)
2012/05/08
4
MPEG for the past, present and future of TV

5. An assessment
 The deployment of 2nd generation digital broadcasting
systems continues successfully
 Use of the AVC on MPEG-2 TS allows packing twice as
many programs as MPEG-2 Video in an analogue
channel
 With IPTV the notion of broadcasting is expanding
 MPEG is helping the broadcasting industry
 To enrich the broadcasted user experience
 Through standard technologies that serve multiple fields
MPEG for the past, present and future of TV
5
2012/05/08

6. MPEG for the future of
television
 Well, that’s what I am supposed to talk about…
2012/05/08
6
MPEG for the past, present and future of TV

7. Broadcasting is a great
business to be in…
 Broadcasting is a proven business with a constant
revenue flow
 Many technologies appear that
 Can help new competing businesses but
 Can also be used to extend the broadcasting business
 MPEG is a neutral body devoted to the development of
standard technologies for user communities to exploit
MPEG for the past, present and future of TV
7
2012/05/08

8. MPEG-H: a video standard for
future broadcasting
 ISO/IEC 23008 High Efficiency Coding and Media
Delivery in Heterogeneous Environments
 A suite of standards for media coding and delivery
 Currently scheduled to appear in 3 parts
 MPEG Media Transport
 High Efficiency Video Coding
 3D Audio
 Timeline
MPEG for the past, present and future of TV
8
2012/05/08
Part CD DIS FDIS
MMT 12/07 13/01 13/07
HEVC 12/02 12/07 13/01
3D Audio 14/04 14/10 15/01

9. Service composition in MMT
3D Earth
UD contents
Bird is flying…
Bird is flying…
<subtitle> composition
information
friendly
with html5
legacy broadcasting future broadcasting
MMT Package
9
MPEG for the past, present and future of TV 2012/05/02

10. Clouds
MMT Encapsulator
- Service Component List
- Component location
- Composition relationship
- QoE requirement
- How to deliver
- How to consume
Web
video
file
- Configuration
- Adaptive delivery
- Sync, Delay, Loss control
MMT Frame
pull pull
Smart delivery in Clouds
10
2012/05/08

11. Mash-up service
Service 1 (MMT Package1) Service 2 (MMT Package2)
MMT
assets
Audio Video1 widget Web page
Video2
11
2012/05/08
MPEG for the past, present and future of TV

12. Second screen management
Additional Information with
Live Content (HTML)
Video content
on 2nd device
composition
information
<device 1>
…
<device2>
…
MMT Package
12
2012/05/08
MPEG for the past, present and future of TV

13. MPEG-H part 2
High Efficiency Video Coding
 Primary goal: ≥ 50% better compression than AVC
 For large displays (e.g. 4kx2k)
 For mobile (e.g. WVGA)
 Developed jointly with ITU-T by Joint Collaborative
Team (JCT) on Video Coding
 Technically in the same stream of past MPEG Video
Coding standards (AVC and prior standards)
 Currently at Committee Draft level
MPEG for the past, present and future of TV
13
2012/05/08

14. HEVC vs AVC Subjective
Performance Measurements
 Preliminary subjective tests carried out for
HEVC Committee Draft
 Reasonably small and non-overlapping
confidence intervals
 Average rate savings at same (interpolated)
MOS points
 HEVC with 67% lower rate in class B (Full HD)
 HEVC with 49% lower rate in class C (WVGA)
 HEVC with 58% lower rate overall
2012/05/08
MPEG for the past, present and future of TV
14

15. A performance sample
2012/05/08
MPEG for the past, present and future of TV
15

16. For those technology aware…
 Block-based/Variable block sizes
 Block motion compensation
 Fractional-pel motion vectors
 Spatial intra prediction
 Spatial transform of residual difference
 Integer-based transform designs
 Arithmetic or VLC-based entropy coding
 In-loop filtering to form final decoded picture
2012/05/08
MPEG for the past, present and future of TV
16

17. Where we are (May 2012)
 Activity: level of 1,000 documents per meeting crossed
 Very diverse participation from industry/academia
 Significant technical advance over prior standards
 Challenge: computational/implementation complexity
 Deliverables:
 Spec
 Reference software
 Conformance
 Currently only one profile
2012/05/08
MPEG for the past, present and future of TV
17

18. HEVC Scalability Extensions/1
 Serve the needs of heterogeneous environment of
future digital TV distribution
 Multiple devices: SHDTVs, set-top boxes, tablets, smart
phones, PCs etc.
 Wide range of processing powers, display sizes and power
consumption needs
 Multiple networks: Cable, Satellite, Terrestrial, CDN,
Cable Modem, xDSL, WiFi, 4G
 Wide range of channel bandwidths and QoS
 Multiple protocols: MPEG-2 TS, DASH, HLS, RTP, UDP
 Multiple services: Broadcasting, On demand, Streaming,
Over The Top
2012/05/08
MPEG for the past, present and future of TV
18

19. HEVC Scalability Extensions/2
 Options for backward compatible migration paths
 From 720p60 to 1080p60
 From1080p60 to 4kx2kx60
 From AVC to HEVC
 Time Line
 May 2012 Preliminary Call for Proposals (public)
 July 2012 Final Call for Proposals (public)
 October 2012
 Evaluation of proposals & start of collaborative design phase
 Development of the time line associated with standardisation
phases (WD, CD/PDAM, FDIS/FDAM, DIS/AMD)
2012/05/08
MPEG for the past, present and future of TV
19

20. Service Transitions
 1950s: Color TV
 Analog, backward compatible
 1990s: Digital TV
 New infrastructure required
 Transitions from SD to HD
 2010s: 3D
 Mixed results from services introduction is
 Not a single format across all services
2012/05/08
MPEG for the past, present and future of TV
20

21. Current Status of 3D Video
 Increasing investments in 3D production and services
 Increasingly capable 3D displays in the market
(many competing and emerging technologies)
 Market is primarily stereo
(mix of different formats being deployed right now)
 Focus of current MPEG work: development of a new
3D format and associated compression techniques that
could facilitate generation of multiview output
2012/05/08
MPEG for the past, present and future of TV
21

22. Existing 3DV Coding Formats
 Pack pixels from left and right views
into a single frame (loss of
resolution)
 Leverage existing infrastructure and
equipment, only require additional
signaling to de-interleave frame
 Embraced by broadcasters for first
phase of 3D broadcast services
 Full-resolution coding of stereo and
MV video as extension of AVC
 High coding efficiency achieved via
inter-view prediction techniques
 2D compatibility supported
 Adopted as format for 3D Blu-ray
Disc
 Being considered for second phase
of broadcast standards
2012/05/08
MPEG for the past, present and future of TV
22
Left Right
Left View Right View
Effective prediction between views for high compression performance
Left View Right View
Effective prediction between views for high compression performance

23. Targets of Future 3DV Format
2012/05/08
MPEG for the past, present and future of TV
23
Data
Format
Data
Format
Constrained Rate
(based on distribution)
Limited
Camera
Inputs
• Wide viewing angle
• Large number of
output views
Left
Right
Auto-stereoscopic
N-view displays
Stereoscopic displays
• Variable stereo baseline
• Adjust depth perception

24. 3DV Reference Framework
Limited # Video Inputs
(e.g., 2 or 3 views)
Depth
Estimation
Video/Depth
Encoder
Decoder and
View Synthesis
Larger #
OutputViews
Binary Representation
& ReconstructionProcess
+
1010001010001
24
2012/05/08
MPEG for the past, present and future of TV

25. Overview of Call For
Proposals
 Test Material & Conditions
 8 test sequences (1920x1080 & 1024x768)
 4 target bit rates per sequence
 2-view and 3-view test scenarios
 AVC and HEVC compatible test categories
 23 responses (12 AVC + 11 HEVC)
 Evaluation in Nov 2011
 Objective quality (PSNR of synthesized views)
 Subjective assessment (stereo and auto-stereo)
2012/05/08
MPEG for the past, present and future of TV
25

26. 3DV Testing Scenarios
2012/05/08
MPEG for the past, present and future of TV
26
Left
Right
Center
Depth
Estimation
3D Video
Codec
Multiview
Rendering
virtual
views
Output to
Auto-Stereo
N-view
Display
Left
Right
Center
Depth
Estimation
3D Video
Codec
Multiview
Rendering
virtual
views
Output to
Auto-Stereo
N-view
Display
Left
Right
Depth
Estimation
3D Video
Codec
Multiview
Rendering
virtual
view
Output to
Stereo
Display
Left
Right
Depth
Estimation
3D Video
Codec
Multiview
Rendering
virtual
view
Output to
Stereo
Display
2-view Configuration
3-view Configuration

27. Tool Categories
 Texture coding
 Independent of depth
 E.g., inter-view prediction of color view, inter-view prediction
of motion parameters and residual data
 Using depth data
 E.g., view synthesis prediction, motion prediction
 Depth coding
 Independent of texture
 E.g., depth modeling modes, weighted prediction, reduced
resolution coding
 Using texture data
 E.g., motion parameter inheritance, intra prediction
2012/05/08
MPEG for the past, present and future of TV
27

28. 3DV Current Status
 Extensive subjective evaluation of
20+ proposals to the 3DV CfP
 Considered both AVC and HEVC
compatible
 Many new coding tools proposed
for MV texture and depth coding
 Substantial rate savings compared
to capabilities of existing standards
 AVC and HEVC extensions
standardisation under wayd
 Add support for depth
 New AVC-based coding tools
 MV extensions of HEVC
 Hybrid architectures
2012/05/08
MPEG for the past, present and future of TV
28

29. Hybrid Architectures
 From a pure compression efficiency point of view, it is
always best to use the most advanced codec
 However, when introducing new services, providers
must also consider capabilities of existing receivers
and transition plan
 Use Case:
 Many terrestrial broadcast systems based on MPEG-2
 US cable systems based on mix of MPEG-2 and AVC
 May not be easy to simply switch codecs in near term
2012/05/08
MPEG for the past, present and future of TV
29

30. Multiview Extensions of
HEVC
 HEVC Stereo/MV extensions being considered
 Extensions expected to be completed approximately 12
months after the base specification, i.e., Jan 2014
 Reportedly simple extensions of HEVC can achieve 30-
40% bit rate reduction compared to HEVC simulcast
 Further gains expected if block-level changes to codec
are considered
2012/05/08
MPEG for the past, present and future of TV
30

31. MPEG-H part 3: 3D Audio
 Well, there is no Systems and Video without Audio…
2012/05/08
MPEG for the past, present and future of TV
31

32. 3D Audio requirements/1
 High quality: quality of decoded sound perceptually
transparent
 Localization and Envelopment: Accurate sound
localization and very high sense of sound envelopment
within a targeted listening area (“sweet spot”)
 Flexible Loudspeaker Placement: the transmitted
audio program to be mapped to a rendering setup in
which loudspeakers are in alternate locations
 Interactivity: Interactive modification of the sound
scene rendered from the coded representation
MPEG for the past, present and future of TV
32
2012/05/08

33. 3D Audio requirements/2
 Rendering on setups with fewer loudspeakers
(including headphones): ability to derive a signal from
the transmitted program material for reproduction with
fewer loudspeakers
 Audio/visual alignment & consistency: ability to
adapt the rendered acoustic scene to be consistent
with the visual
 Efficiency for decoding on different setups: ability to
be rendered on all reproduction loudspeaker setups or
headphone
MPEG for the past, present and future of TV
33
2012/05/08

34. 3D Audio requirements/3
 Appropriate computational complexity and system
latency for the target application (Broadcasting, Spatial
two-(n-)way communication / telepresence
 Transcoding for low bandwidth devices: A lower
bandwidth signal can be extracted from the original
program material
 Backward compatibility: (e.g. to 5.1 channel
programs and decoders or transmission of HRTF
encoded signals)
MPEG for the past, present and future of TV
34
2012/05/08

35. Envisioned Architecture
 Inputs
 Audio Channels
 “normal” content
 Audio Objects
 “helicopter overflight”
 Audio Scene
 Ambisonics
35
3D Audio
Encoder
500 kb/s
bitstream
3D Audio
Encoder
Transcoder
64 kb/s
bitstream
22.2 Channel
Audio Program
Audio Objects
Audio Scene
Smart
Phone
TV
Home
Theatre
Tablet
TV
22.2 Loudspeakers
Spatialized on Headphones
In-Frame Speakers
Spatialized on Headphones
Spatialized on
Headphones
• Outputs
– Loudspeakers
– Headphones
• Spatialized sound
– In-frame loudspeakers
• “Sound Bar”
• Channel
– 500 kb/s primary
channel
– 64 kb/s cellular channel
MPEG for the past, present and future of TV 2012/05/08

36. Audio, part of
an Audio-visual Experience
 What is new in video?
 Larger, high-resolution displays
 Greater sense of envelopment
 Possibly closer viewing distance
 More efficient compression
 Wireless transmission to portable devices
 Immersive and enveloping audio experience
36
2012/05/08
MPEG for the past, present and future of TV

37. Use Case 1
Home Theatre
 Many loudspeakers
 10.1
 22.2
 “3D” Experience
 Height loudspeakers for greater envelopment
 Sense that audio objects are near or distant
 When near, they are consistent with video image
37
2012/05/08
MPEG for the past, present and future of TV

38. 22.2 multichannel sound
system (NHK)
2012/05/08
MPEG for the past, present and future of TV
38
TV Screen
Top layer
9 channels
Middle layer
10 channels
Bottom layer
3 channels
LFE
2 channels
TpFL TpFC TpFR
TpSiL
TpC
TpSiR
TpBL
TpBC
TpBR
BL
BC
BR
SiL
FL
FLc FC FRc
FR
SiR
BtFL
BtFC
BtFR
LFE1 LFE2
TV Screen
Top layer
9 channels
Middle layer
10 channels
Bottom layer
3 channels
LFE
2 channels
TpFL TpFC TpFR
TpSiL
TpC
TpSiR
TpBL
TpBC
TpBR
BL
BC
BR
SiL
FL
FLc FC FRc
FR
SiR
BtFL
BtFC
BtFR
LFE1 LFE2

39. Home Theater – Issues
 Maintain compatibility with legacy systems: 5.1, stereo
 Will content providers really step up to 22.2?
 Will consumers adopt 22.2?: 2 front high
 How to compress 22.2 for transmission?
 MPEG AAC 64 kb/s/channel: -> 1.5 mb/s
 How to render for legacy setups?
 Relatively slow-growth market
2012/05/08
MPEG for the past, present and future of TV
39

40. Use Case 2
Personal TV
 Small but high-resolution display
 “super-tablet”
 Local wireless communications
 To cable or fiber home hub
 Possible audio presentation
 Headphones
 Loudspeakers around perimeter of display
 Issue: To what extent can the user get an enveloping
experience from only “front” loudspeakers?
40
MPEG for the past, present and future of TV 2012/05/08

41. Use Case 3
Mobile TV
 Hand-held display: Smartphone
 Headphone listening:
 Stereo, perhaps with binauralisation
 Fastest growing market: Quick rollout and adoption of
standard
 What is new?
 USAC for stereo; MPEG Surround for binauralisation
 What is the “Wow” factor?
 To spur adoption of new technology
41
MPEG for the past, present and future of TV 2012/05/08

42. User Experience
42
Home Theatre
Tablet TV
Spatialized on headphones
2012/05/08
MPEG for the past, present and future of TV

43. Flexible Rendering
 Content providers create 22.2 program
 Consumed on stereo, 5.1, 10.1, 22.2 layouts
 Consumed on “wrong” layouts
 Mis-positioned surrounds
 Missing surrounds
 Non-standard layouts
 2 front high, 5.1 mid
43
MPEG for the past, present and future of TV 2012/05/08

44. 3D Audio Test Platform
 NHK “Loudspeaker
Array Frame” (LAF)
 Suggestions for
 Alternative platform
for assessing “Home
Theatre” use case
 Model for assessing
“Personal TV” use
case
44
MPEG for the past, present and future of TV 2012/05/02

45. Summary
 Provide compelling immersive audio experience for
audio/visual programs
 Create content once, present on many different
loudspeaker layouts or on headphones
 Provide high compression and high-quality audio
presentation
45
MPEG for the past, present and future of TV 2012/05/08

46. MPEG-V: Humans have
more than two senses
2012/05/08
MPEG for the past, present and future of TV
46

47. MARBle
Media
Servers
Service
Servers
User
Local
Sensors &
Actuators
Remote
Sensors &
Actuators
ARAF file
or stream
Local
Real World
Environment
Remote
Real World
Environment
Augmented Reality
Application Format (ARAF)
MPEG for the past, present and future of TV 2012/05/08
47

48. Augmentation Region in
ARAF
Augmentation
Region
Broadcaster
AR service
provider A
AR service
provider B
User A
User B
MPEG for the past, present and future of TV 2012/05/02
48

49. The past, the present
and the future, again
 25 years ago MPEG selected the digital media way
bringing innovation and interoperability
…and thrived
 This year the broadcasting industry has started a move
in future broadcast services with interoperability at its
core
…and it is bound to thrive
 MPEG looks forward to continuing a cooperation with
the broadcasting industry providing the necessary
standard technologies
2012/05/08
MPEG for the past, present and future of TV
49

50. 50
2012/05/08
MPEG for the past, present and future of TV
http://mpeg.chiariglione.org/