MPEG-DASH provides an interoperable representation format for the adaptive delivery of multimedia content over the top of existing infrastructures (first edition: April 2012; second edition: May 2014). The MPEG-DASH standard is agnostic to the coding format and the DASH Industry Forum (DASH-IF) defines interoperability points for its usage with common codecs such AVC/H2.64, HEVC/H.265, and AAC. In practice, however, the content is being delivered over best effort networks, typically without any guarantees from the underlying network infrastructure. Thus, the characteristics of the end-to-end delivery environment may change dynamically, specifically in the available network conditions (e.g., bandwidth). Additionally, a variety of devices with different capabilities and users with personalized preferences are accessing these services. Hence, the quality as perceived by the end user becomes more and more important which is referred to as the Quality of Experience (QoE). This provides a definition for the QoE in the context of MPEG-DASH-based services. In particular, it will discuss means how to evaluate the QoE for MPEG-DASH-based services both using objective and subjective metrics. The core of the paper will be a crowdsourced user study how to evaluate different DASH solutions and implementations on a large scale.

1. Ultra-High-Definition Quality of
Experience with MPEG-DASH
Priv.-Doz. Dr. Christian Timmerer
Daniel Weinberger, Christopher Mueller, and Stefan Lederer
Chief Innovation Officer (CIO) at bitmovin GmbH
http://www.bitmovin.com  christian.timmerer@bitmovin.com
Alpen-Adria-Universität Klagenfurt (AAU)  Faculty of Technical Sciences (TEWI)  Department of Information
Technology (ITEC)  Multimedia Communication (MMC)  Sensory Experience Lab (SELab)
http://blog.timmerer.com  http://selab.itec.aau.at/  http://dash.itec.aau.at  christian.timmerer@itec.aau.at
http://www.slideshare.net/christian.timmerer

2. Outline
• Introduction
• Quality, Quality of Experience, and DASH
• Evaluation strategies
• Results
• Conclusions
• Acknowledgment [some slides]: Ali C. Begen, CISCO
April 16, 2015 NAB2015 - BEC - QoE-DASH 2

3. Introduction
• Real-time entertainment
– Streaming video and audio
– > 60% of Internet traffic
• All delivered over-the-top (OTT)
• MPEG Dynamic Adaptive
Streaming over HTTP (DASH)
– Coding format agnostic
• DASH Industry Forum
– Interoperability Points (IOPs) for
common codecs and others (v3.0)
– E.g., AVC/H.264, HEVC/H.265, and
AAC
April 16, 2015 NAB2015 - BEC - QoE-DASH 3

4. Over-The-Top – Adaptive Media Streaming
April 16, 2015 NAB2015 - BEC - QoE-DASH 4
Adaptation logic is within the
client, not normatively specified
by the standard, subject to
research and development

5. Open Digital Media Value Chain
April 16, 2015 NAB2015 - BEC - QoE-DASH 5
Create
Content
Aggregate
Monetize
Distribute
Content
Consume
Content
Any Content Any Storefront Any Network Any Device
CDNsMedia
Protocols
Internet
Transport
DRM
Encoding
Encapsulation
Dynamic
Ads
Clients

6. Simplified Example Workflow: bitcodin/bitdash
April 16, 2015 NAB2015 - BEC - QoE-DASH 6
Source: http://www.bitmovin.net/bitcodin-cloud-based-transcoding-streaming-platform/

7. Internet TV vs. Traditional TV in 2010
• Areas most important to
overall TV experience are
– Content
– Timing control
– Quality
– Ease of use
• While traditional TV surpasses
Internet TV only in quality, it
delivers better “overall
experience”
April 16, 2015 NAB2015 - BEC - QoE-DASH 7
When comparing traditional and Internet TV,
which option is better?
Traditional Internet
Content 7%  79%
Timing / Control 7%  83%
Quality  80% 16%
Ease of Use 23%  52%
Control (FF, etc.) 9%  77%
Portability 4%  92%
Interactivity 31%  52%
Sharing 33%  56%
Overall Experience  53% 33%
Source: Cisco IBSG Youth Survey, Cisco IBSG Youth Focus Group Sessions, 2010

8. Quality (of Experience)
• QoE as evolution of QoS [ITU-T
P.10/G.100]
• QoS: totality of characteristics
of a telecommunications
service that bear on its ability
to satisfy stated and implied
needs of the user of the service
• QoE: the overall acceptability
of an application or service, as
perceived subjectively by the
end-user
April 16, 2015 NAB2015 - BEC - QoE-DASH 8
Many definitions but in general, it’s like an elephant

9. Quality of Experience
• COST Action IC1003 – QUALINET (http://www.qualinet.eu/)
“the degree of delight or annoyance of the user of an application or service. It results
from the fulfillment of his or her expectations with respect to the utility and/or
enjoyment of the application or service in the light of the user’s personality and
current state”
• QoE influence factors
– Any characteristic of a user, system, service, application, or context
– Grouped into human, system, and context
• QoE features
– Perceivable, recognized and namable characteristic of the individual’s experience
– Depends on the level of direct perception, interaction, the usage situation
April 16, 2015 NAB2015 - BEC - QoE-DASH 9

10. QoE for DASH
• Different application domains have different QoE
requirements
– Need to provide specializations of the general QoE definition
– Take into account requirements formulated by means of
influence factors and features of QoE
• QoE influence factors for DASH
– Initial/start-up delay (low)
– Buffer underruns, stalls, freezes (zero)
– Quality switches (low)
– Media throughput (high)
– …
April 16, 2015 NAB2015 - BEC - QoE-DASH 10

11. QoE Evaluation for DASH-based Services
• Test sequence
– Many datasets available
– Adopted Big Buck Bunny & DASHed it with bitcodin
• Players
– bitdash
– …and compare it with ten different adaptation algorithms
• Objective evaluation
– Test setup
– Predefined bandwidth trajectory (or real network traces)
• Subjective evaluation
– Lab vs. crowdsourcing
April 16, 2015 NAB2015 - BEC - QoE-DASH 11
http://www.bitcodin.com/
http://www.dash-player.com/

12. Objective Evaluations
April 16, 2015 NAB2015 - BEC - QoE-DASH 12
Stalls (lower is better)Average Bitrate (higher is better)

13. Stalls are really bad…
April 16, 2015 NAB2015 - BEC - QoE-DASH 13
Conviva: Viewer Experience Report. 2014

14. DASH-JS vs. bitdash
April 16, 2015 NAB2015 - BEC - QoE-DASH 14

15. Subjective Evaluation
• Microworker platform
– Limited to Europe, USA/Canada, India
• DASH clients
– DASH-JS (dash.itec.aau.at)
– dash.js (DASH-IF)
– YouTube
• Tears of Steal trailer according to YouTube
configuration
• Screening techniques
– Browser fingerprinting
– Presentation time
– QoE ratings and Pre-Questionnaire
April 16, 2015 NAB2015 - BEC - QoE-DASH 15

16. What about 4K and 8K?
April 16, 2015 NAB2015 - BEC - QoE-DASH 16
• Why? – because we can!
• Supported on the Web
– HTML5, MSE
– AVC/H.264
– [HEVC/H.265 needed
to lower bitrate]
• See demo @
http://www.dash-
player.com/
• UHD-QoE evaluation

17. Conclusions
• QoE for DASH-based services (a rule of thumb)
– Startup delay (low [but live vs. on-demand & short vs. long-tail
content])
– Buffer underrun / stalls (zero)
– Quality switches (low) and media throughput (high)
– Energy- and cost-awareness (data plan)
• No general applicable QoE model for DASH
– (Too) many factors influencing / features of QoE for DASH-based
services
– Methodology for reproducible research is in place and well established
– Ample research opportunities
April 16, 2015 NAB2015 - BEC - QoE-DASH 17
Main QoE
factors for DASH

18. Thank you!
April 16, 2015 NAB2015 - BEC - QoE-DASH 18
Source: http://www.bitmovin.net/bitcodin-cloud-based-transcoding-streaming-platform/