This document summarizes a presentation on evaluating the quality of experience (QoE) of adaptive HTTP streaming clients like DASH in real-world environments using crowdsourcing. The study evaluated three clients (DASH-JS, dash.js, YouTube) playing video content with different quality representations. Results found that DASH-JS had high startup times but low stalls and good throughput/QoE, while dash.js had low startup but high stalls and lower throughput/QoE. YouTube performed best with low startup, few stalls, and highest throughput and QoE. The methodology and results provide insights into QoE aspects of DASH clients over the open internet.

1. Slides:
hVp://www.slideshare.net/chris8an.8mmerer
Quality
of
Experience
of
Web-­‐based
Adap8ve
HTTP
Streaming
Clients
in
Real-­‐World
Environments
using
Crowdsourcing
Benjamin
Rainer
and
Chris8an
Timmerer
Alpen-­‐Adria-­‐Universität
Klagenfurt
(AAU)
w
Faculty
of
Technical
Sciences
(TEWI)
w
Department
of
Informa8on
Technology
(ITEC)
w
Mul8media
Communica8on
(MMC)
w
Sensory
Experience
Lab
(SELab)
h"p://blog.+mmerer.com
w
h"p://dash.itec.aau.at/
w
h"p://selab.itec.aau.at
mailto:chris+an.+mmerer@itec.uni-­‐klu.ac.at
December
2,
2014

2. Outline
• Introduc+on
• How
to
evaluate
DASH
and
QoE
• Methodology
• Results
• Conclusions
December
2,
2014
VideoNext
2014,
Sydney
2

3. Mul+media
is
Predominant
on
the
Internet
• Real-­‐+me
entertainment
– Streaming
video
and
audio
– More
than
50%
of
Internet
traffic
at
peak
periods
• Popular
services
– NeVlix
(34.9%),
YouTube
(14.0%),
Amazon
Video
(2.6%),
Hulu
(1.4%)
– All
delivered
over-­‐the-­‐top
(OTT)
– MPEG
Dynamic
Adap+ve
Streaming
over
HTTP
Global
Internet
Phenomena
Report:
2H
2014
December
2,
2014
VideoNext
2014,
Sydney
3

4. Over-­‐The-­‐Top
–
Adap+ve
Media
Streaming
• In
a
nutshell
…
Adapta8on
logic
is
within
the
client,
not
norma8vely
specified
by
the
standard,
subject
to
research
and
development
C.
Timmerer
and
A.
C.
Begen,
“Over-­‐the-­‐Top
Content
Delivery:
State
of
the
Art
and
Challenges
Ahead”,
In
Proceedings
of
the
ACM
interna+onal
conference
on
Mul+media
(MM
'14),
Orlando,
FL,
USA,
Nov.
2014.
h"p://www.slideshare.net/chris+an.+mmerer/over-­‐the-­‐top-­‐content-­‐delivery-­‐state-­‐of-­‐the-­‐art-­‐and-­‐challenges-­‐ahead
December
2,
2014
VideoNext
2014,
Sydney
4

5. MPEG
Dynamic
Adap+ve
Streaming
over
HTTP
What
is
specified
–
and
what
is
not?
Media
Presenta+on
on
HTTP
Server
Media
Presenta8on
DASH-­‐enabled
Client
Descrip8on
Segment
…
.
.
.
.
.
.
Segment
…
Segment
…
.
.
.
.
.
.
Segment
…
…
Segments
located
by
HTTP-­‐URLs
DASH
Control
Engine
HTTP/1.1
MPD
Parser
On-­‐8me
HTTP
requests
to
segments
HTTP
Client
Media
Engine
December
2,
2014
VideoNext
2014,
Sydney
5

6. MPEG
Dynamic
Adap+ve
Streaming
over
HTTP
What
is
specified
–
and
what
is
not?
Media
Presenta+on
on
HTTP
Server
Media
Presenta8on
DASH-­‐enabled
Client
Descrip8on
Segment
…
.
.
.
.
.
.
Segment
…
Segment
…
.
.
.
.
.
.
Segment
…
…
Segments
located
by
HTTP-­‐URLs
DASH
Control
Engine
HTTP/1.1
MPD
Parser
On-­‐8me
HTTP
requests
to
segments
HTTP
Client
Media
Engine
December
2,
2014
VideoNext
2014,
Sydney
6

7. DASH
Data
Model
Segment Info
Initialization Segment
http://bitmov.in/500/init.mp4 Media
Presentation
Period, start=0s
…
Period, start=100s
…
Period, start=200s
…
…
Period
start=100
baseURL=http://…
bitmov.in/
AdaptationSet 1
500-1500 kbit/s
AdaptationSet 2
1500-3000 kbit/s
…
Media Segment 1
start=100s
http://bitmov.in/500/seg-1.m4s
Media Segment 2
start=102s
http://bitmov.in/500/seg-2.m4s
Media Segment 3
start=104s
http://bitmov.in/500/seg-3.m4s
Media Segment 50
start=198s
http://bitmov.in/500/seg-50.m4s
AdaptationSet 1
width=640-1280
height=360-720
…
Representation 1
500 Kbit/s
Representation 2
1500 Kbit/s
…
Representation 2
bandwidth=1500 kbit/s
width=960, height=540
…
Segment Info
duration=2s
Template:
500/seg-$Number$.m4s
Initialization:
500/init.mp4
December
2,
2014
VideoNext
2014,
Sydney
7

8. December
2,
2014
VideoNext
2014,
Sydney
8

9. How
to
evaluate
DASH?
• Methodology
– Dataset,
tools
(see
backup
slides
for
details)
– Common
evalua+on
setup
– Bandwidth
traces
(real/synthe+c)
vs.
models
• Metrics
– Average
media
bitrate/throughput
at
the
client
– Number
of
representa+on/quality
switches
– Number
of
stalls
(in
seconds)
–
buffer
level
C.
Mueller,
S.
Lederer,
C.
Timmerer,
“An
Evalua+on
of
Dynamic
Adap+ve
Streaming
over
HTTP
in
Vehicular
Environments”,
In
Proceedings
of
the
Fourth
Annual
ACM
SIGMM
Workshop
on
Mobile
Video
(MoVid12),
Chapel
Hill,
North
Carolina,
February
2012.
December
2,
2014
VideoNext
2014,
Sydney
9

10. Quality
of
Experience
• Quality
of
Experience
– “…
is
the
degree
of
delight
or
annoyance
of
the
user
of
an
applica+on
or
service…”
– Factors
influencing
/
features
of
QoE
may
lead
to
applica+on-­‐specific
defini+ons
• Subjec+ve
quality
assessments
– Laboratory
environment
[ITU-­‐T
B.500
/
P.910]
– Crowdsourcing
with
special
plaVorms
or
social
networks
• QoE
of
DASH-­‐based
services
– Startup
delay
(low)
– Buffer
underrun
/
stalls
(zero)
– Quality
switches
(low)
and
media
throughput
(high)
P.
Le
Callet,
S.
Möller
and
A.
Perkis,
eds.,
“Qualinet
White
Paper
on
Defini+ons
of
Quality
of
Experience
(2012)”,
European
Network
on
Quality
of
Experience
in
Mul>media
Systems
and
Services
(COST
Ac>on
IC
1003),
Lausanne,
Switzerland,
Version
1.2,
March
2013."
December
2,
2014
VideoNext
2014,
Sydney
10

11. Methodology
• Quality
of
Experience
…
– Mean
Opinion
Score
[0..100]
– [other
objec+ve
metrics:
start-­‐up
+me,
throughput,
number
of
stalls]
• …
Web-­‐based
Adap+ve
HTTP
Streaming
Clients
…
– HTML5+MSE:
DASH-­‐JS
(dash.itec.aau.at),
dash.js
(DASH-­‐IF,
v1.1.2),
YouTube
• …
Real-­‐World
Environments
…
– DASH-­‐JS,
dash.js
hosted
at
ITEC/AAU
(~
10Gbit/s)
– YouTube
hosted
at
Google
data
centers
– Content:
Tears
of
Steel
@
144p
(250
kbit/s),
240p
(380
kbit/s),
360p
(740
kbit/
s),
480p
(1308
kbit/s),
and
720p
(2300
kbit/s);
segment
size:
2s
– Users
access
content
over
the
open
Internet
• …
Crowdsourcing
– Campaign
at
Microworker
plaVorm
(others
also
possible:
Mechanical
Turk,
social
networks)
limited
to
Europe,
USA/Canada,
India
– Screening
Techniques:
Browser
fingerprin+ng,
s+mulus
presenta+on
+me,
QoE
ra+ngs
and
pre-­‐ques+onnaire
December
2,
2014
VideoNext
2014,
Sydney
11

12. Results:
QoE
December
2,
2014
VideoNext
2014,
Sydney
12

13. Results:
Media
Throughput
December
2,
2014
VideoNext
2014,
Sydney
13

14. Results:
Start-­‐Up
Time
December
2,
2014
VideoNext
2014,
Sydney
14

15. Results:
Number
of
Switches
December
2,
2014
VideoNext
2014,
Sydney
15

16. Results:
Number
of
Stalls
December
2,
2014
VideoNext
2014,
Sydney
16

17. Results:
Summary
• DASH-­‐JS
– High
start-­‐up
+me
– Low
number
of
stalls
– Good
throughput,
QoE
• dash.js
– Low
start-­‐up
+me
– High
#
stalls
– Low
throughput
– Low
QoE
• YouTube
– Low
start-­‐up
+me
– Low
number
of
stalls
– Best
throughput,
QoE
December
2,
2014
VideoNext
2014,
Sydney
17

18. Conclusions
• QoE
evalua+on
of
DASH-­‐like
systems
in
real-­‐world
environments
using
crowdsourcing
– Detailed
methodology
described
in
the
paper
– Results
indicate
that
the
delivered
representa+on
bitrate
(media
throughput)
and
the
number
of
stalls
are
the
main
influence
factors
on
the
QoE
– Results
confirmed
by
previous
evalua+ons
but
within
controlled
environments
– Evidence
about
QoE
aspects
of
DASH-­‐enabled
Web
clients
within
real-­‐
world
environments
– Feasibility
of
using
crowdsourcing
for
subjec+ve
quality
assessments
• Future
work
– Comprehensive
evalua+on
of
various
adapta+on
logics
(both
objec+ve
and
subjec+ve)
and
– the
impact
of
dedicated
delivery
infrastructures
aiming
to
improve
DASH-­‐based
services
December
2,
2014
VideoNext
2014,
Sydney
18

19. Thank
you
for
your
a"en+on
...
ques+ons,
comments,
etc.
are
welcome
…
Priv.-­‐Doz.
Dipl.-­‐Ing.
Dr.
Chris+an
Timmerer
Associate
Professor
Klagenfurt
University,
Department
of
Informa+on
Technology
(ITEC)
Universitätsstrasse
65-­‐67,
A-­‐9020
Klagenfurt,
AUSTRIA
chris+an.+mmerer@itec.uni-­‐klu.ac.at
h"p://research.+mmerer.com/
Tel:
+43/463/2700
3621
Fax:
+43/463/2700
3699
©
Copyright:
Chris>an
Timmerer
December
2,
2014
VideoNext
2014,
Sydney
19

20. December
2,
2014
VideoNext
2014,
Sydney
20

21. December
2,
2014
VideoNext
2014,
Sydney
21

22. BACKUP
SLIDES
December
2,
2014
VideoNext
2014,
Sydney
22

23. End-­‐to-­‐End
DASH
System
Aspects
• (Distributed)
dataset
– Full
movie
length
in
high
quality
– Various
bitrate,
resolu+ons,
segment
lengths
(2-­‐15s),
(sub-­‐)segments
– Distributed:
ini+al
3
sites,
now
9
in
Europe,
USA,
Taiwan
• DASH
encoder
– Encoding
+
Mul+plexing
+
MPD
genera+on
– Fully
configurable
using
a
configura+on
file
– Enables
batch
processing
– x264/ffmpeg
+
GPAC
MP4Box
S.
Lederer,
C.
Müller,
C.
Timmerer,
“Dynamic
Adap+ve
Streaming
over
HTTP
Dataset”,
In
Proceedings
of
the
ACM
Conference
on
Mul+media
Systems
2012,
Chapel
Hill,
North
Carolina,
February
2012.
//
S.
Lederer,
C.
Mueller,
C.
Timmerer,
C.
Concolato,
J.
Le
Feuvre,
K.
Fliegel,
“Distributed
DASH
Dataset”,
In
Proceedings
of
the
ACM
Conference
on
Mul+media
Systems
2013,
Oslo,
Norway,
2013.
December
2,
2014
VideoNext
2014,
Sydney
23

24. End-­‐to-­‐End
DASH
System
Aspects
• Playback
– VLC
plugin
(first
implementa+on)
– DASH-­‐JS
(HTML5
+
MSE)
– libdash
/
qtsampleplayer
• MPD
valida+on
– XML
schema
valida+on
– Xlink
resolver
&
processing
– Addi+onal
valida+on
rules
(Schematron)
• Experimental
– DASH
over
Content-­‐Centric
Networks
(CCN)
– VLC
+
libdash
C.
Müller
and
C.
Timmerer,
“A
VLC
Media
Player
Plugin
enabling
Dynamic
Adap+ve
Streaming
over
HTTP”,
In
Proceedings
of
the
ACM
Mul+media
2011,
Sco"sdale,
Arizona,
November
2011.
//
B.
Rainer,
S.
Lederer,
C.
Müller,
C.
Timmerer,
“A
Seamless
Web
Integra+on
of
Adap+ve
HTTP
Streaming”,
In
Proceedings
of
the
20th
European
Signal
Processing
Conference
2012,
Bucharest,
Romania,
August
2012.
h"p://records.sigmm.ndlab.net/2013/04/open-­‐source-­‐column-­‐dynamic-­‐adap+ve-­‐streaming-­‐over-­‐h"p-­‐toolset/
December
2,
2014
VideoNext
2014,
Sydney
24