Adaptive BitRate (ABR) algorithms play a crucial role in delivering the highest possible viewer’s Quality of Experience (QoE) in HTTP Adaptive Streaming (HAS). Online video streaming service providers use HAS – the dominant video streaming technique on the Internet – to deliver the best QoE for their users. Viewer’s delightfulness relies heavily on how the ABR of a media player can adapt the stream’s quality to the current network conditions. QoE for end-to-end video streaming sessions has been evaluated in many research projects to give better insight into the quality metrics. Objective evaluation models such as ITU Telecommunication Standardization Sector (ITU-T) P.1203 allow for the calculation of Mean Opinion Score (MOS) by considering various QoE metrics, and subjective evaluation is the best assessment approach in investigating the end-user opinion over a video streaming session’s experienced quality. We have conducted subjective evaluations with crowdsourced participants and evaluated the MOS of the sessions using the ITU-T P.1203 quality model. This paper’s main contribution is subjective evaluation analogy with objective evaluation for well-known heuristic-based ABRs.

1. Network and Operating System Support for Digital Audio and Video (NOSSDAV’21)
Understanding Quality of Experience of Heuristic-based HTTP
Adaptive Bitrate Algorithms
Babak Taraghi, Abdelhak Bentaleb, Christian Timmerer, Roger Zimmermann, and Hermann
Hellwagner

2. Overview and the Motivation
• Main Idea
– Evaluate a set of well-known heuristic-based ABR algorithms.
– Conduct subjective evaluations with crowdsourced participants and evaluate the MOS of the
streaming sessions using the ITU-T P.1203 quality model.
– To investigate the correspondence of subjective and objective evaluations for well-known heuristic-
based ABRs.
• Platforms and Participants
– Using CAdViSE as our testbed to conduct the objective evaluations. CAdViSE provides a cloud-based
platform to evaluate multiple ABR algorithms or media players under various network conditions.
– Using Amazon Mechanical Turk (MTurk) to conduct our subjective evaluation. MTurk is a
crowdsourcing website to hire remotely located crowd-workers to perform discrete on-demand tasks.
– 835 participants in our subjective evaluation phase.
– We have gathered 5723 votes in total, out of which 4704 proved reliable.
• Test Sequences
– Sintel, Valkaama, Big Buck Bunny, and Tears of Steel.

3. Heuristic-based ABR Algorithms
• Seven ABR algorithms and two media players in total used for the evaluations:
– dash.js v3.1.3 (Dynamic)
• A combination of throughput-based and BOLA algorithms.
– Shaka v3.0.4 (Throughput-based)
• A simple throughput-based algorithm that uses throughput heuristics with an Exponential Weighted Moving Average
(EWMA) smoothing function.
– BBA0
• A buffer-based ABR algorithm that uses the current buffer occupancy to select the bitrate for the next segment.
– BOLA
• A buffer-based ABR algorithm that formulates ABR decisions as a utility maximization function using Lyapunov
optimization.
– Elastic
• A hybrid ABR that uses feedback control theory that generates a long-lived Transport Control Protocol (TCP).
– FastMPC
• A hybrid ABR algorithm that uses a Model Predictive Control (MPC) approach.
– Quetra
• A buffer-based ABR that formulates ABR decisions as a queuing theory model, which calculates the expected buffer
occupancy given a bitrate choice, network throughput, and buffer capacity.

4. 4
• Using CAdViSE we have simulated 4
network profiles as shown in Figure 1:
– Ramp Down profile
– Ramp Up profile
– Fluctuation profile
– Stable profile
• Recorded the streaming session logs.
• Calculated the MOS using the P.1203
quality model by utilizing the logs.
• Stitched back the media segments and
created a single file for subjective
evaluations.
Powered by CAdViSE*
Objective Evaluation & Network Simulations
*: Available at https://github.com/cd-athena/CAdViSE

5. Significant Metrics
1. Startup delay
2. Stall events
3. Requested segment bitrate

6. • Shaka player default ABR algorithm and BOLA algorithm with a subjective MOS of 3.73 had
the best performance in Ramp Up network profile.
• Bola algorithm also had the best performance in Ramp Down network profile with a MOS
of 3.65.
Result and Findings I
6

7. • 3.39 was the highest gained subjective MOS which shows good performance of BBA0
algorithm in a tough network profile (Fluctuation).
• Dash.js default ABR algorithm had the best performance in Stable network profile with a
MOS of 3.98.
Result and Findings II
7

8. Conclusions and Future Work
• We have evaluated seven well-known ABR algorithms in this paper.
– Learning-based ABR algorithms will be included in our future work.
• This paper’s main contribution is to investigate the correspondence of subjective and
objective evaluations for well-known heuristic-based ABRs.
– Other Quality of Experience models will be compared in our future work.
• Introduced four network profiles which were simulated using our testbed, CAdViSE.
– Real network profiles/traces will be used in our future work.
• Deeper statistical analysis over the findings and obtained result would be another direction
for our future work.

9. Thanks to my co-authors and
NOSSDAV’21 conference reviewers.
Dr. Bentaleb.
Professor Timmerer.
Professor Zimmermann.
Professor Hellwagner.
Babak Taraghi
Klagenfurt 2021