Objective and Subjective QoE Evaluation for Adaptive Point Cloud Streaming

•Als PPTX, PDF herunterladen•

0 gefällt mir•1,337 views

Volumetric media has the potential to provide the six degrees of freedom (6DoF) required by truly immersive media. However, achieving 6DoF requires ultra-high bandwidth transmissions, which real-world wide area networks cannot provide economically. Therefore, recent efforts have started to target the efficient delivery of volumetric media, using a combination of compression and adaptive streaming techniques. It remains, however, unclear how the effects of such techniques on the user-perceived quality can be accurately evaluated. In this paper, we present the results of an extensive objective and subjective quality of experience (QoE) evaluation of volumetric 6DoF streaming. We use PCC-DASH, a standards-compliant means for HTTP adaptive streaming of scenes comprising multiple dynamic point cloud objects. By means of a thorough analysis we investigate the perceived quality impact of the available bandwidth, rate adaptation algorithm, viewport prediction strategy, and user's motion within the scene. We determine which of these aspects has more impact on the user's QoE, and to what extent subjective and objective assessments are aligned.

Technologie

Objective and Subjective QoE Evaluation for
Adaptive Point Cloud Streaming
Jeroen van der Hooft, Maria Torres Vega, Filip De Turck (Ghent University – imec)
Christian Timmerer, Raimund Schatz (AAU Klagenfurt, Bitmovin, AIT)
Ali C. Begen (Özyeğin University)

Dynamic scenes require a significant amount of data
3
4.1 Gb/s 3.8 Gb/s 5.7 Gb/s 5.6 Gb/s

Streaming this scene would require 19.2 Gb/s!
4

We can solve this by using compression
5
4.5 Mb/s 40.4 Mb/s 5.7 Gb/s

HTTP Adaptive Streaming is the de-facto standard
6
2
3
Time
Bitrate
3
2
1
3
2
1
3
2
1
3
2
1
3
2
111
Time
Bitrate
1
Bandwidth
2
33
2
1
3
2
1
3
2
1
3
2
1
3
2
11112
33
2
1
3
2
1
3
2
1
3
2
1
3
2
11112
33
2
1
3
2
1
3
2
1
3
2
1
3
2
1111Object Time
Bitrate
Bandwidth
1
1
1
1
1
1
2
2
3
3
3
2
2
1
2 2
2
1
2
2

Research questions?
RQ1: What is the impact of network and content characteristics on
the perceived video quality?
RQ2: How do objective metrics correlate with subjective ratings for
the perceived video quality?
7

Each of the three videos comes with eight configurations
Bandwidth [Mb/s] Allocation Prediction
20 Visible objects 0
60 Visible objects 0
100 Visible objects 0
20 Visible objects 1
60 Visible objects 1
100 Visible objects 1
60 All objects 0
∞ N/A N/A
12

Streaming at 20 Mb/s illustrates low video quality
13

Participants were asked to rate the video quality
14
h
4h

RQ1: What is the impact of network and content
characteristics on the perceived video quality?
15

Subjects can distinguish between different bitrates
16
20 Mb/s
60 Mb/s
100 Mb/s
Unlimited

Viewport prediction allows to improve the observed quality
17
Most recent
Clairvoyant

RQ2: How do objective metrics correlate with
subjective ratings for the perceived video quality?
18

Clear correlation between objective metrics and MOS scores
19
PSNR
SRC 1
SRC 2
SRC 3

Clear correlation between objective metrics and MOS scores
20
SSIM
SRC 1
SRC 2
SRC 3

Clear correlation between objective metrics and MOS scores
21
VQM
SRC 1
SRC 2
SRC 3

Subjective scores best match with SSIM
22

However, part of the model is explained by the selected SRC
23
SSIM
SRC 1
SRC 2
SRC 3

However, part of the model is explained by the selected SRC
24
VQM
SRC 1
SRC 2
SRC 3

Conclusions
A single-stimulus method works
RQ1: Impact of network and content characteristics is clear and can be
explained
RQ2: High correlation between objective and subjective metrics, but
observed values need to be adjusted to the considered content
More representative metrics and QoE models are required
26

Objective and Subjective QoE Evaluation for Adaptive Point Cloud Streaming

Weitere ähnliche Inhalte

Was ist angesagt?

Recorded cataract surgery videos play a prominent role in training and investigating the surgery, and enhancing the surgical outcomes. Due to storage limitations in hospitals, however, the recorded cataract surgeries are deleted after a short time and this precious source of information cannot be fully utilized. Lowering the quality to reduce the required storage space is not advisable since the degraded visual quality results in the loss of relevant information that limits the usage of these videos. To address this problem, we propose a relevance-based compression technique consisting of two modules: (i) relevance detection, which uses neural networks for semantic segmentation and classification of the videos to detect relevant spatio-temporal information, and (ii) content-adaptive compression, which restricts the amount of distortion applied to the relevant content while allocating less bitrate to irrelevant content. The proposed relevance-based compression framework is implemented considering five scenarios based on the definition of relevant information from the target audience’s perspective. Experimental results demonstrate the capability of the proposed approach in relevance detection. We further show that the proposed approach can achieve high compression efficiency by abstracting substantial redundant information while retaining the high quality of the relevant content.

Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Ne...

Alpen-Adria-Universität

Real-time video streaming traffic and related applications have witnessed significant growth in recent years. However, this has been accompanied by some challenging issues, predominantly resource utilization. IP multicasting, as a solution to this problem, suffers from many problems. Using scalable video coding could not gain wide adoption in the industry, due to reduced compression efficiency and additional computational complexity. The emerging software-defined networking (SDN)and network function virtualization (NFV) paradigms enable re-searchers to cope with IP multicasting issues in novel ways. In this paper, by leveraging the SDN and NFV concepts, we introduce a cost-aware approach to provide advanced video coding (AVC)-based real-time video streaming services in the network. In this study, we use two types of virtualized network functions (VNFs): virtual reverse proxy (VRP) and virtual transcoder (VTF)functions. At the edge of the network, VRPs are responsible for collecting clients’ requests and sending them to an SDN controller. Then, executing a mixed-integer linear program (MILP) determines an optimal multicast tree from an appropriate set of video source servers to the optimal group of transcoders. The desired video is sent over the multicast tree. The VTFs transcode the received video segments and stream to the requested VRPs over unicast paths. To mitigate the time complexity of the proposed MILPmodel, we propose a heuristic algorithm that determines a near-optimal solution in a reasonable amount of time. Using theMiniNet emulator, we evaluate the proposed approach and show it achieves better performance in terms of cost and resource utilization in comparison with traditional multicast and unicast approaches.

On Optimizing Resource Utilization in AVC-based Real-time Video Streaming

Alpen-Adria-Universität

Video traffic on the Internet is constantly growing; networked multimedia applications consume a predominant share of the available Internet bandwidth. A major technical breakthrough and enabler in multimedia systems research and of industrial networked multimedia services certainly was the HTTP Adaptive Streaming (HAS) technique. This resulted in the standardization of MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH) which, together with HTTP Live Streaming (HLS), is widely used for multimedia delivery in today’s networks. Existing challenges in multimedia systems research deal with the trade-off between (i) the ever-increasing content complexity, (ii) various requirements with respect to time (most importantly, latency), and (iii) quality of experience (QoE). Optimizing towards one aspect usually negatively impacts at least one of the other two aspects if not both. This situation sets the stage for our research work in the ATHENA Christian Doppler (CD) Laboratory (Adaptive Streaming over HTTP and Emerging Networked Multimedia Services; https://athena.itec.aau.at/), jointly funded by public sources and industry. In this talk, we will present selected novel approaches and research results of the first year of the ATHENA CD Lab’s operation. We will highlight HAS-related research on: (i) multimedia content provisioning (machine learning for video encoding); (ii) multimedia content delivery (support of edge processing and virtualized network functions for video networking); (iii) multimedia content consumption and end-to-end aspects (player-triggered segment retransmissions to improve video playout quality); and (iv) novel QoE investigations (adaptive point cloud streaming). We will also put the work into the context of the international multimedia systems research.

HTTP Adaptive Streaming – Quo Vadis?

Alpen-Adria-Universität

Due to the growing importance of optimizing quality and efficiency of video streaming delivery, accurate assessment of user perceived video quality becomes increasingly relevant. However, due to the wide range of viewing distances encountered in real-world viewing settings, actually perceived video quality can vary significantly in everyday viewing situations. In this paper, we investigate and quantify the influence of viewing distance on perceived video quality. A subjective experiment was conducted with full HD sequences at three different stationary viewing distances, with each video sequence being encoded at three different quality levels. Our study results confirm that the viewing distance has a significant influence on the quality assessment. In particular, they show that an increased viewing distance generally leads to an increased perceived video quality, especially at low media encoding quality levels. In this context, we also provide an estimation of potential bitrate savings that knowledge of actual viewing distance would enable in practice. Since current objective video quality metrics do not systematically take into account viewing distance, we also analyze and quantify the influence of viewing distance on the correlation between objective and subjective metrics. Our results confirm the need for distance-aware objective metrics when accurate prediction of perceived video quality in real-world environments is required.

On the Impact of Viewing Distance on Perceived Video Quality

Alpen-Adria-Universität

With the increasing demand for video streaming applications, HTTP Adaptive Streaming (HAS) technology has become the dominant video delivery technique over the Internet. Current HAS solutions only consider either client- or server-side optimization, which causes many problems in achieving high-quality video, leading to sub-optimal users’ experience and network resource utilization. Recent studies have revealed that network-assisted HAS techniques, by providing a comprehensive view of the network, can lead to more significant gains in HAS system performance. In this paper, we leverage the capability of Software-Define Networking (SDN), Network Function Virtualization (NFV), and edge computing to introduce a CDN-Aware QoE Optimization in SDN-Assisted Adaptive Video Streaming framework called CSDN. We employ virtualized edge entities to collect various information items (e.g., user-, client, CDN- and network-level information) in a time-slotted method. These components then run an optimization model with a new server/segment selection approach in a time-slotted fashion to serve the clients’ requests by selecting optimal cache servers (in terms of fetch and transcoding times). In case of a cache miss, a client’s request is served (i) by an optimal replacement quality (only better quality levels with minimum deviation) from a cache server, (ii) by a quality transcoded from an optimal replacement quality at the edge, or (iii) by the originally requested quality level from the origin server. By means of comprehensive experiments conducted on a real-world large-scale testbed, we demonstrate that CSDN outperforms the state-of-the-art in terms of playback bitrate, the number of quality switches, the number of stalls, and bandwidth usage by at least 7.5%, 19%, 19%, and 63%, respectively.

CSDN: CDN-Aware QoE Optimization in SDN-Assisted HTTP Adaptive Video Streaming

Alpen-Adria-Universität

With the recent surge in Internet multimedia traffic, the enhancement and improvement of media players, specifically DASH media players happened at an incredible rate. DASH Media players take advantage of adapting a media stream to the network fluctuations by continuously monitoring the network and making decisions in near real-time. The performance of algorithms that are in charge of making such decisions was often difficult to be evaluated and objectively assessed. CAdViSE provides a Cloud-based Adaptive Video Streaming Evaluation framework for the automated testing of adaptive media players. In this talk, I will introduce the CAdViSE framework, its application, and propose the benefits and advantages that it can bring to every web-based media player development pipeline. To demonstrate the power of CAdViSE in evaluating Adaptive Bitrate (ABR) algorithms I will exhibit its capabilities when combined with objective Quality of Experience (QoE) models. For this talk, my team at Bitmovin/ATHENA has selected the ITU-T P.1203 (mode 1) model in order to execute experiments and calculate the Mean Opinion Score (MOS), and better understand the behavior of a set of well-known ABR algorithms in a real-life setting. The talk will display how we tested and deployed our framework using a modular architecture into a cloud infrastructure. This method yields a massive growth to the number of concurrent experiments and the number of media players that can be evaluated and compared at the same time, thus enabling maximum potential scalability. In my team’s most recent experiments, we used Amazon Web Services (AWS) for demonstration purposes. Another awesome feature of CAdViSE that will be discussed here is the ability to shape the test network with endless network profiles. To do so, we used a fluctuation network profile and a real LTE network trace based on the recorded internet usage of a bicycle commuter in Belgium. CAdViSE produces comprehensive logs for each media streaming experimental session. These logs can then be applied against different goals, such as objective evaluation to stitch back media segments and conduct subjective evaluations afterwards. In addition, startup delays, stall events, and other media streaming defects can be imitated exactly as they happened during the experimental streaming sessions.

CAdViSE or how to find the Sweet Spots of ABR Systems

Alpen-Adria-Universität

HTTP adaptive streaming (HAS) with chunked transfer encoding can be used to reduce latency without sacrificing the coding efficiency. While this allows a media segment to be generated and delivered at the same time, it also causes grossly inaccurate bandwidth measurements, leading to incorrect bitrate selections. To overcome this effect, we design a novel Adaptive bitrate scheme for Chunked Transfer Encoding (ACTE) that leverages the unique nature of chunk downloads. It uses a sliding window to accurately measure the available bandwidth and an online linear adaptive filter to predict the available bandwidth into the future. Results show that ACTE achieves 96% measurement accuracy, which translates to a 64% reduction in stalls and a 27% increase in video quality.

Bandwidth Prediction in Low-Latency Chunked Streaming

Alpen-Adria-Universität

Real-time entertainment services deployed over the open, unmanaged Internet – streaming audio and video – account now for more than 70% of the Internet traffic and it is assumed that this number will reach 80% by 2021. The technology used for such services is commonly referred to as HTTP Adaptive Streaming (HAS) and is widely adopted by various platforms such as YouTube, Netflix, Flimmit, etc. thanks to the standardization of MPEG-DASH and HLS. This talk will provide an overview of HAS, the state of the art of selected deployment options, and reviews work-in-progress as well challenges ahead. The main challenge can be characterized by the fact that (i) content complexity increases, (ii) delay or latency are vital application requirements, and (iii) Quality of Experience cannot be neglected anymore.

HTTP Adaptive StreamingState of the Art and Challenges Ahead

Alpen-Adria-Universität

Video streaming is one of the top traffic contributors in the Internet and a frequent research subject. It is expected that streaming traffic will grow 4-fold for video globally and 9-fold for mobile video between 2017 and 2022. In this paper, we present an automatized measurement framework for evaluating video streaming QoE in operational broadband networks, using headless streaming with a Docker-based client, and a server-side implementation allowing for the use of multiple video players and adaptation algorithms. Our framework allows for integration with the MONROE testbed and Bitmovin Analytics, which bring on the possibility to conduct large-scale measurements in different networks, including mobility scenarios, and monitor different parameters in the application, transport, network, and physical layers in real-time.

Docker-Based Evaluation Framework for Video Streaming QoE in Broadband Networks

Alpen-Adria-Universität

Quality impact of scalable video coding tunneling for media aware content del...

Alpen-Adria-Universität

Labmeeting - 20150831 - Overhead and Performance of Low Latency Live Streamin...

Syuan Wang

HTTP Adaptive Streaming (HAS) enables high-quality streaming of video content. In HAS, videos are divided into short intervals called segments, and each segment is encoded at various quality/bitrates to adapt to the available bandwidth. Multiple encodings of the same content impose high costs for video content providers. To reduce the time-complexity of encoding multiple representations, state-of-the-art methods typically encode the highest quality representation first and reuse the information gathered during its encoding to accelerate the encoding of the remaining representations. As encoding the highest quality representation requires the highest time-complexity compared to the lower quality representations, it would be a bottleneck in parallel encoding scenarios and the overall time-complexity will be limited to the time-complexity of the highest quality representation. In this paper and to address this problem, we consider all representations from the highest to the lowest quality representation as a potential, single reference to accelerate the encoding of the other, dependent representations. We formulate a set of encoding modes and assess their performance in terms of BD-Rate and time-complexity, using both VMAF and PSNR as objective metrics. Experimental results show that encoding a middle quality representation as a reference, can significantly reduce the maximum en-coding complexity and hence it is an efficient way of encoding multiple representations in parallel. Based on this fact, a fast multirate encoding method is proposed which utilizes depth and prediction mode of a middle quality representation to accelerate the encoding of the dependent representations.

Towards Optimal Multirate Encoding for HTTP Adaptive Streaming

Alpen-Adria-Universität

Live User Generated Content (UGC) has become very popular in today’s video streaming applications, in particular with gaming and e-sport. However, streaming UGC presents unique challenges for video delivery. When dealing with the technical complexity of managing hundreds or thousands of concurrent streams that are geographically distributed, UGCsystems are forces to made difficult trade-offs with video quality and latency. To bridge this gap, this paper presents a fully distributed architecture for UGC delivery over the Internet, termed QuaLA(joint Quality-Latency Architecture). The proposed architecture aims to jointly optimize video quality and latency for a better user experience and fairness. By using the proximal Jacobi alternating direction method of multipliers(ProxJ-ADMM) technique, QuaLA proposes a fully distributed mechanism to achieve an optimal solution. We demonstrate the effectiveness of the proposed architecture through real-world experiments using the CloudLAB testbed. Experimental results show the outperformance ofQuaLAin achieving high quality with more than 57% improvement while preserving a good level of fairness and respecting a given target latency among all clients compared to conventional client-driven solutions

A Distributed Delivery Architecture for User Generated Content Live Streaming...

Alpen-Adria-Universität

Mobile networks equipped with edge computing nodes enable access to information that can be leveraged to assist client-based adaptive bitrate (ABR) algorithms in making better adaptation decisions to improve both Quality of Experience (QoE) and fairness. For this purpose, we propose a novel on-the-fly edge mechanism, named EADAS (Edge Assisted Adaptation Scheme for HTTP Adaptive Streaming), located at the edge node that assists and improves the ABR decisions on-the-fly. EADAS proposes (i) an edge ABR algorithm to improve QoE and fairness for clients and (ii) a segment prefetching scheme. The results show a QoE increase of 4.6%, 23.5%, and 24.4% and a fairness increase of 11%, 3.4%, and 5.8% when using a buffer-based, a throughput-based, and a hybrid ABR algorithm, respectively, at the client compared with client-based algorithms without EADAS. Moreover, QoE and fairness among clients can be prioritized using parameters of the EADAS algorithm according to service providers’ requirements.

EADAS: Edge Assisted Adaptation Scheme for HTTP Adaptive Streaming

Alpen-Adria-Universität

With the introduction of HTTP/3 (H3) and QUIC at its core, there is an expectation of significant improvements in Web-based secure object delivery. As HTTP is a central protocol to the current adaptive streaming methods in all major over-the-top (OTT) services, an important question is what H3 will bring to the table for such services. To answer this question, we present the new features of H3 and QUIC, and compare them to those of H/1.1/2 and TCP. We also share the latest research findings in this domain.

MHV'22 - Take the Red Pill for H3 and See How Deep the Rabbit Hole Goes

Minh Nguyen

Universal media access as proposed in the late 90s is now closer to reality. Users can generate, distribute and consume almost any media content, anywhere, anytime and with/on any device. A major technical breakthrough was the adaptive streaming over HTTP resulting in the standardization of MPEG-DASH, which is now successfully deployed in most platforms. The next challenge in adaptive media streaming is virtual reality applications and, specifically, omnidirectional (360°) media streaming. This tutorial first presents a detailed overview of adaptive streaming of both traditional and omnidirectional media, and focuses on the basic principles and paradigms for adaptive streaming. New ways to deliver such media are explored and industry practices are presented. The tutorial then continues with an introduction to the fundamentals of communications over 5G and looks into mobile multimedia applications that are newly enabled or dramatically enhanced by 5G. A dedicated section in the tutorial covers the much-debated issues related to quality of experience. Additionally, the tutorial provides insights into the standards, open research problems and various efforts that are underway in the streaming industry.

Delivering Traditional and Omnidirectional Media

Alpen-Adria-Universität

Attempting to cope with fluctuations of network conditions in terms of available bandwidth, latency and packet loss, and to deliver the highest quality of video (and audio) content to users, research on adaptive video streaming has attracted intense efforts from the research community and huge investments from technology giants. How successful these efforts and investments are, is a question that needs precise measurements of the results of those technological advancements. HTTP-based Adaptive Streaming (HAS) algorithms, which seek to improve video streaming over the Internet, introduce video bitrate adaptivity in a way that is scalable and efficient. However, how each HAS implementation takes into account the wide spectrum of variables and configuration options, brings a high complexity to the task of measuring the results and visualizing the statistics of the performance and quality of experience. In this paper, we introduce CAdViSE, our Cloud-based Adaptive Video Streaming Evaluation framework for the automated testing of adaptive media players. The paper aims to demonstrate a test environment which can be instantiated in a cloud infrastructure, examines multiple media players with different network attributes at defined points of the experiment time, and finally concludes the evaluation with visualized statistics and insights into the results.

CAdViSE: Cloud based Adaptive Video Streaming Evaluation Framework for the Au...

Alpen-Adria-Universität

The 131st WG 11 (MPEG) meeting was held online, 29 June – 3 July 2020 Table of Contents WG11 (MPEG) Announces VVC – the Versatile Video Coding Standard Point Cloud Compression – WG11 (MPEG) promotes a Video-based Point Cloud Compression Technology to the FDIS stage MPEG-H 3D Audio – WG11 (MPEG) promotes Baseline Profile for 3D Audio to final stage Call for Proposals on Technologies for MPEG-21 Contracts to Smart Contracts Conversion WG11 (MPEG) issues a Call for Proposals on extension and improvements to ISO/IEC 23092 standard series Widening support for storage and delivery of MPEG-5 EVC Multi-Image Application Format adds support of HDR Carriage of Geometry-based Point Cloud Data progresses to Committee Draft MPEG Immersive Video (MIV) progresses to Committee Draft Neural Network Compression for Multimedia Applications – WG11 (MPEG) progresses to Committee Draft WG11 (MPEG) issues Committee Draft of Conformance and Reference Software for Essential Video Coding (EVC)

Press Release of 131st WG11 (MPEG) Meeting

Alpen-Adria-Universität

The advancement of hardware capabilities in recent years made it possible to apply deep neural network (DNN) based approaches on mobile devices. This paper introduces a lightweight super-resolution (SR) network, namely SR-ABR Net, deployed at mobile devices to upgrade low-resolution/low-quality videos and a novel adaptive bitrate (ABR) algorithm, namely WISH-SR, that leverages SR networks at the client to improve the video quality depending on the client's context. WISH-SR takes into account mobile device properties, video characteristics, and user preferences. Experimental results show that the proposed SR-ABR Net can improve the video quality compared to traditional SR approaches while running in real time. Moreover, the proposed WISH-SR can significantly boost the visual quality of the delivered content while reducing both bandwidth consumption and number of stalling events.

MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...

Minh Nguyen

Omnidirectional video (ODV) streaming applications are becoming increasingly popular. They enable a highly immersive experience as the user can freely choose her/his field of view within the 360-degree environment. Current deployments are fairly simple but viewport-agnostic which inevitably results in high storage/bandwidth requirements and low Quality of Experience (QoE). A promising solution is referred to as tile- based streaming which allows to have higher quality within the user’s viewport while quality outside the user’s viewport could be lower. However, empirical QoE assessment studies in this domain are still rare. Thus, this paper investigates the impact of different tile-based streaming approaches and configurations on the QoE of ODV. We present the results of a lab-based subjective evaluation in which participants evaluated 8K omnidirectional video QoE as influenced by different (i) tile-based streaming approaches (full vs. partial delivery), (ii) content types (static vs. moving camera), and (iii) tile encoding quality levels determined by different quantization parameters. Our experimental setup is characterized by high reproducibility since relevant media delivery aspects (including the user’s head movements and dynamic tile quality adaptation) are already rendered into the respective processed video sequences. Additionally, we performed a complementary objective evaluation of the different test sequences focusing on bandwidth efficiency and objective quality metrics. The results are presented in this paper and discussed in detail which confirm that tile-based streaming of ODV improves visual quality while reducing bandwidth requirements.

Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...

Alpen-Adria-Universität

Was ist angesagt? (20)