Presentation covers various aspects of defining and measuring of the Quality of Experience in IP Multimedia communications, with emphasis on Video. Presented at IMTC 20th Anniversary Forum
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Â
Recent advances in quality of experience in multimedia communication
1. IMTC 20th Anniversary Forum â Porto, Portugal1
Recent advances in
quality of experience in
multimedia
communication
Touradj Ebrahimi
Touradj.Ebrahimi@epfl.ch
International Multimedia Telecommunications Consortium
2. IMTC 20th Anniversary Forum â Porto, Portugal2
â˘What
 is
 âqualityâ?
â˘Origins
 of
 âqualityâ
 evalua7on
 in
 telecommunica7ons
â˘How
 is
 âqualityâ
 measured
 in
 mul7media
 communica7ons
Â
today
 ?
â˘What
 are
 the
 trends
 on
 âqualityâ
 in
 mul7media
Â
communica7on?
â˘
 What
 are
 the
 challenges
 ahead?
Today
 we
 will
 talk
 aboutâŚ
3. IMTC 20th Anniversary Forum â Porto, Portugal3
Quality: a simple yet difficult concept
⢠Like many human sensations, quality is easy to
understand but difficult to define
⢠Quality according to Wikipedia:
â A quality (from Latin - qualitas) is an attribute or a
property
â Some philosophers assert that a quality cannot be
defined
4. IMTC 20th Anniversary Forum â Porto, Portugal4
A fundamental, ancient, but largely under investigated concept
Aristotle classified every object of human
apprehension into 10 Categories
â Substance
â Quantity
â Quality
â Relation
â Place
â Time
â Position
â State
â Action
â Affection
5. IMTC 20th Anniversary Forum â Porto, Portugal5
Qualinet white paper
⢠White Paper produced by COST Action
IC1003 (Qualinet):
â http://www.qualinet.eu/images/stories/
whitepaper_v1.1_dagstuhl_output_corrected.pdf
⢠Several definitions of quality in multimedia
systems and services and other related
concepts
6. IMTC 20th Anniversary Forum â Porto, Portugal6
Quality is like an elephant âŚ
The blind men and the elephant: Poem by John Godfrey Saxe
8. IMTC 20th Anniversary Forum â Porto, Portugal8
Quality
 in
 telecommunica4ons
Network Quality
Capacity
Coverage
Handoff
Link Quality
Bitrate
Frame/Bit/Packet loss
Delay
User Quality
Speech fidelity
Audio fidelity
Image fidelity
Video fidelity
9. IMTC 20th Anniversary Forum â Porto, Portugal9
Quality in telecommunications
Objective: input = output
10. IMTC 20th Anniversary Forum â Porto, Portugal10
QoS in telecommunications
⢠Quality of Service (QoS)
â Resource reservation control mechanisms
â Ability to provide different priority to different
applications or data flows
â Guarantee a certain level of performance (quality) to
a data flow
⢠Requires measuring or estimating quality
⢠(Service) Provider-centric concept
â Tightly related to the concept of Mean Opinion Score
(MOS)
11. IMTC 20th Anniversary Forum â Porto, Portugal11
Mean Opinion Score (MOS)?
⢠Widely used in many fields:
â Politics/Elections
â Marketing/Advertisement
â Food industry
â Multimedia
â âŚ
⢠The likely level of satisfaction of a service or product as
appreciated by an average user in a typical use
⢠Should be performed such that it generates reliable and
reproducible results
â Subjective evaluation methodology
â More complex and difficult that it seems
13. IMTC 20th Anniversary Forum â Porto, Portugal13
What is wrong with MOS?
⢠Ground truth
â It is assumed to be the optimal solution (ground truth)
⢠Defies user preference
â It is assumed to be independent of specific subjects
⢠Ignores key parameters
â It is often assumed to be independent of content and
context
⢠Misuse as a comparison approach
â It is often used to compare performance between
alternative tools for a same task/operation
⢠Impractical or expensive
⢠Statistical assumptions not always realistic
18. IMTC 20th Anniversary Forum â Porto, Portugal18
Objec4ve
 quality
 metrics
⢠Subjec7ve
 tests
 are
 7me
 consuming,
 expensive,
 diďŹcult
 to
Â
design,
 and
 cannot
 be
 used
 in
 real-Ââ7me
 systems
⢠Objec7ve
 algorithms,
 i.e.
 metrics,
 es7ma7ng
 MOS
 with
 high
Â
level
 of
 correla7on
 are
 desired
⢠Full
 reference
 metrics
⢠No
 reference
 metrics
⢠Reduced
 reference
 metrics
19. IMTC 20th Anniversary Forum â Porto, Portugal19
Objec4ve
 quality
 metrics
 based
 on
 ďŹdelity
⢠Full
 Reference
 scenario
⢠Metrics
 which
 look
 at
 the
 ďŹdelity
 of
 the
 signal
 when
Â
compared
 to
 an
 explicit
 reference:
Â
processed
 signal
=
perfect
 quality
 reference
 signal
+
error
 signal
20. IMTC 20th Anniversary Forum â Porto, Portugal20
Objec4ve
 quality
 metrics
 based
 on
 ďŹdelity
⢠Examples
 of
 ďŹdelity
 metrics
 for
 image
 quality
 quality
Â
assessment
â Mean
 Square
 Error
 (MSE)
â Peak
 Signal
 to
 Noise
 Ra7o
 (PSNR)
â Maximum
 Pixel
 Devia7on
 (Linf)
â Weighted
 PSNR
Â
â Masked
 PSNR
â Structural
 SIMilarity
 (SSIM)
â Mul7scale
 Structural
 Similarity
 (MSSIM)
â Visual
 Informa7on
 Fidelity
 (VIF)
â etcâŚ
21. IMTC 20th Anniversary Forum â Porto, Portugal21
What is wrong with objective quality metrics ?
⢠Poor performance in predicting MOS
â Especially in real life situations
⢠How to deal with color components?
â Even in PSNR!
⢠How to deal with video?
â Time versus spatial dimension
⢠How to take into account context?
â Display size/type
â Environmental illumination
â âŚ
⢠How to deal with user preference/expectation?
⢠...
22. IMTC 20th Anniversary Forum â Porto, Portugal
MVC assessment using PSNR as metric
22 Vetro: Update on MPEG 3D Video Coding Activity
MVC Coding Performance
Ballroom
31
32
33
34
35
36
37
38
39
40
0 200 400 600 800 1000 1200 1400 1600 1800
Bitrate (Kb/s)
PSNR(db)
Simulcast
MVC
Race1
32
33
34
35
36
37
38
39
40
41
42
0 200 400 600 800 1000 1200 1400 1600
Bitrate (Kb/s)
PSNR(db)
Simulcast
MVC
Sample comparison of simulcast vs inter-view prediction
8 views (640 480), consider rate for all views
~25% bit rate savings over all views
23. IMTC 20th Anniversary Forum â Porto, Portugal
MVC assessment by subjective evaluation
23
Vetro: Update on MPEG 3D Video Coding Activity
MVC: Subjective Performance
⢠Main finding: MVC achieves comparable quality to
simulcast with as little as 25% rate for dependent view
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
O
riginal
Sim
ulcast
(AVC+AVC
)
12L_50Pct
12L_35Pct
12L_25Pct
12L_20Pct
12L_15Pct
12L_10Pct
12L_5Pct
MeanOpinionScore
Base view fixed at 12Mbps
Dependent view at varying percentage of base view rate
25. IMTC 20th Anniversary Forum â Porto, Portugal
User centered evaluation
25
⢠It is becoming increasing important to evaluate quality from a user
perspective rather than from a provider/system perspective
⢠Constrains the tests to potential users and stimuli and evaluates the
acceptability with respect to the task and the context
⢠Identify underlying quality factors beside the overall quality
26. IMTC 20th Anniversary Forum â Porto, Portugal26
What people mean when they say QoE ?
⢠âThe degree of fulfillment of an intended experience
on a given userâ
as defined by Touradj Ebrahimi, 2001
⢠âThe overall acceptability of an application or service
⌠by the end userâ
as defined by the ITU-T
⢠âThe degree of delight ... of the user of an
application or service ... userâs personality and
current state.â
as defined in white paper by Qualinet
The term âexperienceâ promises individual engagement âŚ
27. IMTC 20th Anniversary Forum â Porto, Portugal27
Quality
 of
 Service
 vs
 Quality
 of
 Experience
⢠Quality
 of
 Service:
 Value
 of
 the
 average
 userâs
Â
experience
 es7mated
 by
 a
 service/product/content
Â
provider
Â
⢠Quality
 of
 Experience:
 Value
 (es7mated
 or
 actually
Â
measured)
 of
 a
 speciďŹc
 userâs
 richness
 of
 experience
⢠Quality
 of
 Experience
 is
 the
 dual
 (and
 extended)
 view
 of
Â
QoS
 problem
QoS=provider-Ââcentric
QoE=user-Ââcentric
28. IMTC 20th Anniversary Forum â Porto, Portugal28
A simple model for QoE
⢠User attributes
â individual attributes â expectation, age, sex, personality, backgroundâŚ
â sensorial attributes â including limitations and deficiencies
â perceptual attributes
â emotional attributes
⢠System attributes
â technical attributes (as in QoS)
⢠Contextual attributes
â environmental attributes
â device attributes
â service attributes
â content attributes
user
QoS
QoE
context
29. IMTC 20th Anniversary Forum â Porto, Portugal
Personas and scenarios [Strohmeier2009]
⢠Personas (user preference)
â Archetypical user representing the needs, behaviors
and goals of a particular group of users
â Not real people but fictional characters derived from
user requirements
⢠Scenarios (context)
â Realistic usage environment
29
30. IMTC 20th Anniversary Forum â Porto, Portugal
Quality of Experience in Mobile Multimedia
⢠The influence of context in mobile multimedia
user experience and QoE
â Various probes to monitor user behavior
â Specific scenario to be followed by subjects
30
needed to
plexity of
was nec-
ur of the
quired to
ondly, as
environ-
around,
d robust
k system
he above
ontent to
eo player
k system
the video
he audio
ical user
following
Figure 1: Test equipment system installed on a user.
(a) (b)
The test equipment selected for the experiment needed to
satisfy two main constraints. Firstly, given the complexity of
the measurement task, a range of diďŹerent devices was nec-
essary to capture the video consumption behaviour of the
users during the experiment. The devices were required to
work and interact with each other in real time. Secondly, as
the experiment was conducted in a non-laboratory environ-
ment and the users needed to carry the equipment around,
the devices were constrained to be lightweight and robust
enough to work under moving conditions.
The test equipment consisted of a video playback system
and a recording system selected with respect to the above
constraints. The former was used to show the content to
the user and capture information regarding the video player
status and the user location. The video playback system
consisted of the following items:
⢠a HTC Tattoo mobile phone used to display the video
content and an external headset to listen to the audio
⢠a standard button based video player graphical user
interface, as illustrated in Figure 2, with the following
functionalities:
â pause/play
â next channel
â previous channel
â menu to change channel
â standard volume controls
⢠a logging software to keep track of the controls acti-
vated at anytime by the user
⢠a GPS integrated in the mobile phone used to deter-
mine the position and the mobility scenario of the user.
The recording system facilitated the capture and storage
of video information recorded to determine the userâs visual
focus of attention. The recording system consisted of the
following items:
⢠a head mounted Logitech Webcam Pro 9000 (cam1) to
record the scene in front of the viewer
⢠a mobile device mounted Logitech Webcam Pro 9000
(cam2) to record the scene in front of the display of
Figure 1: Test equipment system installed on a user.
(a) (b)
(c) (d)
Figure 2: Video GUI for the following modi: (a)
play, (b) pause, (c) menu, and (d) volume control.
4.3 Data processing
The data acquired for each subject during the experiment
has been processed oďŹine. The user attention has been de-
termined by applying an object detection algorithm [25], to
detect the markers in the two sequences. Particularly, the
markers were placed on cam1 and cam2 in such a way that,
when the user is watching at the mobile phone screen, the
scene recorded from cam1 will include the marker on top
of the mobile phone (Figure 3(a)). Vice versa, the scene
recorded from cam2 will include the frontal userâs face as
well as the marker on userâs head (Figure 3(b)). Addition-
(a) (b)
Figure 3: Sample frames of the video sequences
recorded by (a) cam1 and (b) cam2, when the user
is looking at the phone screen.
(a) (b)
Figure 4: Sample frames of the video sequences
recorded by (a) cam1 and (b) cam2, when the user
is not looking at the phone screen.
phone at that instant.
It is worth mentioning that we also tested a face detection
algorithm [26] to detect the face of the user in the video
recorded by cam2. Unfortunately, too many false positive
and false negative face detections were present in the results.
Particularly, false positive occurred when the face was not
frontal, even if the frontal face cascade classiďŹer has been
used. Some false negative frontal face detections can be
explained by the bad quality of the video frames, due to
userâs motion and abrupt scene illumination changes.
Such frame quality degradations were found to create prob-
lems to the marker detection algorithm as well, however,
they were considerably fewer compared to the face detec-
tion. Furthermore, the marker detection did not show any
false positives. The false negative marker detections were
removed by applying a correction algorithm that is based
on the assumption that it is impossible for the user to look
away from the phone for a very short instance in time. To
be precise, if the marker was not detected for a succession of
k frames but it was detected in its m previous and n follow-
Figure 5: Path of the experiment reconstructed us-
ing the GPS data from one test session.
lecting information regarding the focus of attention of the
users.
The information regarding the detection of the marker
on the frames captured at each time instance from the two
cameras, together with the information regarding the video
player status at the same time instance, facilitates to identify
in which of the three consumption states (see Section 3.1)
the user is during the entire duration of the experiment. The
GPS data additionally allows to distinguish between the mo-
bility states (static or dynamic) based on a speed estimation
of the user. The path followed during the experiment, re-
constructed on the Earth map using the acquired GPS data
of one user, is shown in ďŹgure 5.
31. IMTC 20th Anniversary Forum â Porto, Portugal
3D QoE in mobile multimedia
⢠Comparison of different methods
to produce 3D on mobile phones
- 2D
- Anaglyph
- Motion Parallax - Wiggle
- Autostereoscopic
31
(a) Overall quality (b) Perceived depth
Fig. 9. MOS/CI of the individual test conditions.
cessing Magazine, IEE
jan. 2011.
[3] M. Rerabek, L. Goldm
âMotion parallax bas
legacy consumer mob
nal Processing (MMSP
Workshop on. IEEE, 20
[4] P. Benzie, J. Watson
K. Hopf, H. Urey, V. S
survey of 3dtv display
Circuits and Systems fo
Fig. 2. Motion parallax based 3D restitution.
2.2.1. Multi-view motion parallax
The idea of portable device restitution method based on mo-
tion parallax employing the multi-view image data-set is il-
lustrated in Fig. 2. By rotating the portable device, the viewer
controls the view which is rendered on the display and so
he/she can observe the scene from different positions. This
way the viewer imitates a process similar to what he/she does
when observing the real world.
When the number of images in a stereoscopic or multi-
scopic image set is too small to achieve a smooth restitution,
intermediate images are generated using depth image based
rendering [9]. The depth estimation and view synthesis tools
of the 3D video coding (3DV) framework [10] developed by
MPEG are used for the content generation process within
our experiments. The depth estimation reference software
(DERS) uses three camera views (left, center, right) together
with the intrinsic and extrinsic camera parameters to estimate
the depth map of the center view. From the three operation
modes (automatic, segmentation, semi-automatic) of DERS,
the automatic mode has been used. Once the depth maps
were obtained the view synthesis reference software (VSRS)
was used to synthesize intermediate views. A virtual view is
generated based on two reference views with the correspond-
ing depth maps as well as the intrinsic and extrinsic camera
parameters. This is achieved by depth and texture mapping
and hole ďŹlling for each of the reference views followed by
Fig. 3. Content restitu
play.
corresponds to anothe
of a view depends sol
moves the display.
3. QU
3.1. Image data prep
A subset of the multi
Video (3DV) data-set
More speciďŹcally 4 s
Mobile) from the clas
one was used for train
Suitable frames h
each video. Using the
reference software (D
reference software (V
views were synthesize
mobile phone does no
MPEG tools, the resu
ther converted to high
tial resolution of the
match that of the por
Android based smart
Fig. 4. Creation of the individual test conditions from multi-
scopic data-set.
(a) Ballons
32. IMTC 20th Anniversary Forum â Porto, Portugal32
Trends in QoE
⢠Digital world has (re-)discovered the notion of
quality
â Lower quality content is increasingly less tolerated by
end-users
â Digital technology have surpassed barriers of
attainable quality, while remaining cost effective
⢠Increasing interest in QoE
â Extending from device-centric and system-centric
quality optimization to end-to-end and especially
user-centric optimization
33. IMTC 20th Anniversary Forum â Porto, Portugal33
Trends in QoE community building
⢠Increased interest in workshops and conferences
around the notion of quality assessment and metrics
â QoMEX: International Workshop on Quality of Multimedia
Experience (http://www.qomex.org)
â VPQM: International Workshop on Video Processing and Quality
Metrics for Consumer Electronics (http://www.vpqm.org)
â âŚ
⢠QoE is one of the issues referred to in research
programs funded by the EC
â Various calls in the recent year include QoE issues
â QoE is one of the top topics in future framework program
â Already ongoing initiatives such as COST Action IC1003
(Qualinet)
34. IMTC 20th Anniversary Forum â Porto, Portugal34
Trends in standardization
⢠Standardization efforts in quality assessment and metrics
â Video Quality Experts Group (VQEG)
â ITU-T SG 12 (Performance, QoS and QoE)
â JPEG (Advanced Image Coding - AIC, HDR image compression)
â MPEG (HEVC, 3DVC)
â âŚ
35. IMTC 20th Anniversary Forum â Porto, Portugal35
Challenges ahead
⢠Some key issues in QoE:
â Content-dependent quality assessment methods and metrics
â Context-dependent quality assessment methods and metrics
â Quality assessment methods and metrics beyond AV (haptics, âŚ)
â Multi-modal quality assessment methods and metrics (AV, âŚ)
â 3D quality assessment methods and metrics (3D sound, 3D video, âŚ)
â HDR content quality assessment methods and metrics
â Interaction quality metrics (closely related to usability)
â Presence/immersion quality metrics
â âŚ
⢠Need for Quality Certification Mechanisms of multimedia
services and products
â Similar in idea to ISO 9000 series
36. IMTC 20th Anniversary Forum â Porto, Portugal36
What does this all mean to you?
⢠Era of user-centric multimedia has already started
â It is not anymore sufficient to merely add new features and
functionalities to multimedia systems
â True added value in terms of impact on user experience of such
features and functions should be demonstrated
â Quality of Experience plays a central role in this new game
⢠Opportunities in research, technology, business, art and
entertainment
37. IMTC 20th Anniversary Forum â Porto, Portugal37
Thank you for your attention
Questions?