My Ph.D. defense presentation, on October 2nd, 2008.

1. Quality of experience
in High Definition Television:
subjective assessments
and objective metrics
Stéphane Péchard
October, 2nd 2008
IVC 1

2. Motivations
psychological technical
2

3. New technologies
1 10 NEW
11 0
10 0 1 1 N EW
1110100 1 N EW
01 0 0 1
NE W
0
1
1
0
compression
capture transmission restitution
5x SDTV (pixels)
=> new distortions 3

4. Controlling quality
subjective objective
(MOS) (MOSp)
4

5. Outline
Subjective quality Objective quality
assessment metrics
1. global quality 1. H.264-specific metric
assessment (using prior
2. comparing qualities knowledge)
of 2 TV services 2. generic metric based
3. towards a fine on spatio-temporal
quality measurement tubes
5

6. Outline
Subjective quality Objective quality
assessment metrics
1. global quality 1. H.264-specific metric
assessment (using prior
2. comparing qualities knowledge)
of 2 TV services 2. generic metric based
3. towards a fine on spatio-temporal
quality measurement tubes
6

7. What is video quality
subjective assessment?
getting a mean human quality evaluation
observers environment methodology
7

8. Subjective quality
assessment
how quality is globally perceived ?
preference between HDTV and SDTV ?
can we better understand
quality judgment ?
8

9. Outline
Subjective quality Objective quality
assessment metrics
1. global quality 1. H.264-specific metric
assessment (using prior
2. comparing qualities knowledge)
of 2 TV services 2. generic metric based
3. towards a fine on spatio-temporal
quality measurement tubes
9

10. Suitable methodology
HDTV: high quality in a short range
=> quality measure should be precise
and discriminative
+ important part of visual field excited
=> how to consider this in a methodology ?
10

11. Subjective Assessment
Absolute Methodology
Category Rating for Video Quality
European Broadcasting
Union
Video Quality Experts Group
- random order - user-driven order
- only one viewing - multiple viewing (natural?)
- category scale - continuous scale
Good ...
- no explicit reference - explicit reference
11

12. State of the art
[Brotherton, 2006] both MOS (Mean Opinion
Score) populations correlation on CIF (352x288):
CC(MOSACR, MOSSAMVIQ) = 0.94
HDTV
VGA
QVGA
1080
480
240
to confirm: 320
more tests 640
1920
12

13. Results
visual correlation RMSDiff=
field
QVGA 13° 0.969 6.73
VGA 19° 0.942 9.31
HDTV 33° 0.899 14.06
ACR and SAMVIQ are equivalent
up to a certain resolution 13

14. Accuracy vs.
Number of observers
15
confidence interval
10
SAMVIQ
SAMVIQ
ACR'
5
0
5 10 15 20 24
25 30
number of observers
14

15. Outline
Subjective quality Objective quality
assessment metrics
1. global quality 1. H.264-specific metric
assessment (using prior
2. comparing qualities knowledge)
of 2 TV services 2. generic metric based
3. towards a fine on spatio-temporal
quality measurement tubes
15

16. Comparing two videos
with different resolutions
problem: observers can't move!
H
D=3H for HDTV
16

17. Technical solution
How? no specific protocol exists
comparison
HD QHD ~SDTV QHD in HD
17

18. Motivation
same screen for both formats
QHD: 960x540
H h
TVSD:
720x576
D=3H=6h
18

19. Quality and preference
tests
A: quality tests preference tests
with SAMVIQ A vs. B
of SDTV qualities
(good and mid-range)
preference scale
I prefer much more A than B +3
I prefer more A than B +2
B: quality tests I prefer a little more A than B +1
I have no preference 0
with SAMVIQ I prefer a little less A than B -1
of HDTV qualities I prefer less A than B -2
-3
I prefer much less A than B
19

20. Results
preference
ΔQuality =
0 isopreference
MOSHD - MOSSD
0 ΔQuality
HD/SD Qgood: QHD may be less than QSD,
benefit of the size
HD/SD Qmid-range: QHD must be higher
than QSD, size becomes an enemy 20

21. Outline
Subjective quality Objective quality
assessment metrics
1. global quality 1. H.264-specific metric
assessment (using prior
2. comparing qualities knowledge)
of 2 TV services 2. generic metric based
3. towards a fine on spatio-temporal
quality measurement tubes
21

22. Classical approach
...
a global distortion on an entire sequence
22

23. Farias approach-2004 Proposed approach
distortion-based partition content-based partition
blur
homogeneous
... areas
blockiness blur strong textured areas
from disturbance functions
to global distorting system fine textured t
areas
Drawbacks
content dependency
coding system dependency
from spatio-temporal
distortion list exhaustivity category qualities
pooling function?
complex subjective assessment to global quality? 23

24. spatio-temporal classification
source
C1 C2 C3 C4 C5
spatio-temporal segmentation
(tube creation tube classification
along local motion)
categories
masks sequence
class-distorted sequences
H.264 coding generation
……
partly-distorted sequences
usable for subjective tests
24

25. Local to global?
MOS(Ci): partly-distorted sequence qualities
related to global MOSG: f(MOS(Ci)) = MOSG ?
several relation tested:
up to CC(f(MOS(Ci)), MOSg) = 0.95
YES! It's possible to relate spatio-temporal
category qualities to global quality
25

26. Farias approach-2004 Proposed approach
distortion-based partition content-based partition
blur
homogeneous
... areas
blockiness blur strong textured areas
from disturbance functions
to global distorting system fine textured areas t
Drawbacks Advantages
content dependency generic methodology
coding system dependency simple pooling function
distortion list exhaustivity real distortions
pooling function? classical subjective assessment
complex subjective assessment
26

27. Outline
Subjective quality Objective quality
assessment metrics
1. global quality 1. H.264-specific metric
assessment (using prior
2. comparing qualities knowledge)
of 2 TV services 2. generic metric based
3. towards a fine on spatio-temporal
quality measurement tubes
27

28. What are objective
quality metrics?
reference reduced
distorted
reference
sequence
system extraction NR metric
RR metric
FR
performance objective
evaluation
criteria scores
(CC, RMSE, OR,
difference signifiance)
MOS from subjective
assessments 28

29. Usual approaches
high level distorstions
measurement models
PSNR VQM [2002] low level
HVS models
structural
models
VSSIM [2004]
signal perceptual
approach approach
29

30. Performances on HDTV
168 sequences
metric CC RMSE OR
VSSIM 0.790 11.27 0.55
VQM 0.898 8.09 0.40
PSNR 0.543 15.43 0.61
30

31. Outline
Subjective quality Objective quality
assessment metrics
1. global quality 1. H.264-specific metric
assessment (using prior
knowledge)
2. comparing qualities
of 2 TV services 2. generic metric based
on spatio-temporal
3. towards a fine
tubes
quality measurement
31

32. 32

33. reference global motion M
sequence proportions Pi
model
ST content
parameters
analysis
prediction
offset, slope
distorted bitrate B quality
sequence model
quality score Q 33

34. reference global motion M
sequence proportions Pi
model
ST content
parameters
analysis
prediction
use of the spatio-temporal
segmentation offset, slope
10% 20%
distorted bitrate B quality
5%
sequence model
60%
class proportions Pi mean sequence
quality score Q
motion M 34

35. reference global motion M
sequence proportions Pi
model
ST content
parameters
analysis
prediction
offset, slope
offset parameter:
temporal complexity estimation
distorted bitrate Brelated to motion Mi
quality
sequence model
slope parameter:
spatial complexity estimation
related to class proportions Pi
quality score Q 35

36. Performances
metric CC RMSE OR
VSSIM 0.791 11.90 0.45
VQM 0.892 8.79 0.40
proposed 0.901 8.47 0.36
pros cons
reduced reference
metric (6 parameters)
equal performances
faster than VQM H.264-dependent
36

37. Outline
Subjective quality Objective quality
assessment metrics
1. global quality 1. H.264-specific metric
assessment (using prior
knowledge)
2. comparing qualities
of 2 TV services 2. generic metric based
on spatio-temporal
3. towards a fine
tubes
quality measurement
37

38. Interesting HVS features
for this metric
Visual inspection (gaze fixation)
spatially localized
duration (200-300 ms)
smooth local motion tracking
some of them have been used in part 1
38

39. reference spatio-temporal distorted
sequence segmentation sequence
tubes
features features
extraction extraction
features
difference
short-term long-term
quality
spatio-temporal temporal score Q
pooling pooling
39

40. reference spatio-temporal distorted
sequence segmentation sequence
tubes
features features
extraction a tube t extraction
features
difference
short-term
temporal quality
spatio-temporal
pooling score Q
pooling
40

41. reference spatio-temporal distorted
sequence segmentation sequence
tubes
features features
extraction spatial information feature: fSI
extraction
features
temporal information feature: fTI
difference
reference distorted
tube - tube
short-term
temporal quality
spatio-temporal
pooling score Q
pooling
41

42. reference spatio-temporal distorted
sequence segmentation= sequence
tubes
features 5 frames 1 time-slot (200ms)
features
extraction extraction
features
difference
short-term long-term quality
spatio-temporal temporal score Q
pooling pooling
42

43. reference spatio-temporal distorted
sequence segmentation sequence
high level HVS properties
tubes
features mid-term features
asymetrical
extraction long-term
extraction
non linear
temporal features
quality
temporal
filtering difference filtering
judgment
short-term long-term
quality
spatio-temporal temporal score Q
pooling pooling
43

44. Training and testing
168 sequences
testing
training 44

45. Best performances
metric CC RMSE OR
VSSIM 0.837 10.15 0.38
VQM 0.875 8.98 0.43
fixed tubes 0.875 9.08 0.38
motion-oriented tubes 0.898 8.30 0.31
generic metric
slightly better than VQM with less features
45

46. General conclusion
46

47. Subjective quality
assessment
better knowledge of HDTV (visual)
subjective quality assessment
visual image size influences preference
between SDTV/HDTV services
generic methodology to assess fine quality
=> better knowledge
of judgment construction
47

48. Experiment effort
26 sessions (6 months)
(SAMVIQ, ACR and preference)
200 observers for 600 unique sessions
in 300 hours of subjective evaluation
=> 25,000 subjective scores
more than 750 cumulative days
of H.264 coding
48

49. Objective quality metrics
fast RR metric dedicated
to H.264 systems evaluation
generic metric based on motion-oriented
spatio-temporal tubes
both performed slightly better than VQM
49

50. Future works
adapting ACR to HDTV: more than 5 items?
=> work in progress (VQEG)
considering a display model
=> work in progress (Tourancheau)
towards a multimodal quality evaluation
50

51. Q&A
51

52. HDTV sequence database
ref
24
--------
7
52

53. SAMVIQ ACR
100
excellent 5 80% excellent
good 4 good
fair 3 fair
poor 2 poor
bad 1 bad
0
53

54. CC=0.899
RMSE=14.06
54

55. large screen effect distorsions effect
HDTV prefered
mean preference
ΔMOS <MOS
MOSHD=-18 SD
0
MOSHD
ΔMOS0=-8 >MOS SD
SDTV prefered
ΔMOS=MOSHD-MOSSD
55

56. Classes
five spatial activity levels
smooth areas textured areas edges
low high fine strong
luminance textures
C1 C2 C3 C4 C5
56

57. Tube classification
ΔV
4 spatial gradients
per tube space P
C4
plot in spatial space P C5(P')
frontiers defined C3
to get relevant C1
C2 C4
classification ΔH
57

58. DMOS and ΔMOS
MOSref
MOS4 ΔMOS(C4
)
ΔMOS(C
MOS5 5
DMOS(Sj,Bk)=
)
MOSref - MOS(Sj,Bk)
MOS3 ΔMOS(C3
MOS1 ΔMOS(C1
)
global loss ) local
MOS2 ΔMOS(C2
)
losses
MOS(Sj,Bk) 58