Presentation of my scientific paper to the Third International Workshop on Video Processing and Quality Metrics (VPQM2007).

1. A new methodology to estimate
the impact of H.264 artefacts on
subjective video quality
Stéphane Péchard, Patrick Le Callet, Mathieu Carnec, Dominique Barba
Université de Nantes – IRCCyN laboratory – IVC team
Polytech’Nantes, rue Christian Pauc, 44306 Nantes, France
Third International Workshop on Video Processing and Quality Metrics for Consumer Electronics
Scottsdale, 2007-01-26

2. Introduction
• Codec => Coding artefacts
• Quality loss due to artefacts
=> Useful for quality metrics or better coding …
• Possible practical approach
– Artefact classification
– Annoyance or quality loss contribution per artefact type
2

3. Farias and al. methodology
Farias VPQM 05
• Artefacts type set (blockiness, blur, ringing, …)
• Generation of synthetic artefacts
– Strength parameter
– Applied with the same strength on a whole part of the
sequence
• Subjective assessment
=> Annoyance curve per artefact type regarding the
strength
=> Content dependency
Alternative approach : VPQM07
⇒ H.264 coding, Subjective assessment : quality scale, blur
scale, blockiness scale …
3
⇒ No direct control of artefacts strength

4. Proposed approach
• H.264 artefacts due to quantization/decision
• effects are different regarding the local content
(edge, texture, …)
• different perceived annoyance depending on the
local spatio-temporal activity of the content
• H264 distortions only in selected coherent spatio-
temporal regions => define content categories
• Subjective quality assessment
⇒ Quality loss curve per local content category
(e.g. effects of H264 on each category)
⇒ Strength ? 4

5. Outline
• Spatio temporal segmentation
• distorted sequences generation
• subjective quality assessment of sequences
• Quality assessment : Combining categories
• Towards quality loss function per content
category
5

6. The approach
source temporal unlabeled borders
classification
segmentation holes filling processing
categories masks sequence
C-distorted
……
H.264 coding sequences Ci
generation
partly-distorted sequences
usable for subjective tests
6

7. Spatio temporal classification
2 steps
- temporal segmentation :
reliability regarding the motion
=> temporal tubes
- tube classification :
Regarding spatial content
7

8. Segmentation of sequences
source temporal unlabeled borders
classification
segmentation holes filling processing
Class-distorted
……
H.264 coding sequences Ci
generation
partly-distorted sequences
usable for subjective tests
8

9. Segmentation of sequences
• per group of five successive frame, the center frame
is divided into blocks
• motion estimation of each block using the two
previous frames and the two next frames
(motion estimation performed on a multi-resolution representation)
i-2 i-1 i i+1 i+2 9

10. Segmentation of sequences
• temporal tracking of each block of frame i
defines a spatio-temporal “tube” over the
five frames
• a tube is oriented along the local motion
i-2 i-1 i i+1 i+2 10

11. Classification
source temporal unlabeled borders
classification
segmentation holes filling processing
C-distorted
……
H.264 coding sequences Ci
generation
partly-distorted sequences
usable for subjective tests
11

12. Definition of content categories
HVS has different perception of impairments
depending on the local spatio-temporal content.
• low luminance smooth areas;
• high luminance smooth areas;
• fine textured areas;
• edges;
• strong textured areas
12

13. Classification
• 4 spatial gradients means per
tube (directions : 0, 90, 45 and
135°)
• plot in spatial space P (0 and
90°) => C1, C2, C3 and C4
• 2nd step : space P’ (45 and
135°) used to discriminate C5
in P
• frontier determined to obtain
relevant classification
13

14. Classification
• global tracking of moving objects over the whole
sequence
• tubes are classified then merged by categories
smooth areas with
low luminance
smooth areas with
high luminance
fine textured areas
edges
strong textured
areas
14

15. Unlabeled holes filling and tube
intersections
source temporal unlabeled borders
classification
segmentation holes filling processing
C-distorted
……
H.264 coding sequences Ci
generation
partly-distorted sequences
usable for subjective tests
15

16. Unlabeled holes filling and tube
intersection
• every pixel of the source has one and only one label
• unlabeled holes :
– gradient value => class
– closest tube
• Insection pixels : same
16

17. Borders processing
source temporal unlabeled borders
classification
segmentation holes filling processing
C-distorted
……
H.264 coding sequences Ci
generation
partly-distorted sequences
usable for subjective tests
17

18. Borders processing
• borders between original and distorted large regions
are treated so as to smooth the transitions
before after
borders borders
processing processing
18

19. H.264 coding and class-distorted
sequences generation
source temporal unlabeled borders
classification
segmentation holes filling processing
category masks sequence
C-distorted
……
H.264 coding sequences Ci
generation
partly-distorted sequences
usable for subjective tests
19

20. Partly-distorted sequences generation
original sequence
C1
C2
H.264 sequences at different bitrates C3
C4
C5
categories sequence
20
5 sequences per bitrate

21. Original sequence (first frame)
21

22. One caregory distorted sequence (first
frame)
22

23. Subjective quality assessment
• SAMVIQ protocol with at
least 15 validated
observers and
normalized conditions
• 1920x1080 HDTV Philips
LCD display
• Doremi V1-UHD 1080i
HDTV player
23

24. Subjective quality assessment
• 11 sequences in a SAMVIQ session:
– 5 Ci-only distorted at a certain bitrate B
– entirely distorted sequence at B
– entirely distorted sequence at low bitrate
– entirely distorted sequence at intermediate
bitrate
– explicit and hidden references
24

25. Sequences
uncompressed HDTV sequences from SVT
Above marathon Captain Dance in the woods Duck fly
C5 50 % C2 78 % C3 54 % C5 60 %
Fountain man Group disorder Rendezvous Ulriksdals
C2 71 % C2+C3+C1 95 % C5 56 % C2+C3 80 %
25

26. example on sequence Ulriksdals
coded at 1 Mbps
90
80
70
60
50
40
30
20
10
0
1 2 3 4 5
Classes MOS(Sj,Bk) MOSref
26

27. DMOS and ∆MOS
MOSref
• MOS(Ci, Sj ,Bk) for each sequence
∆MOS(C4)
Sj, each category Ci at each bitrate
Bk MOS(C4)
∆MOS(C5)
• DMOS(Sj ,Bk) = MOSref – MOS(Sj ,Bk)
is the quality difference between MOS(C5)
the reference and the entirely ∆MOS(C3)
distorted sequence DMOS(Sj,Bk)
MOS(C3)
• ∆MOS(Ci, Sj ,Bk) = MOSref - MOS(Ci, ∆MOS(C1)
MOS(C1)
Sj ,Bk) is the quality loss induced
by distortions in category Ci
MOS(C2) ∆MOS(C2)
MOS(Sj,Bk) 27

28. Possible relation between
global DMOS and category ∆MOS?
Combination CC
∆MOS(C2)+ ∆MOS(C4) + ∆MOS(C5) 0.9485
∆MOS(C2) + ∆MOS(C5) 0.9440
• relations use ∆MOS(C2) + ∆MOS(C3) + ∆MOS(C4) 0.9094
∆MOS(C1) + ∆MOS(C2) + ∆MOS(C3)
sums of ∆MOS 0.9058
+ ∆MOS(C4) + ∆MOS(C5)
… …
∆MOS(C2) 0.7664
∆MOS(C3) 0.7094
∆MOS(C5) 0.6400
∆MOS(C4) 0.5472
∆MOS(C1) 0.5349
28

29. Non linear functions
• DMOSp = maxi(∆MOSi)
– CC = 0.9467
• DMOSp = maxi(∆MOSi) + maxj(∆MOSj) with j≠i
– CC = 0.9530
• Correlation exists between global DMOS and
category ∆MOS
=> DMOS could be predicted from quality per
category
29

30. Towards a quality loss model
• How to control the distortion level of a given
class ?
– Farias approach :strength of synthetic artefact
• Factors implied in the quality loss of
category Ci:
– distortions themselves
– motion
– proportion of the category
– spatial localisation (not considered here)
30

31. Distortion strength for category C1
• distortion strength = f(M,P,E)
With all along the sequence :
– M the mean motion of the category;
– P the mean proportion of the category;
– E the MSE on the category;
• M decreases the distortion strength while P and E
increase DS
proposed model for f
DS = (1 — M/Mt)×P×E
31

32. Quality loss function for category C1
• Psychometic function as a prediction of ∆MOS1
φ(DS) = (a×DSb)/(c+DSb)
• correlation between φ(DS) and ∆MOS1 : 0.9514
• RMSE = 5.25
• good predictor of the loss of quality induced by
category C1
32

33. Quality loss function for class C1
33
=> Possible prediction of ∆MOS1

34. Conclusion
• design of a new methodology to estimate the
impact of H.264 artefacts on subjective
video quality
• One distortion type but
– Effect related to local content
– possibility to relate the global loss to loss per
category
– quality loss function for category C1
• Other categories and objective models
34

35. Results: segmentation statistics
Above marathon Captain Dance in the woods Duck fly
Séquence
C1 (%) 3.75 13.14 3.80 0.13
C2 (%) 17.45 78.26 22.57 8.97
C3 (%) 27.79 6.81 53.85 19.50
C4 (%) 0.94 1.43 3.02 10.70
C5 (%) 50.06 0.36 16.75 60.70
35

36. Results: segmentation statistics
Fountain man Group disorder Rendezvous Ulriksdals
Séquence
C1 (%) 10.52 25.28 8.78 13.54
C2 (%) 70.71 38.58 12.38 41.31
C3 (%) 13.37 29.80 19.87 40.48
C4 (%) 1.45 1.79 2.05 1.36
C5 (%) 3.93 4.54 56.92 3.30
36