https://imatge.upc.edu/web/publications/keyframe-based-video-summarization-designer This Final Degree Work extends two previous projects and consists in carrying out an improvement of the video keyframe extraction module from one of them called Designer Master, by integrating the algorithms that were developed in the other, Object Maps. Firstly the proposed solution is explained, which consists in a shot detection method, where the input video is sampled uniformly and afterwards, cumulative pixel-to-pixel difference is applied and a classifier decides which frames are keyframes or not. Last, to validate our approach we conducted a user study in which both applications were compared. Users were asked to complete a survey regarding to different summaries created by means of the original application and with the one developed in this project. The results obtained were analyzed and they showed that the improvement done in the keyframes extraction module improves slightly the application performance and the quality of the generated summaries.

1. KEYFRAME-BASED VIDEO
SUMMARIZATION DESIGNER
Carlos Ramos Caballero
Advisors: Horst Eidenberger and Xavier Giró I Nieto

2. Contents
 Introduction
 State of the art
 Methodology
 Results assessment
 Conclusions
2

3.  The application: Designer Master
DEMONSTRATION
3

4. Contents
 Introduction
 State of the art
 Methodology
 Results assessment
 Conclusions
4

5. Introduction
 Motivation
 Designer Master: keyframe-based video summarization interface
 Object Maps: system for automatic video summarization
5
Graphical User Interface
(Designer Master)
Computer Vision Engine
(Object Maps)

6. Introduction
 Goals of the thesis
6

7. Introduction
 Goals of the thesis
 Improving the keyframe extraction module
7

8. Introduction
 Goals of the thesis
 Improving the keyframe extraction module
 Assessing the improvement
8

9. Contents
 Introduction
 State of the art
 Methodology
 Results assessment
 Conclusions
9

10. State of the art
 Shot segmentation
10
Hierarchical decomposition and representation of video content [1]
[1] http://www.scholarpedia.org/article/Video_Content_Structuring

11. State of the art
 Shot segmentation example
11
Shot boundary detection example [2].
[2] Martos, M. “Content-based Video Summarization to Object Maps”, Vienna University of Technology, Austria (2013).

12. State of the art
 Shot segmentation techniques
 Pixel-to-pixel methods
• Global pixel-to-pixel
• Cumulative pixel-to-pixel
 Histogram-based methods
• Simple histogram
• Maximum histogram
• Weighted histogram
 Hausdorff method
12

13. Contents
 Introduction
 State of the art
 Methodology
 Results assessment
 Conclusions
13

14. Methodology: Implemented solution
 System architecture overview
14

15. Methodology : Implemented solution
 Uniform sampling
𝑓𝑝𝑠𝑖: frame rate of the input video.
𝐿𝑖: total number of frames of the input video.
𝑁0: total number of frames we want to keep (𝑁0=100).
15

16. Methodology : Implemented solution
 Gray scale domain
16
Color model transformation RGB to YIQ.

17. Methodology : Implemented solution
 Difference computation
Where 𝐼(𝑡,𝑖,𝑗) represents the intensity value at frame t in pixel(𝑖,𝑗).
X and Y are the width and height of the video frames, respectively.
17

18. Methodology : Implemented solution
 Normalization
Where 𝑑̂ is the normalized value, 256 is the number of grey levels, X and Y are the
width and height of the video frames, respectively.
18

19. Methodology : Implemented solution
 Decision making
The threshold value used in our application is 𝜏 = 0.1 (as defined in [2]).
19
[2] Martos, M. “Content-based Video Summarization to Object Maps”, Vienna University of Technology, Austria (2013).

20. Methodology: Environment
 Environment
20

21. Contents
 Introduction
 State of the art
 Methodology
 Results assessment
 Conclusions
21

22. Results assessment
 TEST 1: Testing the applications + ‘in situ’ survey
 11 participants
 Test data: The intouchables trailer
22

23. Results assessment
 Example: pair of summaries
23
Designer Master v1 Designer Master v2

24. Results assessment
 TEST 2: web-based survey
 43 participants
 Test data: The Intouchables trailer
24

25. Results assessment
 EVALUATION
 Quality of the generated summaries
 Representativeness of the generated summaries
 Mean Opinion Score
• 1. Unacceptable
• 2. Poor
• 3. Good
• 4. Very good
• 5. Excellent
25

26. Results assessment
 Quality generated summaries
“Please, rate summary 1”
26
“Please, rate summary 2”

27. Results assessment
 Quality generated summaries
27
MOS MOS – scores distribution

28. Results assessment
 Representativeness of the summaries
“Which summary let you better recognize the video content?”
28

29. Results assessment
 Representativeness of the summaries
29

30. Results assessment
 Ease-of-use of the application
“Do you think the application is intuitive and easy to use?”
30

31. Results assessment
 Ease-of-use of the application
31

32. Results assessment
 Execution time
32

33. Contents
 Introduction
 State of the art
 Methodology
 Results assessment
 Conclusions
33

34. Conclusions
 Accomplishment of the initial goals
 Improving the keyframe extraction module by integrating both
projects.
 Assessing the improvement.
34

35. Conclusions
 Accomplishment of the initial goals
 Improving the keyframe extraction module by integrating both
projects.
 Assessing the improvement.
 Our work has slightly improved Designer Master
 Users can create better video summaries and easily due the better
quality of the extracted keyframes.
35

36. Conclusions
 Accomplishment of the initial goals
 Improving the keyframe extraction module by integrating both
projects.
 Assessing the improvement.
 Our work has slightly improved Designer Master
 Users can create better video summaries and easily due the better
quality of the extracted keyframes.
 It is hoped to develop this work into a product for the Austrian
Broadcasting station ORF
36

37. Conclusions
 Accomplishment of the initial goals
 Improving the keyframe extraction module by integrating both
projects.
 Assessing the improvement.
 Our work has slightly improved Designer Master
 Users can create better video summaries and easily due the better
quality of the extracted keyframes.
 It is hoped to develop this work into a product for the Austrian
Broadcasting station ORF
37

38. Thank you very much for your attention!
Danke schön!
Moltes gràcies!
38