Presented at UMAP 2020 Abstract: Video summaries or highlights are a compelling alternative for exploring and contextualizing unprecedented amounts of video material. However, the summarization process is commonly automatic, non-transparent and potentially biased towards particular aspects depicted in the original video. Therefore, our aim is to help users like archivists or collection managers to quickly understand which summaries are the most representative for an original video. In this paper, we present empirical results on the utility of different types of visual explanations to achieve transparency for end users on how representative video summaries are, with respect to the original video. We consider four types of video summary explanations, which use in different ways the concepts extracted from the original video subtitles and the video stream, and their prominence. The explanations are generated to meet target user preferences and express different dimensions of transparency: concept prominence, semantic coverage, distance and quantity of coverage. In two user studies we evaluate the utility of the visual explanations for achieving transparency for end users. Our results show that explanations representing all of the dimensions have the highest utility for transparency, and consequently, for understanding the representativeness of video summaries.

1. Eliciting User Preferences
for Personalized Explanations
for Video Summaries
https://oana-inel.github.io
@oana_inel, @navatintarev, @laroyo
Oana Inel, Nava Tintarev, Lora Aroyo
UMAP 2020

2. massive amount of
digital video content
to explore
2
Computer vector created by macrovector - www.freepik.com: https://www.freepik.com/free-photos-vectors/computer

3. On YouTube 500h
of video uploaded
each min
3
Logo vector created by freepik - www.freepik.com: https://www.freepik.com/free-photos-vectors/logo
https://www.tubefilter.com/2019/05/07/number-hours-video-uploaded-to-youtube-per-minute/

4. Baby vector created by freepik - www.freepik.com: https://www.freepik.com/free-photos-vectors/baby
https://www.tubefilter.com/2019/05/07/number-hours-video-uploaded-to-youtube-per-minute/
… one needs more than 82 years
to watch the amount of video uploaded on a day
4

5. The Problem:
How do we quickly find the relevant videos?

6. The Problem:
How do we quickly find the relevant videos?

7. Typical Solution:
video summaries are used to provide a quick look into
long and dense video material

8. For every video we have the possibility of
thousands video summaries

9. 9
But, video summarization process
is not transparent to the end users
● Certain aspects of the video could be
amplified, diminished or omitted
● It is not clear which fragments of the original
video were included in / excluded from the
video summary

10. 10
But, video summarization process
is not transparent to the end users
● Certain aspects of the video could be
amplified, diminished or omitted
● It is not clear which fragments of the original
video were included in / excluded from the
video summary
How can we make this transparent to end users?

11. 11
What kind of visual explanations
are most useful to understand
how representative a video summary is
with regard to the original video?
Research Question

12. What do users find useful?
12
Survey
Media Scholars
& Collection
Owners

13. What do users find useful?
Video Summary should:
- Preserve chronology of the original video in the video summary
- Have an audio track
- Link to original video.
13
Survey
Media Scholars
& Collection
Owners

14. What do users find useful?
Video Summary should:
- Preserve chronology of the original video in the video summary
- Have an audio track
- Link to original video.
Video Summary Explantation should:
- Be short text
- Shown together with video summary
- Indicate what was covered / not covered from the original video
14
Survey
Media Scholars
& Collection
Owners

15. How to make the representativeness of video
summaries transparent to end users?
15
DimensionsofTransparency
Semantic
Coverage
Semantic
Prominence
Quantity
Coverage
Distance

16. How to make the representativeness of video
summaries transparent to end users?
16
DimensionsofTransparency
● concepts covered in a summary and/or original video
● concept prominence in a summary and/or original video
● fraction of concepts covered / not covered in the summary
● differences between the summary and the original video.
Semantic
Coverage
Semantic
Prominence
Quantity
Coverage
Distance

17. How to design transparent
visual explanations for video summaries?
17
DimensionsofTransparency
Semantic
Coverage
Semantic
Prominence
Quantity
Coverage
Distance

18. How to design transparent
visual explanations for video summaries?
18
VisualizingDimensionsofTransparency
Semantic
Coverage
Semantic
Prominence
Quantity
Coverage
Distance
Wordclouds

19. How to design transparent
visual explanations for video summaries?
19
VisualizingDimensionsofTransparency
Semantic
Coverage
Semantic
Prominence
Quantity
Coverage
Distance
Donut charts

20. Explanation Type 1: Summary Wordcloud
Video Summary Video Summary Explanation
Transparency
Dimensions
Semantic
Coverage
Quantity Coverage

21. Explanation Type 2: Overlap Fraction
Video Summary Video Summary Explanation
Transparency
Dimensions
Quantity Coverage
Distance

22. Explanation Type 3: Overlap Wordcloud
Video Summary Video Summary Explanation
Transparency
Dimensions
Semantic
Coverage Distance
Semantic
Prominence

23. Explanation Type 4: Combined
Wordcloud+Fraction
Video Summary Video Summary Explanation
Transparency
Dimensions
Semantic
Coverage Distance
Semantic
Prominence
Quantity Coverage

24. 24
How can we automate the generation of
such visual explanations?

25. Explanation Generation Methodology
Concept
Extraction
Label
Extraction
Entity
Extraction
1

26. Explanation Generation Methodology
Concept
Extraction
Key Concepts
Selection
EVENTPEOPLE ORGANIZATION
Label
Extraction
Entity
Extraction
High # of occurrences
2
LOCATION
1

27. Explanation Generation Methodology
Concept
Extraction
Key Concepts
Selection
EVENT LOCATIONPEOPLE ORGANIZATION
Label
Extraction
Entity
Extraction
High # of occurrences
Key Concept
Prominence
Concept Frequency
Normalization
31 2

28. 28
How do we know it works?

29. 29
How can we measure the utility of each
visual explanation?

30. User Study - Explanation Utility
Online Study
Amazon mTurk
20 videos
1 video summary
for each video
20 participants
per video

31. User Study - Explanation Utility
Online Study
Amazon mTurk
20 videos
1 video summary
for each video
20 participants
per video Video
Summary
Summary
Wordcloud
Combined
Wordcloud +
Fraction
Overlap
Fraction
Overlap
Wordcloud

32. User Study - Explanation Utility
Online Study
Amazon mTurk
20 videos
1 video summary
for each video
20 participants
per video
Video
Summary
Summary
Wordcloud
Combined
Wordcloud +
Fraction
Overlap
Fraction
Overlap
Wordcloud
● Combined Wordcloud + Fraction most preferred

33. User Study - Explanation Utility
Online Study
Amazon mTurk
20 videos
1 video summary
for each video
20 participants
per video
Video
Summary
Summary
Wordcloud
Combined
Wordcloud +
Fraction
Overlap
Fraction
Overlap
Wordcloud
● Overlap Fraction least preferred

34. User Study - Explanation Utility
Online Study
Amazon mTurk
20 videos
1 video summary
for each video
20 participants
per video
Video
Summary
Summary
Wordcloud
Combined
Wordcloud +
Fraction
Overlap
Fraction
Overlap
Wordcloud
● Summary Wordcloud - least preferred when many prominent
concepts in the video are not covered in the summary

35. 35
Are the visual explanation useful when
comparing two video summaries?

36. User Study - Video Summary Representativeness
Online Study
Amazon Mechanical
Turk
18 videos
2 video summaries
for each video
20 participants per
video
Combined
Wordcloud+Fraction

37. User Study - Video Summary Representativeness
Online Study
Amazon Mechanical
Turk
18 videos
2 video summaries
for each video
20 participants per
video
Combined
Wordcloud+Fraction
Video
Summary 1 Summary 2

38. Users make use of the four dimensions to assess the
representativeness of video summaries.
Semantic Coverage
Semantic Prominence
Quantity Coverage
Distance
“Both have small variations but do not show enough of the main
topics to get an understanding of the events.”
Video Summaries are not Representative
Video Summaries are Very Similar
“... the emphasis is different, but the percentages of concepts in the
video summaries are the same”
Video Summaries Representativeness is Different
Semantic Coverage
“The words and summary shown in this image are more prevalent to
the original clip.”
“covers 10% more of the video concepts”Quantity Coverage

39. Our Data
39
● an annotated corpus of 200 videos and 800 video summaries with key concepts extracted
from video subtitles and video frames;
● a corpus of 3200 visual explanations, with different levels of transparency;
Dataset & Code:
https://github.com/oana-inel/FAIRView-VideoSummaryExplanations

40. Future Work
40
● Improve the detection and assessment of key video concepts
● Study the effect of video summary duration
● Experiment with new visual explanations

41. 41
https://oana-inel.github.io
@oana_inel, @navatintarev, @laroyo
Dataset & Code:
https://github.com/oana-inel/FAIRView-VideoSummaryExplanations
User Study - Utility:
https://tinyurl.com/FairView-UtilityStudy
User Study - Representativeness
https://tinyurl.com/FairView-RepresentStudy
Collaborators