Interactive Video Search - Tutorial at ACM Multimedia 2015

Interactive
Video Search
Klaus Schoeffmann, PhD
Klagenfurt University
Institute of Information Technology
Klagenfurt, Austria
Frank Hopfgartner, PhD
University of Glasgow
School of Humanities
Glasgow, UK

Outline
• Search in video content: motivation and challenges
• Video retrieval and its challenges
• What is interactive video search and how can it help?
• Video browsing
• Video navigation
• Break
• Video content visualization
• Ad-hoc similarity search / video exploration
• Sketch-based search in video
• Evaluation of interactive video search tools
• Visual lifelogging
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 2

Motivation
3Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search

Videos Everywhere
• Ubiquitous use of videos nowadays
Entertainment and commercials
Social gaming (screencasts)
Personal videos (family, kids, …)
Sports documentation and analysis (e.g., GoPro)
Product usage instructions (e.g., furniture)
Surveillance (buildings, places, street, …)
Lifelogging
Health care and medical science (endoscopic procedures)
• Enormous amount of data!

Video as the Ultimate Media?
[Mary Meeker, Liang Wu, Internet Trends, D11 Conference, May, 2013]
As of 2014, every
minute 300 hours of
video are uploaded
to YouTube!

Video Cameras
• Increasingly powerful
These days you can record 4K content with your mobile!
Video sensors use auto-focus, object tracking, color
correction, and image stabilization
Storage space not a big problem
 Current smartphones have 128 GB of memory
 NAS devices cheaply available
Network bandwidth also dramatically increased over years
 Video streaming on the go is simple and common
 LTE connections provide 30 Mbit/s and even much more!

[Mary Meeker, Liang Wu, Internet Trends, D11 Conference, May, 2013]

Challenges
• Video data
are a continuous media: the content depends on time!
often contain several media types: image, text, audio
cannot be simply stored and indexed in a data base,
requires own indexing and search methods!
require huge amount of storage space without compression!
may contain a lot of important information,
which is, however, often very subjective!

Challenges
• Subjectivity of data
How many people use it?
The more the easier!?
• Different levels
 Internet scale (YouTube)
 Country/region
 Company/organization/group
 Individual
Available
meta-data
How to query for a video clip?
How to efficiently retrieve results?
How to effectively present content to the user?

“Poor Man’s Video Search Tool”
VCR in the 1970s provided a similar functionality!

Video Data Still Tedious to Use
• Even with video retrieval tools it is still challenging
to find desired video content
Especially if it is not a publicly available (and popular)
Many problems with querying, in particular for novice users!
• The ultimate goal is to make use of and search in
video as effective as for text
Quickly find relevant content
Compare to interactivity of a text book
 Index, ToC, list of figures/tables, etc.
 Change, extend, copy, bookmark, highlight, etc.

Traditional
Video Retrieval
“Query and browse results”

Search Example (TRECVID KIS 2010)
Find the “video of President Bush standing near sea vessels with
Coast Guard members talking about his pride of the Coast Guard,
immigration, and security issues”.
Video from IACC
public data set!
TRECVID:
see later!

Video Clip Hidden in Huge Collection
14
Internet Archive
with Creative
Commons (IACC)
data set, as used
for TRECVID:
146,788 shots
(~9,000 videos)
Page 1 2 3 …. 38 39 40
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search

1st Trial at YouTube

2nd Trial at YouTube

[ Heesch, D., Howarth, P., Magalhaes, J., May, A., Pickering, M., Yavlinsky, A., & Rüger, S. (2004, November). Video retrieval using search and browsing. In TREC Video Retrieval Evaluation Online Proceedings. ]
17
Content-
based
Feature
Example
Image
Text
Ranked list
of shots
In IACC about
5800 pages. 
Temporal
Context
Traditional Video Retrieval Tool

More Interactive Retrieval Tool
[A. Moumtzidou et al., “VERGE: A Multimodal Interactive Video Search Engine”, Proc. of the 21st International Conference on MultiMedia Modeling (MMM 2015), Sydney, 2015]
kNN Similarity search
based on VLAD vectors
Concept detection with SVM and
five local descriptors (SIFT, SURF,
ORB, ...) and PCA
Hierarchical
keyframe clustering
 Interaction
details later

Challenges for
Video Retrieval

Traditional Video Retrieval Approach
“Query-and-Browse-Resoluts” Paradigm
Works well if (and only if)
users can properly express their needs.
content features can sufficiently describe visual content.
computer vision can accurately detect semantics.
20
Content-
based
Search
Ranked Results
Unfortunately, in practice these assumptions do not hold.

Challenges
Content-based features
 How to understand semantics from pixels? Semantic Gap
Both images show
bears in front
of a landscape.

Database affinity of concept classifiers
Low performance in broad domain
P(k) Precision at level k (after k results)
rel(k) defines if kth retrieved document is relevant
Performance
Gap
22
Challenges
TRECVID 2013 Semantic Indexing (SIN-500):
median “inferred average precision” (infAP) < 0.13
In other words:
88% of the results
are not correct!

 Query-by-concept
 Which concept to use? Choose from a long list of results…
 Query-by-example
 Typically no perfect example available.
 Query-by-sketch
 Users are no artists 
 Query-by-text
 How to describe a desired image by text?
Usability Gap
23
A picture tells a 1000 words.
by marfis75
How to describe a video clip by text???
Challenges

Example: Query by Motion Sketch
• Matching based on trajectory descriptor
• Challenge: may differ a lot among different users
[Ghosal, Koustav, and Anoop Namboodiri. "A Sketch-Based Approach To Video Retrieval Using Qualitative Features." Proceedings of the 2014 Indian
Conference on Computer Vision Graphics and Image Processing. ACM, 2014.]

Well-Known Issues
of “Query-and-Browse-Results” Paradigm
Users cannot formulate or have no query
  provide exploratory search features!
 For example: browsing, filtering, similarity search
Users expect good results (on first page!)
  Use relevance feedback / active learning instead of long lists!
Shots have a temporal context
Videos are dynamic
 Static thumbnails are not informative
 Esp. true for long shots and self-similar content
  skims and visual summaries (“smart playback”)
  sophisticated navigation & content structure visualization
Grid interfaces are not always the best choice
Usability Gap
 See later

Uniform Sampled Frames
from a Video with High Self-Similarity

Needs Special Keyframe Extraction
and Object Detection
[Klaus Schoeffmann, Manfred Del Fabro, Tibor Szkaliczki, Laszlo Böszörmenyi, and Jörg Keckstein, “Keyframe Extraction in Endoscopic Video“, in Multimedia Tools and Applications, Springer, August, 2014]
[Manfred J. Primus, Klaus Schoeffmann, and Laszlo Böszörmenyi, “Instrument Classification in Laparoscopic Videos“, in Proceedings of the International Workshop on Content-Based Multimedia Indexing (CBMI 2015),
Prague, Czech Republic, IEEE, 2015, pp. 1-6]

And Special Browsing Tools /
Visualization
Agglomerative clustering based on visual similarity and temporal information
[Jakub Lokoc, Klaus Schoeffmann, and Manfred Del Fabro, “Dynamic Hierarchical Visualization of Keyframes in Endoscopic Video“, in Proceedings of the 21st International Conference on
MultiMedia Modelling 2015 (MMM 2015), Sydney, Australia, Lecture Notes on Computer Science (LNCS), Vol. 8936, Springer International Publishing, 2015, pp. 291-294]

Where Is the User in Multimedia Retrieval?
29
[Marcel Worring et al., „Where Is the User in Multimedia Retrieval?“, IEEE Multimedia, Vol. 19, No. 4, Oct.-Dec. 2012, pp. 6-10 ]

Interactive vs. “Traditional” Retrieval

Interactive Video Search
And how it can help…

Performance
Gap
32
• Mostly interactive search
• Human computation
• Simple-to-use
• Inflexible and tedious for archives
• Low performance (?)
• Mostly automatic search
• Retrieval engine
• Complicated to use
• Flexible and easier (?) for archives
• Limited performance too!
Usability Gap
Novices Experts
 Combines HCI with CV and MIR

Traditional video retrieval + interactive inspection/exploration/navigation
and rich content visualization in order to satisfy an information need
33
Focuses on search and exploration in
(i) single videos as well as (ii) video collections
Directed Search
Find a specific shot or segment in a video
Find a specific video in an archive
Undirected Search
Searching to discover information
E.g., browse through a video in order to
Learn how the content looks like
See if it is interesting
Not supported by
trad. video retrieval

• Tries to strongly integrate user into search process
Search – Inspect – Think – Repeat
 Exploratory search (“will know it when I see it”)
 Instead of „query-and-browse-results“
User controls search process
 Inspects and interacts
 Most meaningful tool for current need, e.g.
• Content Browsing/Navigation
• Content Visualization and Summarization
• Ad-hoc Querying (e.g., by sketch, filtering, ad-hoc example)

User Studies with Significance Tests!
• Many interfaces proposed without proper evaluation
• Interface A better than interface B?
 comparative user study needed!
 Perform search tasks in exactly the
same setting (data, environment, etc.)
 Logging of interaction behavior
and task solve time
 Questionnaire about subjective workloads
 Statistical analysis with proper tests
(e.g., t-test, ANOVA, Wilcoxon signed-rank, etc.)
• User simulations?
• Evaluation competitions
 Same data set
 Comparative evaluation
 TRECVID, MediaEval, Video Browser Showdown (see later)

IVS Tools:
Video Browsing

37
Video Browsing
[ F. Arman, R. Depommier, A. Hsu, and M-Y. Chiu, Content-based Browsing of Video Sequences, in Proc. of ACM International Conference on Multimedia, 1994, pp. 97-103 ]

The ThumbBrowser
[Marco Hudelist, Klaus Schoeffmann, Laszlo Böszörmenyi. “Mobile Video Browsing with the ThumbBrowser”, Proc. of the International Conference on Multimedia, 2013, pp. 405-406 ]

Video Browser for the Digital Native
39
[Adams, Brett, Stewart Greenhill, and Svetha Venkatesh. "Towards a video browser for the digital native." Multimedia and Expo Workshops (ICMEW), 2012 IEEE International
Conference on. IEEE, 2012.]
“Temporal Semantic Compression” based on tempo function and shot popularity (insight)

Video Browser for the Digital Native
• User study with 8 participants
 Test configuration elements by two tasks
(after presentation + 5 minutes training)
 (i) Browse a familiar movie to find scenes you remember
 (ii) Browse an unfamiliar movie to get a feel for its story or structure
 Questionnaire with
Likert-scale ratings
40
[Adams, Brett, Stewart Greenhill, and Svetha Venkatesh. "Towards a video browser for the digital native." Multimedia and Expo Workshops (ICMEW), 2012 IEEE International
Conference on. IEEE, 2012.]

Thread-Based Browsing of Retrieval Results
• Thread: linked seq. of shots in a specified order
 Query results, visual similarity, semantic similarity, textual similarity
time, …
[De Rooij, Ork, Cees GM Snoek, and Marcel Worring. "Balancing thread based navigation for targeted video search." Proceedings of the 2008 international conference on Content-based image
and video retrieval (CIVR). ACM, 2008.]

Thread-Based Browsing of Retrieval Results
[Ork de
Rooij et al.]

IVS Tools:
Video Navigation

Improving Navigation
44
e.g., on YouTube
default window:
640 pixels = frames
(25 seconds)
Common seeker-bar limits
navigation granularity
[Huerst et al., ICME 2007]
ZoomSlider
Improvements (selected):

Navigation with a Seeker-Bar
Idea of the ZoomSlider
45
Wolfgang Hürst, Georg Götz, and Martina Welte, “Interactive video browsing on mobile devices”, in Proceedings of the 15th International Conference on Multimedia (MULTIMEDIA '07). ACM, pp. 247-256, 2007

Improving Navigation
47
e.g., on YouTube
default window:
640 pixels = frames
(25 seconds)
Common seeker-bar limits
navigation granularity
[Dragicevic et al., CHI 2008]
Direct
Manipulation
[Huerst et al., ICME 2007]
ZoomSlider
Improvements (selected):

Relative Flow Dragging
Background Stabilization
48
Pierre Dragicevic, Gonzalo Ramos, Jacobo Bibliowitcz, Derek Nowrouzezahrai, Ravin Balakrishnan, and Karan Singh. “Video browsing by direct manipulation”, in Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems (CHI '08). ACM, pp. 237-246, 2008
Video browsing by direct manipulation / relative flow dragging

Relative Flow Dragging
• Evaluation with a user study
 16 participants (18-44 years old)
 Direct comparison to seeker-bar navigation
 Navigation tasks, 2 videos (ladybug, cars)
 “Find the position where the ladybug passes over marker X”
 “Find the moment when car X starts moving”
 Flow dragging significantly faster (RM-ANOVA)
by at least 250% (also significantly less errors)
49
Pierre Dragicevic, Gonzalo Ramos, Jacobo Bibliowitcz, Derek Nowrouzezahrai, Ravin Balakrishnan, and Karan Singh. “Video browsing by direct manipulation”, in Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems (CHI '08). ACM, pp. 237-246, 2008

How Do Users Search in
Video With a Common
Video Player?

How do Users Search with Video Players
(Navigate with Seeker-Bars)?
• User study with more than 30 participants
• Known Item Search Tasks
51
[Claudiu Cobarzan and Klaus Schoeffmann, “How do Users Search with Basic HTML5 Video Players?“, in Proceedings of The 20th International Conference on MultiMedia Modeling (MMM2014), 2014]

52

53
Vast amount of content was checked with normal playback!?

Timeinvideo(ms)
54
User start with rough navigation and look more carefully after narrowing down the search area!
Target segment

Improving
Video Navigation on
Touch Devices
Session continues at 3:20 pm

Keyframe Navigation Tree
• Consider findings of study on navigation behavior
• Basic idea inspired by frame stripes (MO images)
• Goal
 very compact visualization
 not as fine as frames but not as coarse as keyframes of shots
 provide different granularity levels for navigation
 previous work has shown that
users typically navigate in a coarse-to-fine grained manner
57
[ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single video based
on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247-258). ACM. ]
[Mueller-Seelich, H., Tan, E.: Visualizing the semantic structure of film and video (2000) ]
compact overview but abstract & very high level of detail (frame-based!)
[Xiaoxiao Luo, Qing Xu, Mateu Sbert, Klaus Schoeffmann, “F-Divergences Driven Video Key Frame Extraction“, in Proc. of the IEEE Int. Conference on Multimedia & Expo (ICME 2014), Chengdu, China, 2014, pp. 6]

• Keyframe selection based on sub-shots (JSD with color histograms)
Cover all important scenes even for long shots (e.g. pans)
Excerpts with three levels of detail: L1: narrow, L2: wide, L3: full keyframe
Used as seeker-bar with synchronized interaction for all levels
Simple touch-based interaction (tap and wipe gestures)
58
L1 (30 shots)
L2 (~ 12 shots)
L3 (~ 3 shots)
[Hudelist, Marco A., Klaus Schoeffmann, and Qing Xu. "Improving interactive known-item search in video with the keyframe navigation tree." MultiMedia Modeling. Springer International Publishing, 2015.]

• Video Player vs. KNT Browser (iPad, 4th generation, 9.7-inch)
• User study with 20 participants (15m/5f)
 Age: 18-40 (mean 28.15, s.d. 6.08)
• Known-item search tasks
 Given 20 seconds long target clip
 Find correct clip in 1-h long video as fast as possible
 200 search tasks in total
 Each participant performed random selection of 10 tasks (5/5)
 latin-square principle to avoid familiarization effects
 Time-out after 3 minutes (“unanswered”)
 Wrong results marked as “erroneous”
• Time measurement, logging, questionnaire (Likert-scale ratings)

Search Time & Performance
• Task solve time, erroneous trials, unanswered trials…
61
28.11
44.18
KNT Browser statistically significantly faster
acc. to dependent paired-samples t-test (t(19) = -3.937 p < 0.005)
7 trials
10 trials
5 trials
22 trials

Subjective Rating
• NASA Task-Load-Index (TLX) questionnaires
62
KNT browser significantly
better in all 7 categories!
Acc. to Wilcoxon signed-
rank tests
(for details see paper)
KNT browser is preferred
search tool for 85% of
tested users (17/20)

IVS Tools:
Content Visualization

Grid Interfaces Aren‘t Enough!
• Many video retrieval systems use a Grid interface!?
Moreover, a grid interface does not allow
for fast human visual search (see later)!
64
A ranked list of results does not convey
the temporal content structure!
• To which video does a shot belong to?
• What is the sequence of shots?
• How long is a shot / scene?

Table of Video Content
(TOVC)
[Goeau et al., ICME 2007]
65
Squeeze / Fisheye
Rapid Visual Serial
Presentation (RSVP)
Improving Visualization
aka “Video Surrogates”
[Wildemuth et al., 2003]
[Wittenburg et al., 2005]

66
VideoTree
[Jansen et al., CBMI 2008]
However, outperformed by
simple “grid of keyframes”
in terms of search time.
Similar concept proposed later
[Girgensohn et al., ICMR 2011]
• Split-based clustering algorithm with
color correlograms.
• Tree not directly shown to the user
(only one level).
Improving Visualization
aka “Video Surrogates”

3D Ring Interface
• Utilization of screen real estate
 Large set of images
 Minor occlusion, slight distortion
• Intuitive interaction
 Rotate and zoom
• Content-based sorting
• “Pop-out images” (in the back)
• Further advantages
 Immediately continue on miss,
scaling
67
Klaus Schoeffmann, David Ahlström, and Marco Andrea Hudelist, “3-D Interfaces to Improve the Performance of Visual Known-Item Search“,
in IEEE Transactions on Multimedia, Vol. 16, No. 7, November, 2014, pp. 1942-1951.

3D Ring Interface - Perspectives
Preferred Design acc. to user study
25% Vertical 66% Horizontal 8.3% Frontal
68

3D interface significantly faster than grid by 12.7%
User Study: Grid vs. Ring (both sorted)
150 images, 12 participants, 1440 trials
69

Extension: Multiple Rings with Vertical Scrolling
70
Klaus Schoeffmann. 2014. The Stack-of-Rings Interface for Large-Scale Image Browsing on Mobile Touch Devices. In Proc. of the ACM Int. Conference on Multimedia (MM '14). ACM, New York, NY, USA, 1097-1100.
Significantly faster search (by about 48%) than common image browser on iPad!

IVS Tools:
Ad-Hoc Similarity Search

The Video Explorer
72
[ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single video based on fast content analysis. In Proceedings of the first annual
ACM SIGMM conference on Multimedia systems (pp. 247-258). ACM. ]

Interactive Navigation Summaries
Allows a user to quickly identify
similar/repeating scenes
73
[ Schoeffmann, K., & Boeszoermenyi, L. (2009, June). Video browsing using interactive navigation summaries. In Content-Based Multimedia Indexing, 2009. CBMI'09. Seventh Int.Workshop on (pp. 243-248). IEEE. ]

Motion Layout: Direction + Intensity
Motion Vector (µ) classification into
K=12 equidistant motion directions
Mapping to Hue channel
74
[ Schoeffmann, K., Lux, M., Taschwer, M., & Boeszoermenyi, L. (2009, June). Visualization of video motion in context of video browsing. In Multimedia and Expo, 2009. ICME 2009. IEEE Int. Conf. on (pp. 658-661). IEEE. ]

75
[ Schoeffmann, K., Lux, M., Taschwer, M., & Boeszoermenyi, L. (2009, June). Visualization of video motion in context of video browsing. In Multimedia and Expo, 2009. ICME 2009. IEEE Int. Conf. on (pp. 658-661). IEEE. ]
Similarity Search (SOI) with Motion Layout

• SOI Search
 Motion-based search by example sequence
 Using Motion Direction histogram Db
 User-selected sequence
 Find most similar sequences
 Compute distance to any possible seq. of same length
 Match if below spec. threshold
Motion Layout (Db)
Match 1 Match 2 Match 3
frame 1 frame n
Similarity Search (SOI) with Motion Layout

Region-of-Interest (ROI) Search
 User selects spatial region-of-interest
 On search
 Compute Euclidian distance of frame F
to every other frame f (acc. to selected region)
 Based on color layout descriptor
…
frame F
frame 1 frame k frame n
User-selected
region (I)
…
d(F,1)=350 d(F,k)=8 d(F,n)=400
77
Similarity Search (ROI) with Color Layout

78
Similarity Search (ROI) with Color Layout

IVS Tools:
Sketch-Based Search

• Color sketches mapped to
feature signatures
• Matched to those of
keyframes
80
1. Sampling keypoints
2. Description through location (x,y),
CIE Lab, contrast and entropy of
surrounding pixels
3. K-means clustering
Feature Signatures
[ Kruliš, M., Lokoč, J. and Skopal, T. (2013). Efficient Extraction of Feature Signatures Using Multi-GPU Architecture. Springer Berlin Heidelberg, LNCS 7733, pp.446-456. ]

Feature Signature-Based Video Browser
81
Color Sketch
(Signature)
Player
Winner of Video Browser Showdown 2014 + 2015
Download demo at: http://siret.ms.mff.cuni.cz/lokoc/vbs.zip
2nd Color Sketch
(optional)
[ Lokoč, J., Blažek, A., & Skopal, T. (2014, January). Signature-Based Video Browser. In MultiMedia Modeling (pp. 415-418). Springer International Publishing. ]

Compact visualization
Simple color-position sketch
Negative
example
Matched key-frames
Time to
2nd sketch
2nd optional
sketch
Interactive-navigation summaryOn demand neighborhood expansion
[Slide: Adam Blazek et al.
(siret research group, Czech Republic)]

Compact Visualization in Detail
[Courtesy of Jakub Lokoc et al.]

Another Example of a Color-Based Browser
Browsing
Area
Map
Segment
Inspection
Color SketchCategories
[Kai Uwe Barthel, Nico Hezel, Radek Mackowiak. Navigating a graph of scenes for exploring large video collections, in Proc. of 22nd International Conference on MultiMedia Modeling (MMM 2016), Lecture Notes in
Computer Science (LNCS), Vol. tbd, Springer International Publishing, 2016, pp. 1-7]

Evaluation of
IVS Tools

TRECVID
http://trecvid.nist.gov/
• International video retrieval competition evaluation
 Annually performed by NIST (Gaithersburg, Maryland, USA)
 Funded by NIST and other US government agencies
 Benchmark for researchers using same data
 Origin in TREC (Text REtrieval Conference, since 1992)
• Founded in 2003, by
 Alan Smeaton (Dublin City University)
 Wessel Kraaij (TNO-ICT, Delft)
• International advisory Committee
 Alex Hauptmann (CMU)
 Michael Lew (Leiden Institute of Advanced Computer Science)
 Georges Quenot (LIG, Grenoble)
 John Smith (IBM Research)
 …
• Local organisation
 Paul Over (NIST)

TRECVID Known-item Search
TRECVID KIS (2010-2012)
models the situation in which
“someone knows of a video, has seen it before, believes it is
contained in a collection, but doesn‘t know where to look”
Automatic Search
 Text-description about the video
 Return ranked list of 100 videos (out of 9000)
Interactive Search
 Pre-processing based on text query
 Searcher browses through result list (e.g., keyframes of shots)
• Interactively find target video as fast as possible
• Within 5 minutes

TRECVID Known-item Search
The Performance of State-of-The-Art Video Retrieval Tools
Known items not found by any team:
Interactive Automatic out of
2010 5 / 24 21% 69 / 300 22% 15 teams
2011 6 / 25 24% 142 / 391 36% 9 teams
2012 2 / 24 17% 108 / 361 29% 9 teams
From: [Alan Smeaton, Paul Over, “Known-Item Search @ TRECVID 2012”, NIST, 2012]

MediaEval 2015
• Search and Anchoring in Video Archives
“Search for Multimedia Content”
 Multi-model textual and visual descriptions of content of interest
“Automatic Anchor Selection”
 Predict key elements of videos as anchor points for hyperlinking
Professional (BBC) and non-professional content (users)
http://www.multimediaeval.org

Video Browser Showdown (VBS)
• Annual performance evaluation competition
 Live evaluation of search performance
 Special session at Int. Conference on MultiMedia Modeling (MMM)
 Demonstrates and evaluates state-of-the-art interactive video search tools
 Idea influenced by VideOlympics (Snoek et al., IEEE Multimedia 2008)
• Focus
 Known-item Search tasks
 Target clips are presented on site
 Teams search in shared data set
 Highly interactive search
 Should push research on interfaces
and interaction/navigation
 Experts and Novices
 Easy-to-use tools and methods
90
Teams connected to a server that issues tasks and evaluates submitted results
http://videobrowsershowdown.org/

Video Browser Showdown (VBS)
• Scoring through VBS Server
• Score (s) [0-100] for task i and team k is based on
Solve time (t)
Penalty (p) based on
number of submissions (m)
91
Maximum solve time (Tmax)
typically 3-5 minutes
[Schoeffmann, K., Ahlström, D., Bailer, W., Cobârzan, C., Hopfgartner, F., McGuinness, K., ... & Weiss, W. (2013). The Video Browser Showdown: a live evaluation of interactive video search tools. International Journal
of Multimedia Information Retrieval, 1-15. ]

Video Browser Showdown 2015
• Search in mid-sized video collections (2016: 200 hours)
 Originally only single video search
• Two different kind of tasks:
 Visual: visual presentation of a 30s target clip
 Textual: textual description of a 30s target clip
• Shared video data from BBC
 2015: 153 video files, about 100.000 shots (9 Mio frames)
Participants:
• Need to find the target clips as quickly as possible
• Get points for each task (the faster the better)
• But only for submission of exact location of target clip
[Schoeffmann, Klaus. "A user-centric media retrieval competition: The video browser showdown 2012-2014." MultiMedia, IEEE 21.4 (2014): 8-13.]

2012: Klagenfurt
11 teams
2013: Huangshan
6 teams
2014: Dublin
7 teams
2015: Sydney
9 teams
VBS 2016: January 5, 2016, Miami, USA (MMM 2016)
http://www.videobrowsershowdown.org/

Two examples (of the 11 tools; single video search only)
94
[Xiangyu Chen, Jin Yuan, Liqiang Nie, Zheng-Jun Zha, Shuicheng Yan, and Tat-Seng Chua, "TRECVID 2010
Known-item Search by NUS", in Proceedings of TRECVID 2010 workshop, NIST, Gaithersburgh, USA, 2011
Jin Yuan, Huanbo Luan, Dejun Hou, Han Zhang, Yan-Tao Zheng, Zheng-Jun Zha, and Tat-Seng Chua, "Video
Browser Showdown by NUS", in Proceedings of th 18th International Conference on Multimedia Modeling
(MMM) 2012, Klagenfurt, Austria, pp. 642-645]
• Keyframe extraction (shots)
• ASR and OCR
• HLF (Concepts)
• RF with Related Samples
• Uniform sampled keyframes
(with flexible distance)
• Parallel playback + navigation
[Manfred Del Fabro and Laszlo Böszörmenyi, "AAU Video Browser: Non-
Sequential Hierarchical Video Browsing without Content Analysis", in
Proceedings of th 18th International Conference on Multimedia Modeling
(MMM) 2012, Klagenfurt, Austria, pp. 639-641]
Winner of VBS 2012

Winner 2014 and 2015
(2014: single video and collection search, 2015: collection only)
95
Color Sketch
(Signature)
Player
2nd Color Sketch
(optional)
[ Lokoč, J., Blažek, A., & Skopal, T. (2014, January). Signature-Based Video Browser. In MultiMedia Modeling (pp. 415-418). Springer International Publishing. ]

Two examples (of the 9tools, collection search only)
96
Moumtzidou, A., Avgerinakis, K., Apostolidis, E., Markatopoulou, F., Apostolidis, K., Mironidis, T., ... &
Patras, I. (2015, January). VERGE: A Multimodal Interactive Video Search Engine. In MultiMedia Modeling
(pp. 249-254). Springer International Publishing.
• Shot and scene detection
• HLF (Concepts) with
SIFT/SURF and VLAD
• Similarity search
• Uniform sampled frames
• Human computation
Hürst, W., van de Werken, R., & Hoet, M. (2015, January). A Storyboard-Based
Interface for Mobile Video Browsing. In MultiMedia Modeling (pp. 261-265).
Springer International Publishing.
3rd place

URL: http://mklab-services.iti.gr/vss2015/ [Courtesy of Stefanos Vrochidis]

[Courtesy of Stefanos Vrochidis]

Similarity Search Results

Human vs. Machine
• Utrecht University @ VBS 2015
 Wolfgang Huerst et al., The Netherlands
 Strong experience in HCI
• Features
 Uniformly sampled thumbs
(1 second distance)
 Huge storyboard on tablet
 Vertical scrolling, paging
101
625 thumbnails in one screen
[Hürst, W., van de Werken, R., & Hoet, M. (2015, January). A Storyboard-Based Interface for Mobile Video Browsing. In MultiMedia Modeling (pp. 261-265). Springer International Publishing.]

Visual Lifelogging
Slides by Frank Hopfgartner

What is The Quantified Self?
The Quantified Self is about obtaining self-knowledge
through self-tracking.

What is The Quantified Self?
Self-tracking is also referred to as lifelogging, self-
analysis, or self-hacking.

Memex
Bush, Vannevar. "As We May Think." The Atlantic Monthly. July 1945.
ImagesofMemex:http://trevor.smith.name/memex/

MyLifeBits
• Gordon Bell (Microsoft)
digitized his life:
Books written
Personal documents
Photos
Posters, paintings, photo of
things
Home movies and videos
CD collection
PC files
…
Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton 2009
http://research.microsoft.com/en-us/projects/mylifebits/

Creating Personal Lifebraries
A lifebrary consists of heterogeneous data recorded
using many different sensors.

Recording what I eat
Aizawa, Kiyoharu, Maruyama, Yutu, Li, He, and Morikawa, Chamin. “Food Balance Estimation by Using Personal Dietrary Tendencies in a Multimedia Food Log." IEEE Transactions on Multimedia, 15(8):2176-2185, 2013.
Semantic Gap
http://foodlog.jp/
http://mealsnap.com/

Recording what I see
"LifeGlogging cameras 1998 2004 2006 2013 labeled" by Glogger - Own work. Licensed under CC BY-SA 3.0 via Commons -
https://commons.wikimedia.org/wiki/File:LifeGlogging_cameras_1998_2004_2006_2013_labeled.jpg#/media/File:LifeGlogging_cameras_1998_2004_2006_2013_labeled.jpg

Visual Lifelogging

Example: Visual Lifelog of a day
5,500 pictures a day
[Slide: C. Gurrin, DCU]

Big Data
Cathal Gurrin, Alan F. Smeaton and Aiden R. Doherty (2014), "LifeLogging: Personal Big Data", Foundations and Trends® in Information Retrieval: Vol. 8: No. 1, pp 1-125.

Semantic Analysis
• Context cues help us to
remember (Naaman et al.)
• Context in lifelogging data:
 Location, bluetooth, time, date,
…
 Derived Knowledge (e.g.
activities)
• Approaches:
 Combine cues from different
sources
 Perform content analysis to
identify objects, people, events…
 Annotate lifelogs in form of
narrative text
Mor Naaman, Susumu Harada, QianYing Wang, Hector Garcia-Molina, Andreas Paepcke: Context data in geo-referenced digital photo collections. ACM Multimedia 2004: 196-203

Visual Feature Extraction
 Steering wheel (72%)
 Shopping (75%)
 Inside of vehicle when not driving (airplane, taxi, car,
bus) (60%)
 Toilet/Bathroom (58%)
 Giving Presentation / Teaching (29%)
 View of Horizon (23%)
 Door (62%)
 Staircase (48%)
 Hands (68%)
 Holding a cup/glass (35%)
 Holding a mobile phone (39%)
 Eating food (41%)
 Screen (computer/laptop/tv) (78%)
 Reading paper/book (58%)
 Meeting (34%)
 Road (47%)
 Vegetation (64%)
 Office Scene (72%)
 Faces (61%)
 People (45%)
 Grass (61%)
 Sky (79%)
 Tree (63%)
Byrne, Daragh, Doherty, Aiden R., Snoek, Cees G. M., Jones, Gareth J. F., Smeaton, Alan F. “Everyday concept detection in visual lifelogs: validation, relationships and trends." Multimedia Tools and Applications, 49(1):119-144, 2010.

A day
This does not work well…
Let’s add event segmentation.

Event Segmentation & Annotation
• Segment 5,500 photos per day into a set of events
 Similar to SBD in digital video processing
 We employ visual features and output of on-device sensors
Multiple Events
Finishing work in
the lab
At the bus stop Chatting at Skylon Hotel lobby Moving to a
room
Tea time On the way
back home
Event Segmentation
Summarization

Non-supervised Event
Segmentation
2. Arriving
in the office
6. Walking in
the building 12. Leaving
the office
Na Li et al. “Random Matrix Ensembles of Time Correlation Matrices to Analyze Visual Lifelogs." In Proc. Multimedia Modeling Conference, Dublin, Ireland, pp. 400-411, 2014.
Event Segmentation based on the
extraction of low level features and
computation of semantic concepts
requires knowledge about dataset.
Alternative: Highlight “significant events”
by performing time series analysis

MyLifeBits

Virtual reality
“Bad Trip is an immersive virtual
reality installation […] that enables
people to navigate the creator's
mind using a game controller.
Since November 2011, every
moments of his life has been
documented by a video camera
mounted on glasses, producing an
expanding database of digitalized
visual memories. Using custom
virtual reality software, he created a
virtual mindscape where people
could navigate, and experience his
memories and dreams.”
Souce: http://www.kwanalan.com

Art installations
Kelly, Philip and Doherty, Aiden R. and Smeaton, Alan F. and Gurrin, Cathal and O’Connor, Noel E. “The Colour of Life: Novel Visualisations of Population Lifestyles." In Proc. ACM Multimedia, pp. 1063-1066, 2010.
Image:CourtesyofC.Gurrin

Video Summary
[Courtesy of T. Plumbaum]

NTCIR
• Workshop series focusing on research on
Information Access technologies (information
retrieval, question answering, text
summarisation, etc)
• Sponsored by Japan Society for Promotion of
Science (JSPS)
• Organised since 1997 in an 18-months cycle
• NTCIR-12: January 2015 – June 2016
NII Test Collection for IR Systems

NTCIR-12 Tasks
NTCIR-12
 Second round:
 Search-Intent Mining
 Mobile Click
 Temporal Information Access
 Spoken Query & Spoken Document Retrieval
 QA Lab for Entrance Exam
 First round:
 Medical NLP for Clinical Documents
 Personal Lifelog Access & Retrieval
 Short Text Conversation

Encourage research advances in organising
and retrieving from lifelog data.
LifeLog @ NTCIR-12

Multimodal dataset with information
needs
Created by
various
individuals over
10+ days
TESTCOLLECTION
 1,500 images, location, GSR,
heart-rate, others… per
lifelogger per day
 Accompanying output of
1,000 concepts
 Data processed pre-release
(removal of personal content;
face blurring, translation of
concepts)
 Detailed user queries and
judgments generated by the
lifelogging data gatherers

Tasks
Evaluate different methods of
retrieval and access.
T1:LIFELOGSEMANTICACCESS(LSAT)
 Models the retrieval need
from lifelogs (Known-item
Search)
 Retrieve N segments that
match information need
 Interactive or Automatic
participation
 Interactive: Time limit for fair
and comparative evaluation in
an interactive system with
users
 Automatic: Fully-automatic
retrieval system. Automated
query processing
T2:LIFELOGINSIGHT
 Models the need for
reflection over lifelog data
 Exploratory task, the aim is to:
 Encourage broad
participation
 Novel methods to
visualize and explore
lifelogs
 Same data as LSAT task
 Presented via demo/poster

Task 1: Lifelog Semantic Access
Find the
moment(s)
where I use my
coffee machine.
Find the
moment(s)
where I am in
the kitchen
Find the
moment(s)
where I am
playing with my
phone.
Find the
moment(s)
where I am
preparing
breakfast.

Task 2: Lifelog Insight Task
Provide insights
on the time I
spend taking
breakfast.
Provide insights
on the time I
spend driving to
work.
Provide insights
on the time I
spend reading a
paper.
Provide insights
on the time I
spend working
on the computer.

Further information
http://ntcir-lifelog.computing.dcu.ie/
21 Sep 2015: Release of formal run
collection and task data (topics)
15 Dec 2015: Deadline for formal run
submissions
15 Jan 2016: Formal run evaluation
results return
01 Mar 2016: Paper for the Proceedings
7-10 Jun 2016: NCTIR-12 conference

The End

Interactive Video Search - Tutorial at ACM Multimedia 2015

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Interactive Video Search - Tutorial at ACM Multimedia 2015

Ähnlich wie Interactive Video Search - Tutorial at ACM Multimedia 2015 (20)

Mehr von klschoef

Mehr von klschoef (8)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Interactive Video Search - Tutorial at ACM Multimedia 2015

Hinweis der Redaktion