Interactive
Video Search
Klaus Schoeffmann, PhD
Klagenfurt University
Institute of Information Technology
Klagenfurt, Aust...
Outline
• Search in video content: motivation and challenges
• Video retrieval and its challenges
• What is interactive vi...
Motivation
3Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
Videos Everywhere
• Ubiquitous use of videos nowadays
Entertainment and commercials
Social gaming (screencasts)
Persona...
Video as the Ultimate Media?
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 5...
Video Cameras
• Increasingly powerful
These days you can record 4K content with your mobile!
Video sensors use auto-focu...
7Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
[Mary Meeker, Liang Wu, Inter...
Challenges
• Video data
are a continuous media: the content depends on time!
often contain several media types: image, t...
Challenges
• Subjectivity of data
How many people use it?
The more the easier!?
• Different levels
 Internet scale (You...
“Poor Man’s Video Search Tool”
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search...
Video Data Still Tedious to Use
• Even with video retrieval tools it is still challenging
to find desired video content
E...
Traditional
Video Retrieval
12Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
...
Search Example (TRECVID KIS 2010)
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Sea...
Video Clip Hidden in Huge Collection
14
Internet Archive
with Creative
Commons (IACC)
data set, as used
for TRECVID:
146,7...
1st Trial at YouTube
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 15
2nd Trial at YouTube
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 16
[ Heesch, D., Howarth, P., Magalhaes, J., May, A., Pickering, M., Yavlinsky, A., & Rüger, S. (2004, November). Video retri...
More Interactive Retrieval Tool
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Searc...
Challenges for
Video Retrieval
19Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Sear...
Traditional Video Retrieval Approach
“Query-and-Browse-Resoluts” Paradigm
Works well if (and only if)
users can properly ...
Challenges
Content-based features
 How to understand semantics from pixels? Semantic Gap
Both images show
bears in front...
Database affinity of concept classifiers
Low performance in broad domain
P(k) Precision at level k (after k results)
rel...
 Query-by-concept
 Which concept to use? Choose from a long list of results…
 Query-by-example
 Typically no perfect e...
Example: Query by Motion Sketch
• Matching based on trajectory descriptor
• Challenge: may differ a lot among different us...
Well-Known Issues
of “Query-and-Browse-Results” Paradigm
Users cannot formulate or have no query
  provide exploratory ...
Uniform Sampled Frames
from a Video with High Self-Similarity
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tut...
Needs Special Keyframe Extraction
and Object Detection
[Klaus Schoeffmann, Manfred Del Fabro, Tibor Szkaliczki, Laszlo Bös...
And Special Browsing Tools /
Visualization
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive ...
Where Is the User in Multimedia Retrieval?
29
[Marcel Worring et al., „Where Is the User in Multimedia Retrieval?“, IEEE M...
Interactive vs. “Traditional” Retrieval
30Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive V...
Interactive Video Search
And how it can help…
31Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interac...
Performance
Gap
Interactive Video Search
32
• Mostly interactive search
• Human computation
• Simple-to-use
• Inflexible a...
Interactive Video Search
Traditional video retrieval + interactive inspection/exploration/navigation
and rich content visu...
Interactive Video Search
• Tries to strongly integrate user into search process
Search – Inspect – Think – Repeat
 Explo...
User Studies with Significance Tests!
• Many interfaces proposed without proper evaluation
• Interface A better than inter...
IVS Tools:
Video Browsing
36Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
37
Video Browsing
[ F. Arman, R. Depommier, A. Hsu, and M-Y. Chiu, Content-based Browsing of Video Sequences, in Proc. of ...
The ThumbBrowser
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 38
[Marco Hud...
Video Browser for the Digital Native
39
[Adams, Brett, Stewart Greenhill, and Svetha Venkatesh. "Towards a video browser f...
Video Browser for the Digital Native
• User study with 8 participants
 Test configuration elements by two tasks
(after pr...
Thread-Based Browsing of Retrieval Results
• Thread: linked seq. of shots in a specified order
 Query results, visual sim...
Thread-Based Browsing of Retrieval Results
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive ...
IVS Tools:
Video Navigation
43Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
Improving Navigation
44
e.g., on YouTube
default window:
640 pixels = frames
(25 seconds)
Common seeker-bar limits
navigat...
Navigation with a Seeker-Bar
Idea of the ZoomSlider
45
Wolfgang Hürst, Georg Götz, and Martina Welte, “Interactive video b...
46Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
Improving Navigation
47
e.g., on YouTube
default window:
640 pixels = frames
(25 seconds)
Common seeker-bar limits
navigat...
Relative Flow Dragging
Background Stabilization
48
Pierre Dragicevic, Gonzalo Ramos, Jacobo Bibliowitcz, Derek Nowrouzezah...
Relative Flow Dragging
• Evaluation with a user study
 16 participants (18-44 years old)
 Direct comparison to seeker-ba...
How Do Users Search in
Video With a Common
Video Player?
50Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutori...
How do Users Search with Video Players
(Navigate with Seeker-Bars)?
• User study with more than 30 participants
• Known It...
How do Users Search with Video Players
(Navigate with Seeker-Bars)?
52
[Claudiu Cobarzan and Klaus Schoeffmann, “How do Us...
How do Users Search with Video Players
(Navigate with Seeker-Bars)?
53
[Claudiu Cobarzan and Klaus Schoeffmann, “How do Us...
How do Users Search with Video Players
(Navigate with Seeker-Bars)?
Timeinvideo(ms)
54
[Claudiu Cobarzan and Klaus Schoeff...
Improving
Video Navigation on
Touch Devices
55Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interacti...
56Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
Keyframe Navigation Tree
• Consider findings of study on navigation behavior
• Basic idea inspired by frame stripes (MO im...
• Keyframe selection based on sub-shots (JSD with color histograms)
Cover all important scenes even for long shots (e.g. ...
59Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
Keyframe Navigation Tree
• Video Player vs. KNT Browser (iPad, 4th generation, 9.7-inch)
• User study with 20 participants...
Keyframe Navigation Tree
Search Time & Performance
• Task solve time, erroneous trials, unanswered trials…
61
28.11
44.18
...
Keyframe Navigation Tree
Subjective Rating
• NASA Task-Load-Index (TLX) questionnaires
62
KNT browser significantly
better...
IVS Tools:
Content Visualization
63Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Se...
Grid Interfaces Aren‘t Enough!
• Many video retrieval systems use a Grid interface!?
Moreover, a grid interface does not a...
Table of Video Content
(TOVC)
[Goeau et al., ICME 2007]
65
Squeeze / Fisheye
Rapid Visual Serial
Presentation (RSVP)
Impro...
66
VideoTree
[Jansen et al., CBMI 2008]
However, outperformed by
simple “grid of keyframes”
in terms of search time.
Simil...
3D Ring Interface
• Utilization of screen real estate
 Large set of images
 Minor occlusion, slight distortion
• Intuiti...
3D Ring Interface - Perspectives
Preferred Design acc. to user study
25% Vertical 66% Horizontal 8.3% Frontal
68
Klaus Sch...
3D interface significantly faster than grid by 12.7%
User Study: Grid vs. Ring (both sorted)
150 images, 12 participants, ...
Extension: Multiple Rings with Vertical Scrolling
70
Klaus Schoeffmann. 2014. The Stack-of-Rings Interface for Large-Scale...
IVS Tools:
Ad-Hoc Similarity Search
71Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video...
The Video Explorer
72
[ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool fo...
Interactive Navigation Summaries
Allows a user to quickly identify
similar/repeating scenes
73
[ Schoeffmann, K., & Boeszo...
Motion Layout: Direction + Intensity
Motion Vector (µ) classification into
K=12 equidistant motion directions
Mapping to H...
75
[ Schoeffmann, K., Lux, M., Taschwer, M., & Boeszoermenyi, L. (2009, June). Visualization of video motion in context of...
• SOI Search
 Motion-based search by example sequence
 Using Motion Direction histogram Db
 User-selected sequence
 Fi...
Region-of-Interest (ROI) Search
 User selects spatial region-of-interest
 On search
 Compute Euclidian distance of fram...
78
[ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and se...
IVS Tools:
Sketch-Based Search
79Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Sear...
• Color sketches mapped to
feature signatures
• Matched to those of
keyframes
80
1. Sampling keypoints
2. Description thro...
Feature Signature-Based Video Browser
81
Color Sketch
(Signature)
Player
Winner of Video Browser Showdown 2014 + 2015
Down...
Compact visualization
Simple color-position sketch
Negative
example
Matched key-frames
Time to
2nd sketch
2nd optional
ske...
Compact Visualization in Detail
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Searc...
Another Example of a Color-Based Browser
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Vi...
Evaluation of
IVS Tools
85Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
TRECVID
http://trecvid.nist.gov/
• International video retrieval competition evaluation
 Annually performed by NIST (Gait...
TRECVID Known-item Search
TRECVID KIS (2010-2012)
models the situation in which
“someone knows of a video, has seen it bef...
TRECVID Known-item Search
The Performance of State-of-The-Art Video Retrieval Tools
Known items not found by any team:
Int...
MediaEval 2015
• Search and Anchoring in Video Archives
“Search for Multimedia Content”
 Multi-model textual and visual ...
Video Browser Showdown (VBS)
• Annual performance evaluation competition
 Live evaluation of search performance
 Special...
Video Browser Showdown (VBS)
• Scoring through VBS Server
• Score (s) [0-100] for task i and team k is based on
Solve tim...
Video Browser Showdown 2015
• Search in mid-sized video collections (2016: 200 hours)
 Originally only single video searc...
93Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
2012: Klagenfurt
11 teams
20...
Video Browser Showdown 2012
Two examples (of the 11 tools; single video search only)
94
[Xiangyu Chen, Jin Yuan, Liqiang N...
Winner 2014 and 2015
(2014: single video and collection search, 2015: collection only)
95
Color Sketch
(Signature)
Player
...
Video Browser Showdown 2015
Two examples (of the 9tools, collection search only)
96
Moumtzidou, A., Avgerinakis, K., Apost...
97Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
URL: http://mklab-services.i...
98Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
[Courtesy of Stefanos Vrochi...
99Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
Similarity Search Results
[C...
100Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
[Courtesy of Stefanos Vroch...
Human vs. Machine
• Utrecht University @ VBS 2015
 Wolfgang Huerst et al., The Netherlands
 Strong experience in HCI
• F...
Visual Lifelogging
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 102
Slides ...
What is The Quantified Self?
The Quantified Self is about obtaining self-knowledge
through self-tracking.
Klaus Schoeffman...
What is The Quantified Self?
Self-tracking is also referred to as lifelogging, self-
analysis, or self-hacking.
Klaus Scho...
Memex
Bush, Vannevar. "As We May Think." The Atlantic Monthly. July 1945.
ImagesofMemex:http://trevor.smith.name/memex/
Kl...
MyLifeBits
• Gordon Bell (Microsoft)
digitized his life:
Books written
Personal documents
Photos
Posters, paintings, p...
Creating Personal Lifebraries
A lifebrary consists of heterogeneous data recorded
using many different sensors.
Klaus Scho...
Recording what I eat
Aizawa, Kiyoharu, Maruyama, Yutu, Li, He, and Morikawa, Chamin. “Food Balance Estimation by Using Per...
Recording what I see
"LifeGlogging cameras 1998 2004 2006 2013 labeled" by Glogger - Own work. Licensed under CC BY-SA 3.0...
Visual Lifelogging
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 110
Example: Visual Lifelog of a day
5,500 pictures a day
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: I...
Big Data
Cathal Gurrin, Alan F. Smeaton and Aiden R. Doherty (2014), "LifeLogging: Personal Big Data", Foundations and Tre...
Semantic Analysis
• Context cues help us to
remember (Naaman et al.)
• Context in lifelogging data:
 Location, bluetooth,...
Visual Feature Extraction
 Steering wheel (72%)
 Shopping (75%)
 Inside of vehicle when not driving (airplane, taxi, ca...
A day
This does not work well…
Let’s add event segmentation.
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tuto...
Event Segmentation & Annotation
• Segment 5,500 photos per day into a set of events
 Similar to SBD in digital video proc...
Non-supervised Event
Segmentation
2. Arriving
in the office
6. Walking in
the building 12. Leaving
the office
Na Li et al....
MyLifeBits
Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton...
MyLifeBits
Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton...
Virtual reality
“Bad Trip is an immersive virtual
reality installation […] that enables
people to navigate the creator's
m...
Art installations
Kelly, Philip and Doherty, Aiden R. and Smeaton, Alan F. and Gurrin, Cathal and O’Connor, Noel E. “The C...
Video Summary
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 122
[Courtesy of...
NTCIR
• Workshop series focusing on research on
Information Access technologies (information
retrieval, question answering...
NTCIR-12 Tasks
NTCIR-12
 Second round:
 Search-Intent Mining
 Mobile Click
 Temporal Information Access
 Spoken Query...
Encourage research advances in organising
and retrieving from lifelog data.
LifeLog @ NTCIR-12
Klaus Schoeffmann, Frank Ho...
Multimodal dataset with information
needs
Created by
various
individuals over
10+ days
TESTCOLLECTION
 1,500 images, loca...
Tasks
Evaluate different methods of
retrieval and access.
T1:LIFELOGSEMANTICACCESS(LSAT)
 Models the retrieval need
from ...
Task 1: Lifelog Semantic Access
Find the
moment(s)
where I use my
coffee machine.
Find the
moment(s)
where I am in
the kit...
Task 2: Lifelog Insight Task
Provide insights
on the time I
spend taking
breakfast.
Provide insights
on the time I
spend d...
Further information
http://ntcir-lifelog.computing.dcu.ie/
21 Sep 2015: Release of formal run
collection and task data (to...
The End
Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 131
Nächste SlideShare
Wird geladen in …5
×

Interactive Video Search - Tutorial at ACM Multimedia 2015

4.882 Aufrufe

Veröffentlicht am

This is the presentation given by Klaus Schoeffmann and Frank Hopfgartner at the ACM Multimedia 2015 Tutorial in Brisbane, Australia (October 26, 2015). #acmmm15

Find paper here:
http://dl.acm.org/citation.cfm?id=2807417

Veröffentlicht in: Wissenschaft
0 Kommentare
3 Gefällt mir
Statistik
Notizen
  • Als Erste(r) kommentieren

Keine Downloads
Aufrufe
Aufrufe insgesamt
4.882
Auf SlideShare
0
Aus Einbettungen
0
Anzahl an Einbettungen
2.685
Aktionen
Geteilt
0
Downloads
24
Kommentare
0
Gefällt mir
3
Einbettungen 0
Keine Einbettungen

Keine Notizen für die Folie
  • http://www.kwanalan.com/#!blank/czod
  • A video of a lifelog is created by playing back all pictures based on their creation time. An audio layer could be added as well. But as we have seen in the first part of the tutorial, that might not be very efficient. It allows, however, to give you a good overview of the day.

    This is a video of a trip that my research group a while ago.

    Klaus: You do not need to play the whole video, just the beginning should be fine to give the idea.
  • NTCIR is a Japanese version of TREC. It’s main aim is to promote research on information access technologies. It is organised in an 18 months cycle. The current cycle started in January 2015 and will end with a conference in Tokyo in June 2016.
  • Interactive Video Search - Tutorial at ACM Multimedia 2015

    1. 1. Interactive Video Search Klaus Schoeffmann, PhD Klagenfurt University Institute of Information Technology Klagenfurt, Austria Frank Hopfgartner, PhD University of Glasgow School of Humanities Glasgow, UK
    2. 2. Outline • Search in video content: motivation and challenges • Video retrieval and its challenges • What is interactive video search and how can it help? • Video browsing • Video navigation • Break • Video content visualization • Ad-hoc similarity search / video exploration • Sketch-based search in video • Evaluation of interactive video search tools • Visual lifelogging Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 2
    3. 3. Motivation 3Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    4. 4. Videos Everywhere • Ubiquitous use of videos nowadays Entertainment and commercials Social gaming (screencasts) Personal videos (family, kids, …) Sports documentation and analysis (e.g., GoPro) Product usage instructions (e.g., furniture) Surveillance (buildings, places, street, …) Lifelogging Health care and medical science (endoscopic procedures) • Enormous amount of data! Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 4
    5. 5. Video as the Ultimate Media? Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 5 [Mary Meeker, Liang Wu, Internet Trends, D11 Conference, May, 2013] As of 2014, every minute 300 hours of video are uploaded to YouTube!
    6. 6. Video Cameras • Increasingly powerful These days you can record 4K content with your mobile! Video sensors use auto-focus, object tracking, color correction, and image stabilization Storage space not a big problem  Current smartphones have 128 GB of memory  NAS devices cheaply available Network bandwidth also dramatically increased over years  Video streaming on the go is simple and common  LTE connections provide 30 Mbit/s and even much more! Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 6
    7. 7. 7Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search [Mary Meeker, Liang Wu, Internet Trends, D11 Conference, May, 2013]
    8. 8. Challenges • Video data are a continuous media: the content depends on time! often contain several media types: image, text, audio cannot be simply stored and indexed in a data base, requires own indexing and search methods! require huge amount of storage space without compression! may contain a lot of important information, which is, however, often very subjective! Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 8
    9. 9. Challenges • Subjectivity of data How many people use it? The more the easier!? • Different levels  Internet scale (YouTube)  Country/region  Company/organization/group  Individual Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 9 Available meta-data How to query for a video clip? How to efficiently retrieve results? How to effectively present content to the user?
    10. 10. “Poor Man’s Video Search Tool” Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 10 VCR in the 1970s provided a similar functionality!
    11. 11. Video Data Still Tedious to Use • Even with video retrieval tools it is still challenging to find desired video content Especially if it is not a publicly available (and popular) Many problems with querying, in particular for novice users! • The ultimate goal is to make use of and search in video as effective as for text Quickly find relevant content Compare to interactivity of a text book  Index, ToC, list of figures/tables, etc.  Change, extend, copy, bookmark, highlight, etc. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 11
    12. 12. Traditional Video Retrieval 12Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search “Query and browse results”
    13. 13. Search Example (TRECVID KIS 2010) Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 13 Find the “video of President Bush standing near sea vessels with Coast Guard members talking about his pride of the Coast Guard, immigration, and security issues”. Video from IACC public data set! TRECVID: see later!
    14. 14. Video Clip Hidden in Huge Collection 14 Internet Archive with Creative Commons (IACC) data set, as used for TRECVID: 146,788 shots (~9,000 videos) Page 1 2 3 …. 38 39 40 Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    15. 15. 1st Trial at YouTube Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 15
    16. 16. 2nd Trial at YouTube Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 16
    17. 17. [ Heesch, D., Howarth, P., Magalhaes, J., May, A., Pickering, M., Yavlinsky, A., & Rüger, S. (2004, November). Video retrieval using search and browsing. In TREC Video Retrieval Evaluation Online Proceedings. ] 17 Content- based Feature Example Image Text Ranked list of shots In IACC about 5800 pages.  Temporal Context Traditional Video Retrieval Tool Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    18. 18. More Interactive Retrieval Tool Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 18 [A. Moumtzidou et al., “VERGE: A Multimodal Interactive Video Search Engine”, Proc. of the 21st International Conference on MultiMedia Modeling (MMM 2015), Sydney, 2015] kNN Similarity search based on VLAD vectors Concept detection with SVM and five local descriptors (SIFT, SURF, ORB, ...) and PCA Hierarchical keyframe clustering  Interaction details later
    19. 19. Challenges for Video Retrieval 19Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    20. 20. Traditional Video Retrieval Approach “Query-and-Browse-Resoluts” Paradigm Works well if (and only if) users can properly express their needs. content features can sufficiently describe visual content. computer vision can accurately detect semantics. 20 Content- based Search Ranked Results Unfortunately, in practice these assumptions do not hold. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    21. 21. Challenges Content-based features  How to understand semantics from pixels? Semantic Gap Both images show bears in front of a landscape. 21Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    22. 22. Database affinity of concept classifiers Low performance in broad domain P(k) Precision at level k (after k results) rel(k) defines if kth retrieved document is relevant Performance Gap 22 Challenges TRECVID 2013 Semantic Indexing (SIN-500): median “inferred average precision” (infAP) < 0.13 In other words: 88% of the results are not correct! Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    23. 23.  Query-by-concept  Which concept to use? Choose from a long list of results…  Query-by-example  Typically no perfect example available.  Query-by-sketch  Users are no artists   Query-by-text  How to describe a desired image by text? Usability Gap 23 A picture tells a 1000 words. by marfis75 How to describe a video clip by text??? Challenges Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    24. 24. Example: Query by Motion Sketch • Matching based on trajectory descriptor • Challenge: may differ a lot among different users Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 24 [Ghosal, Koustav, and Anoop Namboodiri. "A Sketch-Based Approach To Video Retrieval Using Qualitative Features." Proceedings of the 2014 Indian Conference on Computer Vision Graphics and Image Processing. ACM, 2014.]
    25. 25. Well-Known Issues of “Query-and-Browse-Results” Paradigm Users cannot formulate or have no query   provide exploratory search features!  For example: browsing, filtering, similarity search Users expect good results (on first page!)   Use relevance feedback / active learning instead of long lists! Shots have a temporal context Videos are dynamic  Static thumbnails are not informative  Esp. true for long shots and self-similar content   skims and visual summaries (“smart playback”)   sophisticated navigation & content structure visualization Grid interfaces are not always the best choice Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 25 Usability Gap  See later
    26. 26. Uniform Sampled Frames from a Video with High Self-Similarity Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 26
    27. 27. Needs Special Keyframe Extraction and Object Detection [Klaus Schoeffmann, Manfred Del Fabro, Tibor Szkaliczki, Laszlo Böszörmenyi, and Jörg Keckstein, “Keyframe Extraction in Endoscopic Video“, in Multimedia Tools and Applications, Springer, August, 2014] [Manfred J. Primus, Klaus Schoeffmann, and Laszlo Böszörmenyi, “Instrument Classification in Laparoscopic Videos“, in Proceedings of the International Workshop on Content-Based Multimedia Indexing (CBMI 2015), Prague, Czech Republic, IEEE, 2015, pp. 1-6] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 27
    28. 28. And Special Browsing Tools / Visualization Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 28 Agglomerative clustering based on visual similarity and temporal information [Jakub Lokoc, Klaus Schoeffmann, and Manfred Del Fabro, “Dynamic Hierarchical Visualization of Keyframes in Endoscopic Video“, in Proceedings of the 21st International Conference on MultiMedia Modelling 2015 (MMM 2015), Sydney, Australia, Lecture Notes on Computer Science (LNCS), Vol. 8936, Springer International Publishing, 2015, pp. 291-294]
    29. 29. Where Is the User in Multimedia Retrieval? 29 [Marcel Worring et al., „Where Is the User in Multimedia Retrieval?“, IEEE Multimedia, Vol. 19, No. 4, Oct.-Dec. 2012, pp. 6-10 ] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    30. 30. Interactive vs. “Traditional” Retrieval 30Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    31. 31. Interactive Video Search And how it can help… 31Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    32. 32. Performance Gap Interactive Video Search 32 • Mostly interactive search • Human computation • Simple-to-use • Inflexible and tedious for archives • Low performance (?) • Mostly automatic search • Retrieval engine • Complicated to use • Flexible and easier (?) for archives • Limited performance too! Usability Gap Novices Experts  Combines HCI with CV and MIR Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    33. 33. Interactive Video Search Traditional video retrieval + interactive inspection/exploration/navigation and rich content visualization in order to satisfy an information need 33 Focuses on search and exploration in (i) single videos as well as (ii) video collections Directed Search Find a specific shot or segment in a video Find a specific video in an archive Undirected Search Searching to discover information E.g., browse through a video in order to Learn how the content looks like See if it is interesting Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search Not supported by trad. video retrieval
    34. 34. Interactive Video Search • Tries to strongly integrate user into search process Search – Inspect – Think – Repeat  Exploratory search (“will know it when I see it”)  Instead of „query-and-browse-results“ User controls search process  Inspects and interacts  Most meaningful tool for current need, e.g. • Content Browsing/Navigation • Content Visualization and Summarization • Ad-hoc Querying (e.g., by sketch, filtering, ad-hoc example) 34Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    35. 35. User Studies with Significance Tests! • Many interfaces proposed without proper evaluation • Interface A better than interface B?  comparative user study needed!  Perform search tasks in exactly the same setting (data, environment, etc.)  Logging of interaction behavior and task solve time  Questionnaire about subjective workloads  Statistical analysis with proper tests (e.g., t-test, ANOVA, Wilcoxon signed-rank, etc.) • User simulations? • Evaluation competitions  Same data set  Comparative evaluation  TRECVID, MediaEval, Video Browser Showdown (see later) Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 35
    36. 36. IVS Tools: Video Browsing 36Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    37. 37. 37 Video Browsing [ F. Arman, R. Depommier, A. Hsu, and M-Y. Chiu, Content-based Browsing of Video Sequences, in Proc. of ACM International Conference on Multimedia, 1994, pp. 97-103 ] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    38. 38. The ThumbBrowser Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 38 [Marco Hudelist, Klaus Schoeffmann, Laszlo Böszörmenyi. “Mobile Video Browsing with the ThumbBrowser”, Proc. of the International Conference on Multimedia, 2013, pp. 405-406 ]
    39. 39. Video Browser for the Digital Native 39 [Adams, Brett, Stewart Greenhill, and Svetha Venkatesh. "Towards a video browser for the digital native." Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on. IEEE, 2012.] “Temporal Semantic Compression” based on tempo function and shot popularity (insight) Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    40. 40. Video Browser for the Digital Native • User study with 8 participants  Test configuration elements by two tasks (after presentation + 5 minutes training)  (i) Browse a familiar movie to find scenes you remember  (ii) Browse an unfamiliar movie to get a feel for its story or structure  Questionnaire with Likert-scale ratings 40 [Adams, Brett, Stewart Greenhill, and Svetha Venkatesh. "Towards a video browser for the digital native." Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on. IEEE, 2012.] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    41. 41. Thread-Based Browsing of Retrieval Results • Thread: linked seq. of shots in a specified order  Query results, visual similarity, semantic similarity, textual similarity time, … Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 41 [De Rooij, Ork, Cees GM Snoek, and Marcel Worring. "Balancing thread based navigation for targeted video search." Proceedings of the 2008 international conference on Content-based image and video retrieval (CIVR). ACM, 2008.]
    42. 42. Thread-Based Browsing of Retrieval Results Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 42 [Ork de Rooij et al.]
    43. 43. IVS Tools: Video Navigation 43Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    44. 44. Improving Navigation 44 e.g., on YouTube default window: 640 pixels = frames (25 seconds) Common seeker-bar limits navigation granularity [Huerst et al., ICME 2007] ZoomSlider Improvements (selected): Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    45. 45. Navigation with a Seeker-Bar Idea of the ZoomSlider 45 Wolfgang Hürst, Georg Götz, and Martina Welte, “Interactive video browsing on mobile devices”, in Proceedings of the 15th International Conference on Multimedia (MULTIMEDIA '07). ACM, pp. 247-256, 2007 Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    46. 46. 46Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    47. 47. Improving Navigation 47 e.g., on YouTube default window: 640 pixels = frames (25 seconds) Common seeker-bar limits navigation granularity [Dragicevic et al., CHI 2008] Direct Manipulation [Huerst et al., ICME 2007] ZoomSlider Improvements (selected): Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    48. 48. Relative Flow Dragging Background Stabilization 48 Pierre Dragicevic, Gonzalo Ramos, Jacobo Bibliowitcz, Derek Nowrouzezahrai, Ravin Balakrishnan, and Karan Singh. “Video browsing by direct manipulation”, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, pp. 237-246, 2008 Video browsing by direct manipulation / relative flow dragging Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    49. 49. Relative Flow Dragging • Evaluation with a user study  16 participants (18-44 years old)  Direct comparison to seeker-bar navigation  Navigation tasks, 2 videos (ladybug, cars)  “Find the position where the ladybug passes over marker X”  “Find the moment when car X starts moving”  Flow dragging significantly faster (RM-ANOVA) by at least 250% (also significantly less errors) 49 Pierre Dragicevic, Gonzalo Ramos, Jacobo Bibliowitcz, Derek Nowrouzezahrai, Ravin Balakrishnan, and Karan Singh. “Video browsing by direct manipulation”, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, pp. 237-246, 2008 Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    50. 50. How Do Users Search in Video With a Common Video Player? 50Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    51. 51. How do Users Search with Video Players (Navigate with Seeker-Bars)? • User study with more than 30 participants • Known Item Search Tasks 51 [Claudiu Cobarzan and Klaus Schoeffmann, “How do Users Search with Basic HTML5 Video Players?“, in Proceedings of The 20th International Conference on MultiMedia Modeling (MMM2014), 2014] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    52. 52. How do Users Search with Video Players (Navigate with Seeker-Bars)? 52 [Claudiu Cobarzan and Klaus Schoeffmann, “How do Users Search with Basic HTML5 Video Players?“, in Proceedings of The 20th International Conference on MultiMedia Modeling (MMM2014), 2014] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    53. 53. How do Users Search with Video Players (Navigate with Seeker-Bars)? 53 [Claudiu Cobarzan and Klaus Schoeffmann, “How do Users Search with Basic HTML5 Video Players?“, in Proceedings of The 20th International Conference on MultiMedia Modeling (MMM2014), 2014] Vast amount of content was checked with normal playback!? Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    54. 54. How do Users Search with Video Players (Navigate with Seeker-Bars)? Timeinvideo(ms) 54 [Claudiu Cobarzan and Klaus Schoeffmann, “How do Users Search with Basic HTML5 Video Players?“, in Proceedings of The 20th International Conference on MultiMedia Modeling (MMM2014), 2014] User start with rough navigation and look more carefully after narrowing down the search area! Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search Target segment
    55. 55. Improving Video Navigation on Touch Devices 55Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search Session continues at 3:20 pm
    56. 56. 56Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    57. 57. Keyframe Navigation Tree • Consider findings of study on navigation behavior • Basic idea inspired by frame stripes (MO images) • Goal  very compact visualization  not as fine as frames but not as coarse as keyframes of shots  provide different granularity levels for navigation  previous work has shown that users typically navigate in a coarse-to-fine grained manner 57 [ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single video based on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247-258). ACM. ] [Mueller-Seelich, H., Tan, E.: Visualizing the semantic structure of film and video (2000) ] compact overview but abstract & very high level of detail (frame-based!) [Xiaoxiao Luo, Qing Xu, Mateu Sbert, Klaus Schoeffmann, “F-Divergences Driven Video Key Frame Extraction“, in Proc. of the IEEE Int. Conference on Multimedia & Expo (ICME 2014), Chengdu, China, 2014, pp. 6] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    58. 58. • Keyframe selection based on sub-shots (JSD with color histograms) Cover all important scenes even for long shots (e.g. pans) Excerpts with three levels of detail: L1: narrow, L2: wide, L3: full keyframe Used as seeker-bar with synchronized interaction for all levels Simple touch-based interaction (tap and wipe gestures) 58 L1 (30 shots) L2 (~ 12 shots) L3 (~ 3 shots) Keyframe Navigation Tree [Hudelist, Marco A., Klaus Schoeffmann, and Qing Xu. "Improving interactive known-item search in video with the keyframe navigation tree." MultiMedia Modeling. Springer International Publishing, 2015.] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    59. 59. 59Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    60. 60. Keyframe Navigation Tree • Video Player vs. KNT Browser (iPad, 4th generation, 9.7-inch) • User study with 20 participants (15m/5f)  Age: 18-40 (mean 28.15, s.d. 6.08) • Known-item search tasks  Given 20 seconds long target clip  Find correct clip in 1-h long video as fast as possible  200 search tasks in total  Each participant performed random selection of 10 tasks (5/5)  latin-square principle to avoid familiarization effects  Time-out after 3 minutes (“unanswered”)  Wrong results marked as “erroneous” • Time measurement, logging, questionnaire (Likert-scale ratings) 60Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    61. 61. Keyframe Navigation Tree Search Time & Performance • Task solve time, erroneous trials, unanswered trials… 61 28.11 44.18 KNT Browser statistically significantly faster acc. to dependent paired-samples t-test (t(19) = -3.937 p < 0.005) 7 trials 10 trials 5 trials 22 trials Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search [Hudelist, Marco A., Klaus Schoeffmann, and Qing Xu. "Improving interactive known-item search in video with the keyframe navigation tree." MultiMedia Modeling. Springer International Publishing, 2015.]
    62. 62. Keyframe Navigation Tree Subjective Rating • NASA Task-Load-Index (TLX) questionnaires 62 KNT browser significantly better in all 7 categories! Acc. to Wilcoxon signed- rank tests (for details see paper) KNT browser is preferred search tool for 85% of tested users (17/20) Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search [Hudelist, Marco A., Klaus Schoeffmann, and Qing Xu. "Improving interactive known-item search in video with the keyframe navigation tree." MultiMedia Modeling. Springer International Publishing, 2015.]
    63. 63. IVS Tools: Content Visualization 63Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    64. 64. Grid Interfaces Aren‘t Enough! • Many video retrieval systems use a Grid interface!? Moreover, a grid interface does not allow for fast human visual search (see later)! 64 A ranked list of results does not convey the temporal content structure! • To which video does a shot belong to? • What is the sequence of shots? • How long is a shot / scene? Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    65. 65. Table of Video Content (TOVC) [Goeau et al., ICME 2007] 65 Squeeze / Fisheye Rapid Visual Serial Presentation (RSVP) Improving Visualization aka “Video Surrogates” [Wildemuth et al., 2003] [Wittenburg et al., 2005] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    66. 66. 66 VideoTree [Jansen et al., CBMI 2008] However, outperformed by simple “grid of keyframes” in terms of search time. Similar concept proposed later [Girgensohn et al., ICMR 2011] • Split-based clustering algorithm with color correlograms. • Tree not directly shown to the user (only one level). Improving Visualization aka “Video Surrogates” Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    67. 67. 3D Ring Interface • Utilization of screen real estate  Large set of images  Minor occlusion, slight distortion • Intuitive interaction  Rotate and zoom • Content-based sorting • “Pop-out images” (in the back) • Further advantages  Immediately continue on miss, scaling 67 Klaus Schoeffmann, David Ahlström, and Marco Andrea Hudelist, “3-D Interfaces to Improve the Performance of Visual Known-Item Search“, in IEEE Transactions on Multimedia, Vol. 16, No. 7, November, 2014, pp. 1942-1951. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    68. 68. 3D Ring Interface - Perspectives Preferred Design acc. to user study 25% Vertical 66% Horizontal 8.3% Frontal 68 Klaus Schoeffmann, David Ahlström, and Marco Andrea Hudelist, “3-D Interfaces to Improve the Performance of Visual Known-Item Search“, in IEEE Transactions on Multimedia, Vol. 16, No. 7, November, 2014, pp. 1942-1951. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    69. 69. 3D interface significantly faster than grid by 12.7% User Study: Grid vs. Ring (both sorted) 150 images, 12 participants, 1440 trials 69 Klaus Schoeffmann, David Ahlström, and Marco Andrea Hudelist, “3-D Interfaces to Improve the Performance of Visual Known-Item Search“, in IEEE Transactions on Multimedia, Vol. 16, No. 7, November, 2014, pp. 1942-1951. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    70. 70. Extension: Multiple Rings with Vertical Scrolling 70 Klaus Schoeffmann. 2014. The Stack-of-Rings Interface for Large-Scale Image Browsing on Mobile Touch Devices. In Proc. of the ACM Int. Conference on Multimedia (MM '14). ACM, New York, NY, USA, 1097-1100. Significantly faster search (by about 48%) than common image browser on iPad! Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    71. 71. IVS Tools: Ad-Hoc Similarity Search 71Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    72. 72. The Video Explorer 72 [ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single video based on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247-258). ACM. ] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    73. 73. Interactive Navigation Summaries Allows a user to quickly identify similar/repeating scenes 73 [ Schoeffmann, K., & Boeszoermenyi, L. (2009, June). Video browsing using interactive navigation summaries. In Content-Based Multimedia Indexing, 2009. CBMI'09. Seventh Int.Workshop on (pp. 243-248). IEEE. ] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    74. 74. Motion Layout: Direction + Intensity Motion Vector (µ) classification into K=12 equidistant motion directions Mapping to Hue channel 74 [ Schoeffmann, K., Lux, M., Taschwer, M., & Boeszoermenyi, L. (2009, June). Visualization of video motion in context of video browsing. In Multimedia and Expo, 2009. ICME 2009. IEEE Int. Conf. on (pp. 658-661). IEEE. ] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    75. 75. 75 [ Schoeffmann, K., Lux, M., Taschwer, M., & Boeszoermenyi, L. (2009, June). Visualization of video motion in context of video browsing. In Multimedia and Expo, 2009. ICME 2009. IEEE Int. Conf. on (pp. 658-661). IEEE. ] Similarity Search (SOI) with Motion Layout Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    76. 76. • SOI Search  Motion-based search by example sequence  Using Motion Direction histogram Db  User-selected sequence  Find most similar sequences  Compute distance to any possible seq. of same length  Match if below spec. threshold Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 76 Motion Layout (Db) Match 1 Match 2 Match 3 frame 1 frame n Similarity Search (SOI) with Motion Layout
    77. 77. Region-of-Interest (ROI) Search  User selects spatial region-of-interest  On search  Compute Euclidian distance of frame F to every other frame f (acc. to selected region)  Based on color layout descriptor … frame F frame 1 frame k frame n User-selected region (I) … d(F,1)=350 d(F,k)=8 d(F,n)=400 77 [ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single video based on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247-258). ACM. ] Similarity Search (ROI) with Color Layout Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    78. 78. 78 [ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single video based on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247-258). ACM. ] Similarity Search (ROI) with Color Layout Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    79. 79. IVS Tools: Sketch-Based Search 79Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    80. 80. • Color sketches mapped to feature signatures • Matched to those of keyframes 80 1. Sampling keypoints 2. Description through location (x,y), CIE Lab, contrast and entropy of surrounding pixels 3. K-means clustering Feature Signatures [ Kruliš, M., Lokoč, J. and Skopal, T. (2013). Efficient Extraction of Feature Signatures Using Multi-GPU Architecture. Springer Berlin Heidelberg, LNCS 7733, pp.446-456. ] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    81. 81. Feature Signature-Based Video Browser 81 Color Sketch (Signature) Player Winner of Video Browser Showdown 2014 + 2015 Download demo at: http://siret.ms.mff.cuni.cz/lokoc/vbs.zip 2nd Color Sketch (optional) [ Lokoč, J., Blažek, A., & Skopal, T. (2014, January). Signature-Based Video Browser. In MultiMedia Modeling (pp. 415-418). Springer International Publishing. ] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    82. 82. Compact visualization Simple color-position sketch Negative example Matched key-frames Time to 2nd sketch 2nd optional sketch Interactive-navigation summaryOn demand neighborhood expansion [Slide: Adam Blazek et al. (siret research group, Czech Republic)] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 82
    83. 83. Compact Visualization in Detail Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 83 [Courtesy of Jakub Lokoc et al.]
    84. 84. Another Example of a Color-Based Browser Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 84 Browsing Area Map Segment Inspection Color SketchCategories [Kai Uwe Barthel, Nico Hezel, Radek Mackowiak. Navigating a graph of scenes for exploring large video collections, in Proc. of 22nd International Conference on MultiMedia Modeling (MMM 2016), Lecture Notes in Computer Science (LNCS), Vol. tbd, Springer International Publishing, 2016, pp. 1-7]
    85. 85. Evaluation of IVS Tools 85Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    86. 86. TRECVID http://trecvid.nist.gov/ • International video retrieval competition evaluation  Annually performed by NIST (Gaithersburg, Maryland, USA)  Funded by NIST and other US government agencies  Benchmark for researchers using same data  Origin in TREC (Text REtrieval Conference, since 1992) • Founded in 2003, by  Alan Smeaton (Dublin City University)  Wessel Kraaij (TNO-ICT, Delft) • International advisory Committee  Alex Hauptmann (CMU)  Michael Lew (Leiden Institute of Advanced Computer Science)  Georges Quenot (LIG, Grenoble)  John Smith (IBM Research)  … • Local organisation  Paul Over (NIST) 86Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    87. 87. TRECVID Known-item Search TRECVID KIS (2010-2012) models the situation in which “someone knows of a video, has seen it before, believes it is contained in a collection, but doesn‘t know where to look” Automatic Search  Text-description about the video  Return ranked list of 100 videos (out of 9000) Interactive Search  Pre-processing based on text query  Searcher browses through result list (e.g., keyframes of shots) • Interactively find target video as fast as possible • Within 5 minutes 87Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    88. 88. TRECVID Known-item Search The Performance of State-of-The-Art Video Retrieval Tools Known items not found by any team: Interactive Automatic out of 2010 5 / 24 21% 69 / 300 22% 15 teams 2011 6 / 25 24% 142 / 391 36% 9 teams 2012 2 / 24 17% 108 / 361 29% 9 teams From: [Alan Smeaton, Paul Over, “Known-Item Search @ TRECVID 2012”, NIST, 2012] 88Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    89. 89. MediaEval 2015 • Search and Anchoring in Video Archives “Search for Multimedia Content”  Multi-model textual and visual descriptions of content of interest “Automatic Anchor Selection”  Predict key elements of videos as anchor points for hyperlinking Professional (BBC) and non-professional content (users) Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 89 http://www.multimediaeval.org
    90. 90. Video Browser Showdown (VBS) • Annual performance evaluation competition  Live evaluation of search performance  Special session at Int. Conference on MultiMedia Modeling (MMM)  Demonstrates and evaluates state-of-the-art interactive video search tools  Idea influenced by VideOlympics (Snoek et al., IEEE Multimedia 2008) • Focus  Known-item Search tasks  Target clips are presented on site  Teams search in shared data set  Highly interactive search  Should push research on interfaces and interaction/navigation  Experts and Novices  Easy-to-use tools and methods 90 Teams connected to a server that issues tasks and evaluates submitted results http://videobrowsershowdown.org/ Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    91. 91. Video Browser Showdown (VBS) • Scoring through VBS Server • Score (s) [0-100] for task i and team k is based on Solve time (t) Penalty (p) based on number of submissions (m) 91 Maximum solve time (Tmax) typically 3-5 minutes [Schoeffmann, K., Ahlström, D., Bailer, W., Cobârzan, C., Hopfgartner, F., McGuinness, K., ... & Weiss, W. (2013). The Video Browser Showdown: a live evaluation of interactive video search tools. International Journal of Multimedia Information Retrieval, 1-15. ] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    92. 92. Video Browser Showdown 2015 • Search in mid-sized video collections (2016: 200 hours)  Originally only single video search • Two different kind of tasks:  Visual: visual presentation of a 30s target clip  Textual: textual description of a 30s target clip • Shared video data from BBC  2015: 153 video files, about 100.000 shots (9 Mio frames) Participants: • Need to find the target clips as quickly as possible • Get points for each task (the faster the better) • But only for submission of exact location of target clip [Schoeffmann, Klaus. "A user-centric media retrieval competition: The video browser showdown 2012-2014." MultiMedia, IEEE 21.4 (2014): 8-13.] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 92
    93. 93. 93Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 2012: Klagenfurt 11 teams 2013: Huangshan 6 teams 2014: Dublin 7 teams 2015: Sydney 9 teams VBS 2016: January 5, 2016, Miami, USA (MMM 2016) http://www.videobrowsershowdown.org/
    94. 94. Video Browser Showdown 2012 Two examples (of the 11 tools; single video search only) 94 [Xiangyu Chen, Jin Yuan, Liqiang Nie, Zheng-Jun Zha, Shuicheng Yan, and Tat-Seng Chua, "TRECVID 2010 Known-item Search by NUS", in Proceedings of TRECVID 2010 workshop, NIST, Gaithersburgh, USA, 2011 Jin Yuan, Huanbo Luan, Dejun Hou, Han Zhang, Yan-Tao Zheng, Zheng-Jun Zha, and Tat-Seng Chua, "Video Browser Showdown by NUS", in Proceedings of th 18th International Conference on Multimedia Modeling (MMM) 2012, Klagenfurt, Austria, pp. 642-645] • Keyframe extraction (shots) • ASR and OCR • HLF (Concepts) • RF with Related Samples • Uniform sampled keyframes (with flexible distance) • Parallel playback + navigation [Manfred Del Fabro and Laszlo Böszörmenyi, "AAU Video Browser: Non- Sequential Hierarchical Video Browsing without Content Analysis", in Proceedings of th 18th International Conference on Multimedia Modeling (MMM) 2012, Klagenfurt, Austria, pp. 639-641] Winner of VBS 2012 Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    95. 95. Winner 2014 and 2015 (2014: single video and collection search, 2015: collection only) 95 Color Sketch (Signature) Player 2nd Color Sketch (optional) [ Lokoč, J., Blažek, A., & Skopal, T. (2014, January). Signature-Based Video Browser. In MultiMedia Modeling (pp. 415-418). Springer International Publishing. ] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    96. 96. Video Browser Showdown 2015 Two examples (of the 9tools, collection search only) 96 Moumtzidou, A., Avgerinakis, K., Apostolidis, E., Markatopoulou, F., Apostolidis, K., Mironidis, T., ... & Patras, I. (2015, January). VERGE: A Multimodal Interactive Video Search Engine. In MultiMedia Modeling (pp. 249-254). Springer International Publishing. • Shot and scene detection • HLF (Concepts) with SIFT/SURF and VLAD • Similarity search • Uniform sampled frames • Human computation Hürst, W., van de Werken, R., & Hoet, M. (2015, January). A Storyboard-Based Interface for Mobile Video Browsing. In MultiMedia Modeling (pp. 261-265). Springer International Publishing. 3rd place Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    97. 97. 97Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search URL: http://mklab-services.iti.gr/vss2015/ [Courtesy of Stefanos Vrochidis]
    98. 98. 98Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search [Courtesy of Stefanos Vrochidis]
    99. 99. 99Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search Similarity Search Results [Courtesy of Stefanos Vrochidis]
    100. 100. 100Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search [Courtesy of Stefanos Vrochidis]
    101. 101. Human vs. Machine • Utrecht University @ VBS 2015  Wolfgang Huerst et al., The Netherlands  Strong experience in HCI • Features  Uniformly sampled thumbs (1 second distance)  Huge storyboard on tablet  Vertical scrolling, paging 101 625 thumbnails in one screen [Hürst, W., van de Werken, R., & Hoet, M. (2015, January). A Storyboard-Based Interface for Mobile Video Browsing. In MultiMedia Modeling (pp. 261-265). Springer International Publishing.] Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search
    102. 102. Visual Lifelogging Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 102 Slides by Frank Hopfgartner
    103. 103. What is The Quantified Self? The Quantified Self is about obtaining self-knowledge through self-tracking. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 103
    104. 104. What is The Quantified Self? Self-tracking is also referred to as lifelogging, self- analysis, or self-hacking. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 104
    105. 105. Memex Bush, Vannevar. "As We May Think." The Atlantic Monthly. July 1945. ImagesofMemex:http://trevor.smith.name/memex/ Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 105
    106. 106. MyLifeBits • Gordon Bell (Microsoft) digitized his life: Books written Personal documents Photos Posters, paintings, photo of things Home movies and videos CD collection PC files … Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton 2009 http://research.microsoft.com/en-us/projects/mylifebits/ Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 106
    107. 107. Creating Personal Lifebraries A lifebrary consists of heterogeneous data recorded using many different sensors. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 107
    108. 108. Recording what I eat Aizawa, Kiyoharu, Maruyama, Yutu, Li, He, and Morikawa, Chamin. “Food Balance Estimation by Using Personal Dietrary Tendencies in a Multimedia Food Log." IEEE Transactions on Multimedia, 15(8):2176-2185, 2013. Semantic Gap http://foodlog.jp/ http://mealsnap.com/ Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 108
    109. 109. Recording what I see "LifeGlogging cameras 1998 2004 2006 2013 labeled" by Glogger - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:LifeGlogging_cameras_1998_2004_2006_2013_labeled.jpg#/media/File:LifeGlogging_cameras_1998_2004_2006_2013_labeled.jpg Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 109
    110. 110. Visual Lifelogging Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 110
    111. 111. Example: Visual Lifelog of a day 5,500 pictures a day Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 111 [Slide: C. Gurrin, DCU]
    112. 112. Big Data Cathal Gurrin, Alan F. Smeaton and Aiden R. Doherty (2014), "LifeLogging: Personal Big Data", Foundations and Trends® in Information Retrieval: Vol. 8: No. 1, pp 1-125. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 112
    113. 113. Semantic Analysis • Context cues help us to remember (Naaman et al.) • Context in lifelogging data:  Location, bluetooth, time, date, …  Derived Knowledge (e.g. activities) • Approaches:  Combine cues from different sources  Perform content analysis to identify objects, people, events…  Annotate lifelogs in form of narrative text Mor Naaman, Susumu Harada, QianYing Wang, Hector Garcia-Molina, Andreas Paepcke: Context data in geo-referenced digital photo collections. ACM Multimedia 2004: 196-203 Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 113 [Slide: C. Gurrin, DCU]
    114. 114. Visual Feature Extraction  Steering wheel (72%)  Shopping (75%)  Inside of vehicle when not driving (airplane, taxi, car, bus) (60%)  Toilet/Bathroom (58%)  Giving Presentation / Teaching (29%)  View of Horizon (23%)  Door (62%)  Staircase (48%)  Hands (68%)  Holding a cup/glass (35%)  Holding a mobile phone (39%)  Eating food (41%)  Screen (computer/laptop/tv) (78%)  Reading paper/book (58%)  Meeting (34%)  Road (47%)  Vegetation (64%)  Office Scene (72%)  Faces (61%)  People (45%)  Grass (61%)  Sky (79%)  Tree (63%) Byrne, Daragh, Doherty, Aiden R., Snoek, Cees G. M., Jones, Gareth J. F., Smeaton, Alan F. “Everyday concept detection in visual lifelogs: validation, relationships and trends." Multimedia Tools and Applications, 49(1):119-144, 2010. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 114
    115. 115. A day This does not work well… Let’s add event segmentation. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 115 [Slide: C. Gurrin, DCU]
    116. 116. Event Segmentation & Annotation • Segment 5,500 photos per day into a set of events  Similar to SBD in digital video processing  We employ visual features and output of on-device sensors Multiple Events Finishing work in the lab At the bus stop Chatting at Skylon Hotel lobby Moving to a room Tea time On the way back home Event Segmentation Summarization Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 116 [Slide: C. Gurrin, DCU]
    117. 117. Non-supervised Event Segmentation 2. Arriving in the office 6. Walking in the building 12. Leaving the office Na Li et al. “Random Matrix Ensembles of Time Correlation Matrices to Analyze Visual Lifelogs." In Proc. Multimedia Modeling Conference, Dublin, Ireland, pp. 400-411, 2014. Event Segmentation based on the extraction of low level features and computation of semantic concepts requires knowledge about dataset. Alternative: Highlight “significant events” by performing time series analysis Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 117
    118. 118. MyLifeBits Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton 2009 Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 118
    119. 119. MyLifeBits Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton 2009 Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 119
    120. 120. Virtual reality “Bad Trip is an immersive virtual reality installation […] that enables people to navigate the creator's mind using a game controller. Since November 2011, every moments of his life has been documented by a video camera mounted on glasses, producing an expanding database of digitalized visual memories. Using custom virtual reality software, he created a virtual mindscape where people could navigate, and experience his memories and dreams.” Souce: http://www.kwanalan.com Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 120
    121. 121. Art installations Kelly, Philip and Doherty, Aiden R. and Smeaton, Alan F. and Gurrin, Cathal and O’Connor, Noel E. “The Colour of Life: Novel Visualisations of Population Lifestyles." In Proc. ACM Multimedia, pp. 1063-1066, 2010. Image:CourtesyofC.Gurrin Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 121
    122. 122. Video Summary Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 122 [Courtesy of T. Plumbaum]
    123. 123. NTCIR • Workshop series focusing on research on Information Access technologies (information retrieval, question answering, text summarisation, etc) • Sponsored by Japan Society for Promotion of Science (JSPS) • Organised since 1997 in an 18-months cycle • NTCIR-12: January 2015 – June 2016 NII Test Collection for IR Systems Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 123
    124. 124. NTCIR-12 Tasks NTCIR-12  Second round:  Search-Intent Mining  Mobile Click  Temporal Information Access  Spoken Query & Spoken Document Retrieval  QA Lab for Entrance Exam  First round:  Medical NLP for Clinical Documents  Personal Lifelog Access & Retrieval  Short Text Conversation Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 124
    125. 125. Encourage research advances in organising and retrieving from lifelog data. LifeLog @ NTCIR-12 Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 125
    126. 126. Multimodal dataset with information needs Created by various individuals over 10+ days TESTCOLLECTION  1,500 images, location, GSR, heart-rate, others… per lifelogger per day  Accompanying output of 1,000 concepts  Data processed pre-release (removal of personal content; face blurring, translation of concepts)  Detailed user queries and judgments generated by the lifelogging data gatherers Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 126
    127. 127. Tasks Evaluate different methods of retrieval and access. T1:LIFELOGSEMANTICACCESS(LSAT)  Models the retrieval need from lifelogs (Known-item Search)  Retrieve N segments that match information need  Interactive or Automatic participation  Interactive: Time limit for fair and comparative evaluation in an interactive system with users  Automatic: Fully-automatic retrieval system. Automated query processing T2:LIFELOGINSIGHT  Models the need for reflection over lifelog data  Exploratory task, the aim is to:  Encourage broad participation  Novel methods to visualize and explore lifelogs  Same data as LSAT task  Presented via demo/poster Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 127
    128. 128. Task 1: Lifelog Semantic Access Find the moment(s) where I use my coffee machine. Find the moment(s) where I am in the kitchen Find the moment(s) where I am playing with my phone. Find the moment(s) where I am preparing breakfast. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 128
    129. 129. Task 2: Lifelog Insight Task Provide insights on the time I spend taking breakfast. Provide insights on the time I spend driving to work. Provide insights on the time I spend reading a paper. Provide insights on the time I spend working on the computer. Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 129
    130. 130. Further information http://ntcir-lifelog.computing.dcu.ie/ 21 Sep 2015: Release of formal run collection and task data (topics) 15 Dec 2015: Deadline for formal run submissions 15 Jan 2016: Formal run evaluation results return 01 Mar 2016: Paper for the Proceedings 7-10 Jun 2016: NCTIR-12 conference Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 130
    131. 131. The End Klaus Schoeffmann, Frank Hopfgartner ACM Multimedia 2015 Tutorial: Interactive Video Search 131

    ×