Multimedia content produced on a daily basis and in constantly growing quantity by professionals and individual users,
requires creation of navigation systems that allow access to
this data on different levels of granularity in order to con-
tribute to further discovery of a topic of interest for the user
or to facilitate individual user browsing within a collection.
In this paper we describe our approach to enable users
to browse through the multimedia collection. We implement the hyperlinking approach that uses the fine-grained
segmentation of the visual content based on the scene segmentation, as well as available metadata, transcripts, and
information about extracted visual concepts.
The approach was tested at the MediaEval Search and
Hyperlinking 2014 evaluation task, where it has shown its effectiveness at locating accurately relevant content in a large
media archive.
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Exploring Video Hyperlinking in Broadcast Media
1. Maria Eskevich, Quoc-Minh Bui, Hoang-An Le,
Benoit Huet
Multimedia Department
EURECOM
Sophia-Antipolis, France
Exploring Video Hyperlinking
in Broadcast Media
2. Motivation
CURRENTLY: Constantly growing quantity of
multimedia content produced by both
professionals and individual users.
NEED: navigation systems that allow access to this
data on different levels of granularity in order to
contribute to further discovery of a topic of interest
for the user or to facilitate individual user browsing
within a collection.
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 2
3. Task and Challenges
We envisage the users to be interested in creating their
own path within the multimedia collection based on
their level of awareness/knowledge of it, and tasks.
How do we create a hyperlinked collection to allow
each individual user to follow their own path of interest
within?
Why is it challenging?
Lack of interpretability of the content:
Overall textual description on the video level which leads to:
Retrieval process is based on text features, while the rich multimodality of
videos is not exploited;
Not possible to partially retrieve a video, a specific fragment without having
to watch the entire video.
Archives are not static:
Need for dynamic creation of links
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 3
4. Insights in Hyperlinking
Hyperlinking
Creating “links” between media
Video Hyperlinking
video to video
video fragment to video fragment
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 4
5. Related experiments: Search and
Hyperlinking @ MediaEval/TRECVid
Different techniques based on:
Video segmentation method:
Fixed-length units (same length, sentences, speech segments) with
further adjustment to match full sentences, using speech segment
boundaries and pauses. they build a probabilistic framework to
model the importance of words and refine segment boundaries
accordingly;
Meaningful segments, based on topics derived from transcripts:
classification trees to define the starting and ending times of these
segments; lexical cohesion within segment.
Features used for retrieval:
Text similarity (vector-based models and TF-IDF weightings);
Named entities or synonyms;
Visual information: visual concepts or SURF and SIFT features,
detected at the shot level.
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 5
6. Search and Hyperlinking @
MediaEval/TRECVid
Our solution:
Scene segmentation technique based on visual
and temporal coherence of the video segments
that constitute the scenes of the video.
Experiment with hybrid segmentation
Visual, Topic and Temporal Coherence
Use visual features to improve the ranking of
results of the hyperlinking including visual
analysis during the search process.
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 6
7. System overview
Broadcast Media: BBC content
Manually transcribed
subtitles
Metadata:
title, cast, description, broadcast time
Shot segmentation,
keyframes
Visual content analysis:
Scene segmentation
Concept detection
on shot level
Lucene/Solr:
Indexing/Retrieval on shot level
Media database:
76 214 fragments
Webservice:
(HTML5/AJAX/PHP)
Videos:
2323 items/1697 hours
User Interface
Shot segmentation,
keyframe extraction
Result list:
Media fragments level
9. Scene segmentation
Scene : group of shots based on content similarity
and temporal consistency among shots;
Content similarity : visual similarity between HSV
histograms extracted from the keyframes of
different shots;
Grouping is performed using two extensions of the
Scene Transition Graph (STG):
reduces the computational cost of STG-based shot grouping
by considering shot linking transitivity,
builds on the former to construct a probabilistic framework that
alleviates the need for manual STG parameter selection.
[Apostolidis et al. “Automatic fine-grained hyperlinking of videos within a closed
collection using scene segmentation.” ACM MM 2014]
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 9
10. Handling Visual Concepts in Solr
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 10
http://localhost:8983/solr/collection_mediaEval/select?q=text:(Children
out on poetry trip Exploration of poetry by school children Poem writing)
Animal:[0.2 TO 1] Building:[0.2 TO 1]
Schema = structure of document using fields of
different types
Query
<doc>
<field name="id"> 20080401_013000_bbcfour_legends_marty_feldman_six_degrees_of#t=399,402</field>
<field name="begin">00:06:39.644</field>
<field name="end">00:06:42.285</field>
<field name="videoId">20080401_013000_bbcfour_legends_marty_feldman_six_degrees_of</field>
<field name="subtitle">'It was very, very successful.'</field>
<field name="Actor">0.143</field>
<field name="Adult">0.239</field>
<field name="Animal">0.0572</field>
</doc>
12. Results discussion
Text (Audio) performs best on the 2014
MediaEval Hyperlinking task
Using visual features isn’t straightforward
The ‘MoreLikeThis’ approach is outperformed
by the Text only search
Possibly due to query formulation differences
Sentences vs Keywords
Visual Scenes outperform other fragmentation
levels (Topic, Sentences)
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 12
[Safadi et al. “When textual and visual information join forces
for multimedia retrieval”, ICMR 2014]
13. DEMO
Hyper Video Browser
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 13
Search/Browse Hyperlinking
14. Conclusion and Future work
We proposed and evaluated an approach to include
visual properties in the search of video segments for
hyperlinking.
Experimental results show that :
Visual properties provide meaningful cues for segmenting the video
Mapping text-based queries to visual concepts is not easy
Automatic selection of relevant concepts is required (human
intervention is impractical and does not necessarily lead to perfect
results)
Operating at the keyword level rather than full text offers
improvements
Current/Future work: incorporate query semantics
when identifying key visual semantic concepts
based on named entity recognition approaches and keyword/visual
concepts co-occurrences
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 14
15. References
E. Apostolidis, V. Mezaris, M. Sahuguet, B. Huet, B. Cervenkova, D. Stein, S. Eickeler, J.
L. Redondo Garcia,R. Troncy, and L. Pikora. “Automatic ne-grained hyperlinking of videos
within a closed collection using scene segmentation”. In ACMMM 2014, 22nd ACM
International Conference on Multimedia, Orlando, Florida, USA, 11 2014.
H. Le, Q. Bui, B. Huet, B. Cervenkova, J. Bouchner, E. Apostolidis, F. Markatopoulou, A.
Pournaras, V. Mezaris, D. Stein, S. Eickeler, and M. Stadtschnitzer,.“LinkedTV at
MediaEval 2014 Search and Hyperlinking Task”. In Proceedings of MediaEval 2014
Workshop, 2014.
R. J. F. Ordelman, M. Eskevich, R. Aly, B. Huet, and G. J. F. Jones. “Dening and
evaluating video hyperlinking for navigating multimedia archives”. In Proceedings of the
24th International Conference on World Wide Web Companion, WWW 2015, Florence,
Italy, May 18-22, 2015 - Companion Volume, pages 727-732, 2015.
M. Sahuguet, B. Huet, B. Cervenkova, E. Apostolidis, V. Mezaris, D. Stein, S. Eickeler, J.
L. Redondo Garcia, and L. Pikora. “LinkedTV at MediaEval 2013 search and hyperlinking
task”. In MediaEval 2013 Workshop, Barcelona, Spain, 10 2013.
B. Safadi, M. Sahuguet, and B. Huet. “When textual and visual information join forces for
multimedia retrieval”. In Proceedings of International Conference on Multimedia
Retrieval (ICMR '14). ACM, New York, NY, USA, , Pages 265 , 8 pages, 2014.
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 15