This document describes the CERTH team's participation in the MediaEval 2014 Synchronization of Multi-User Event Media task. The team proposed a method using near duplicate image detection, graph construction and traversal to synchronize the timestamps of images across different users. They then clustered the images into events using either a single clustering approach across all images or pre-clustering within each user's gallery. The team submitted 5 runs and evaluated the results, finding their modified near duplicate detection worked best for timestamp alignment on one dataset while the standard method was better for another. Color histogram similarity also helped improve clustering performance on one dataset compared to concept detection scores.

1. 1Information Technologies Institute
Centre for Research and Technology Hellas
CERTH at MediaEval 2014
Synchronization of Multi-User
Event Media Task
Konstantinos Apostolidis,
Christina Papagiannopoulou,
Vasileios Mezaris
Information Technologies Institute / Centre for Research and Technology Hellas

2. 2Information Technologies Institute
Centre for Research and Technology Hellas
• Media from different users related to an event are
exchanged/shared in a number of ways.
– time information attached to some of the captured media may be
wrong
– geolocation information may be missing
 Objective: The alignment and presentation of media of
different users in a consistent way, preserving the temporal
evolution of the event.
Introduction

3. 3Information Technologies Institute
Centre for Research and Technology Hellas
Proposed method

4. 4Information Technologies Institute
Centre for Research and Technology Hellas
Time Synchronization (1/4)
• Near Duplicate image detection
– SIFT descriptors extraction.
– Clustering to form a vocabulary.
– Encoding of each image using VLAD encoding.
– Keypoints geometry consistency checking using Geometric Coding to
refine the returned near duplicate images.
• Modified Near Duplicate image detection
– Use color information (HSV histograms). Near duplicate candidates
that are very similar in color are considered even if the Geometric
Coding score is relatively low.

5. 5Information Technologies Institute
Centre for Research and Technology Hellas
Time Synchronization (2/4)
• Apply the modified NDD on the union of all galleries to come
up with pairs of near duplicate images belonging to different
images.
• Filter results, rejecting pairs of which:
– the location distance of the two photos is greater than a distance threshold.
– the time difference between the photos is above a time threshold.
• The remaining of near duplicate photos belonging to different
galleries are considered as links between those galleries.

6. 6Information Technologies Institute
Centre for Research and Technology Hellas
Time Synchronization (3/4)
• Graph Construction
– Galleries → Nodes
– Links between galleries → Edges
– Number of links between galleries → Edge weights

7. 7Information Technologies Institute
Centre for Research and Technology Hellas
Time Synchronization (4/4)
• Graph traversal:
1. Start from the node corresponding to the reference gallery.
2. Select the edge with the highest weight.
3. Compute the time offset of the node on the other end.
 Time offset is equal to the median of time differences of near duplicate
image pairs from the corresponding galleries.
4. Repeat steps 2 and 3 considering as possible starting point any
visited node.
• Once the galleries time offsets are calculated, align the image
timestamps.

8. 8Information Technologies Institute
Centre for Research and Technology Hellas
Media Clustering to Events
• Cluster events using the corrected timestamps.
• Two approaches are followed:
1. Consider all photo galleries as a single photo collection
and perform clustering in this collection.
2. Make pre-clustering within each gallery separately and
then merge the clusters across the galleries.

9. 9Information Technologies Institute
Centre for Research and Technology Hellas
1st
Clustering Approach (1/3)
• Sort all the photos according to the corrected timestamps
• Construct the similarity matrix using the similarity measure:
where ti, tj the timestamps of the images i,j and K is a parameter
for the sensitivity of the measure.
• The larger the value of K the coarser the clusters created.
K = 30 min K = 12 h

10. 10Information Technologies Institute
Centre for Research and Technology Hellas
• Compute the novelty scores for each photo i, vK, to detect the
cluster boundaries
– Correlate a checkerboard kernel g
along the main diagonal of each SK. Kernel g
– , where l = 1
• Detect the peaks that have difference greater than a threshold.
K = 30 min K = 12 h
Repeat the above procedure for K = 15, 30, 60, 120, 720, 1440 min
1st
Clustering Approach (2/3)

11. 11Information Technologies Institute
Centre for Research and Technology Hellas
1st
Clustering Approach (3/3)
• Select the boundary list, BK = {b1, … , bnK}, which forms the larger within-
cluster similarity and the lower between-cluster similarity, C(BK), using :
• The time alignment may not be perfect.
• Split each of the clusters using the geolocation information.
– Photos with close geolocation are assigned to the same cluster
– Photos which have great geolocation difference are assigned to
different clusters
• The photos that do not have geolocation information are assigned to the
geo-cluster which is more similar according to the color information (e.g.
HSV histogram).

12. 12Information Technologies Institute
Centre for Research and Technology Hellas
2nd
Clustering Approach (1/2)
• In each gallery, find the minimum time difference of dissimilar
photos which is greater than the maximum time difference of
the near-duplicate photos (based on the similarity matrix of
GC).

13. 13Information Technologies Institute
Centre for Research and Technology Hellas
2nd
Clustering Approach (2/2)
• The clusters that are formed by these time gaps, are merged
according to time and geolocation similarity.
• For the clusters that do not have geolocation information, the
merging is continued by considering:
– the time and low-level feature similarity
– the time and the concept detector confidence similarity scores

14. 14Information Technologies Institute
Centre for Research and Technology Hellas
Submitted runs
• Run #1: Compute gallery time offsets using our modified NDD. Concept
Detection scores are used to merge clusters using the 2nd
approach in
clustering.
• Run #2: Compute gallery time offsets using our modified NDD. HSV
histogram similarity is used to merge clusters using 2nd
clustering
approach.
• Run #3: Compute gallery time offsets using our modified NDD. Clustering
is performed using the 1st
clustering approach.
• Run #4: Compute gallery time offsets using our modified NDD and
traverse of the graph by considering the weights of multiple edges.
Concept Detection scores are used to merge clusters using the 2nd
clustering approach.
• Run #5: Compute gallery time offsets using NDD without HSV color
information. Concept Detection scores are used to merge certain events
using the 2nd
clustering approach.

15. 15Information Technologies Institute
Centre for Research and Technology Hellas
Evaluation (1/2)
• Time synchronization:
– Modified NDD approach gives the best results in time
alignment for the Vancouver testset.
– Standard NDD yields a slightly better time synchronization
than modified NDD, for the London testset.
Run #1 Run #2 Run #3 Run #4 Run #5
Vancouver
testset
Time Sync. Precision 0.9118 0.9118 0.9118 0.5294 0.9118
Time Sync. Accuracy 0.7375 0.7375 0.7375 0.7014 0.7279
Run #1 Run #2 Run #3 Run #4 Run #5
London
testset
Time Sync. Precision 0.6111 0.6111 0.6111 0.2222 0.6389
Time Sync. Accuracy 0.7127 0.7127 0.7127 0.6996 0.7299

16. 16Information Technologies Institute
Centre for Research and Technology Hellas
Evaluation (2/2)
• Sub-event Clustering:
– Exploitation of consistent timestamps in a gallery and the use of
Concept Detection scores gives a good performance for the
Vancouver testset.
– HSV histogram similarity clearly gives the best clustering results for
the London testset.
Run #1 Run #2 Run #3 Run #4 Run #5
Vancouver
testset
Clustering Rand Index 0.9770 0.9734 0.9526 0.9601 0.9656
Clustering Jaccard Index 0.2581 0.2315 0.2856 0.1782 0.2861
Clustering F-Measure 0.2052 0.1880 0.2222 0.1512 0.2225
Run #1 Run #2 Run #3 Run #4 Run #5
London
testset
Clustering Rand Index 0.9885 0.9910 0.9838 0.9829 0.9863
Clustering Jaccard Index 0.5051 0.5614 0.3232 0.2739 0.4849
Clustering F-Measure 0.3356 0.3596 0.2443 0.2150 0.3266

17. 17Information Technologies Institute
Centre for Research and Technology Hellas
Conclusions
• Using color information in modified NDD, we can discover
more pairs of images between different galleries.
• However, due to multiple events being carried out at the
same location with characteristic color background in London
testset, modified NDD returned some wrong pairs, yielding to
worse results than the standard method.
• In sub-event clustering, the use of HSV histograms similarity
seems to give a better performance than the Concept
Detector confidence scores, in a dataset with increased color
diversity, such as London testset.

18. 18Information Technologies Institute
Centre for Research and Technology Hellas
Questions?
More information:
http://www.iti.gr/~bmezaris
bmezaris@iti.gr