The presentation of the CERTH presentation in the Social Event Detection (SED) task @ MediaEval (Pisa, 2 September 2011).

1. CERTH @ MediaEval 2011 Social
Event Detection Task
Symeon Papadopoulos, Christos Zigkolis, Yiannis
Kompatsiaris, Athena Vakali
Pisa, 1-2 September 2011

2. The problem
• Identify social events in tagged photos collections:
– Challenge 1: Soccer matches @ Barcelona, Rome
– Challenge 2: Events @ Paradiso (Amsterdam) and
Parc del Forum (Barcelona)
• Alternative formulation:
– For each photo of the collection answer the questions:
Q1. Is this photo related to a social event of the given types?
Q2. If yes, to which event is it related?
– Points to classification and clustering as methods to
address the problem.
2

3. Approach
Q1
Q2
Q1 / Q2
3

4. Photo Filtering (1)
• City classification
– If geo-tagging available (~20%), use it  simple
nearest-neighbour classifier
– If not, match against city-specific tag models:
• Created from processing independent geo-tagged
photo collections TAG MODEL SAMPLES
Amsterdam (74) Barcelona (57) London (89) Paris (51) Rome (42)
amsterdam barcelona london paris rome
netherlands catalunya uk france italy
holland catalonia united kingdom francia vaticano
nederland españa great britain versailles italia
…. …. …. …. ….
4

5. Photo Filtering (2)
• Soccer/Venue classification
– In the case of venue classification, use geo-tagging
information if available.
– Match against soccer/venue tag model:
• Parameter (cf. evaluation)
TAG MODEL SAMPLES (baseline)
Soccer (53, m1,b) Paradiso (6, m2,b) Parc del Forum (8, m2,b)
soccer paradiso parc del forum
football names of Spanish FCs concert primavera sound
+
goal names of Italian FCs festival concert
goalkeeper gig festival
… live music …
+
domain names of scheduled bands (last.fm)
knowledge 5

6. Event Partitioning
• Very simple implementation:
– Find all unique dates of photos that “passed” the
first filtering step.
– For each date, find all associated photos and split
them into groups based on the city they are
classified (same classifier as in Step 1).
– Consider the resulting groups of photos, as the set
of events.
6

7. Event Expansion
• Expand in three ways:
– Photos having the same owner as one of the
owners in the event & captured at the same date.
– Photos captured at the same location (<200m)
with the event center & at the same date (only for
geo-tagged photos)
– Photos belonging to the same cluster (by use of
method [1]) & having the same owner as one of
the owners in the event (parameter: cluster type)
[1] S. Papadopoulos, C. Zigkolis, Y. Kompatsiaris, A. Vakali. “Cluster-based
Landmark and Event Detection on Tagged Photo Collections”. In IEEE Multimedia
Magazine 18(1), pp. 52-63, 2011
7

8. Evaluation (1)
Challenge 1
Notation
Parameter 1 (p1): m1,b (baseline tag model), m1,+ (extended soccer tag model)
Parameter 2 (p2): tt (use photo title + tags), ttd (use photo description + tt)
Parameter 3 (p3): ∅ (no clustering), T (tag-based clustering), V (visual clustering)
8

9. Evaluation (2)
Challenge 2
Notation
Parameter 1 (p1): m2,b (baseline tag model), m2,+ (extended venue tag model)
Parameter 3 (p3): ∅ (no clustering), T (tag-based clustering), V (visual clustering),
H (hybrid clustering)
m2,+ was created by adding to baseline the names of the bands that played in these
venues in the same month (collected from last.fm API)
9

10. Failure examples (1)
C1 - Run1 / False positives
3559542192 3618132279 3580841609
Title: AVUÍ SOM 77.331 Title: Sant Pere Title: roma 09.
Tags: …, Campions, Trophy, Tags: Barcelone, Barcelona, Tags: rome, italy coliseum,
campnou, soccer, football, Night Ambiance, Light palatino, chuch, soccer,
caosasuna, barça, fiesta, … statues, art
Many of the photo tags Captured at the same Just one of the tags
are related to soccer and date and in the vicinity of (soccer) is related to
even to a soccer event the event. soccer.
(fiesta, champions).
Most of the false positives were due to the expansion step
(i.e. same day + close by, or same day + same user)
10

11. Failure examples (2)
C1 - Run2 / False negatives
3559542192 3571654936 3583033760
Title: near Tor di Quinto, Title: Barcelona v.
Latium, Italy Title: DSC_0029
Manchester United
Tags: N/A Tags: FC Barcelona Fiesta Tri
Tags: Sigma 10-20mm, F4-5.6
Campions
Description: s.s. lazio wins EX DC HSM, barcelona, spain,
the coppa italia moo2
Here the event The information could be Event information is
information is only inferred from title if our tag encoded in a single tag,
present in the photo model contained FC names but we don’t tokenize
description. from different countries. tags, so we miss it.
Most of the false negatives were due to failure in matching
the textual metadata of photos to the soccer tag model. 11

12. Discussion (1)
• Most important factor:
– a good tag model to be used for classification
• Marginal contribution of clustering:
– expansion by spatio-temporal metadata already captures
most related photos
– tag-based clusters tend to include many of the photos of
the same user at the same date
– visual clusters did not yield further improvements as one
would hope (at least with employed visual similarity
measure: 500 feature vector from clustering SIFT features)
12

13. Discussion (2)
• Future action: study in detail failure cases and
make necessary modifications to approach
• Ways to improve:
– better topic/entity classification methods
• better/richer tag models + text matching methods
• more sophisticated methods: e.g. SVMs, relational
learning + more discriminative features (text, visual,
social)
– more elaborate city classification methods or even
precise geo-tagging methods
13

14. Questions
14