Interactive Lifelog Retrieval Engine with CNN features, WordNet, Temporal Browsing and Query by Sample.

1. LEMoRe: A Lifelogging Engine for Moments
Retrieval at NTCIR-12 Lifelog Task
LEMoRe Team:
de Oliveira Barra, G., Cartas Ayala, A.; Bolaños, M.; Dimiccoli, M.;
Aghaei, M.; Carné, M.; Giro-i-Nieto, X.; Radeva, P
Contact:
Gabriel de Oliveira Barra, gabriel.deoliveira@ub.edu, http://www.ub.edu/cvub/

2. NTCIR-12 Challenge
- 90,586 egocentric images obtained from 3 users during 1
month.
- 89,593 tags
- 6 activity classes
- 35 clustered locations
Slide 4
15:26
Queries
• Precision (image-level) – “Find the moment(s) when I was getting a key made.“
• Recall (event-level) – “Find the moment(s) in which I was grocery shopping in the supermarket.”

3. Motivation
How to find a needle in a haystack?
All-in-one system covering the 3 main blocks of retrieval:
1. Parsing
2. Indexing
3. Retrieval
Slide 5
Requirements:
• Fast
• Scalable
• Flexible
• OS and device independent
• Automated
15:26

4. Motivation: Our Proposal
Interactive Retrieval
Slide 4
15:26
Visual featuresTemporal browsingTextual information

5. Methodology: Visual features
Slide 7
Visual descriptors:
• Hand-crafted:
• HOG – Histogram of Oriented Gradients
• CL – Color Layout
• EH – Edge Histogram
• JCD – Jaccard Composite Descriptor
• CNNs
• Layer “fc6” extracted from CaffeNet
15:26

6. Methodology: Visual indexing
Slide 7
15:26
Object detection by CNN:
-CaffeNet -> 1000 object classes
-LSDA -> 3822 additional tags
-4308 total (merged) unique tags

7. Methodology: Textual features
Slide 7
WordNet is a large lexical database of English
where nouns, verbs, adjectives and adverbs are
grouped into sets of cognitive synonyms, each
expressing a distinct concept.
15:26
It suggests semantically similar tags from
within the 4308 unique tags available in the
corpus.
i.e. Domestic cat
Egyptian cat
Siamese cat
Abyssinian
Persian cat
Mouser
Kitty
Tabby
Tiger cat
Manx

8. LEMoRe: the query
15:26
“Find the moments when I’m drinking coffee in front of my laptop”

9. LEMoRe: the query
15:26
“Find the moments when I’m drinking coffee in front of my laptop”

10. LEMoRe: the query
15:26
“Find the moments when I’m drinking coffee in front of my laptop”

11. LEMoRe: textual retrieval
15:26
“Find the moments when I’m drinking coffee in front of my laptop”
Visual indexing by a CNN

12. LeMoRE: Temporal Browsing
15:26
“Find the moments when I’m drinking coffee in front of my laptop”

13. LeMoRE: Visual Search
15:26
“Find the moments when I’m drinking coffee in front of my laptop”

14. LeMoRE: Visual Search
15:26
“Find the moments when I’m drinking coffee in front of my laptop”

15. LEMoRe: the image retrieval
15:26
“Find the moments when I’m drinking coffee in front of my laptop”

16. Validation
• Users (12) – 3 runs:
• Run1 (experts wo wordnet),
• Run2 (experts with wordnet),
• Run3 (beginners with wordnet).
• Measures:
• mAP
• rPresicion
• Binary preference, etc.
15:26

17. Results (I)
Slide 14
50 100 150 200 250 300
Seconds
0.0
0.1
0.2
0.3
0.4
0.5
MeanNDCG
Run 1
Run 2
Run 3
0.0 0.2 0.4 0.6 0.8 1.0
Recall
0.0
0.2
0.4
0.6
0.8
1.0
InterpolatedPrecision
Run 1 Sec 10
Run 1 Sec 120
Run 1 Sec 300
Run 2 Sec 10
Run 2 Sec 120
Run 2 Sec 300
Run 3 Sec 10
Run 3 Sec 120
Run 3 Sec 300
Mean average precision over time for each
run of event-level retrieval.
Event-level interpolated precision over
recall for all submitted runs on seconds
10,120 and 300.
15:26

18. Results (II)
Slide 15
Image-Level and Event-Level results for the total number of images retrieved over all event queries.
15:26

19. Conclusions
Slide 16
• Egocentric lifelog retrieval tool based on
• Semantic search
• Image query-by-sample search
• Visual relevance-over-time browsing.
• Better performance over event-level vs. image-level queries.
• Better semantics improves event retrieval results (difference between first and second runs).
• Improved semantic and tags (activity, places, objects) recognition.
• Scalability.
• Privacy issues.
Future Work
15:26

20. Foto del grupo
15:26

21. Live Demo
Slide 13
LEMoRe Live demo
15:26