https://imatge.upc.edu/web/publications/visual-memorability-egocentric-cameras This project explores visual memorability of egocentric in different ways having three main contributions. The first and the main contribution of the project is a new tool visual memorability in egocentric images. This tool that consists in a web application that allows the annotation of the visual memorability associated to still images with an online game. The second contribution of this work is a convolutional neural network model for visual memorability prediction that adapts an off-the-shelf model to egocentric images. Moreover, a visualization study has been pursued to localize the regions of the images that are more memorable than others. With this maps a comparison with saliency maps and is explored. This part of the research opens a new branch in visual memorability that consists in use memorability maps for saliency prediction. Also the memorability of the images is related with a sentiment analysis applying a model that predicts that feature. The final contribution is related to join visual memorability of images with human behaviour and physical state, finding a relation between memory and some physiological signals as: heart rate, galvanic skin response and electroencephalographic signals.

1. Visual Memorability for
Egocentric Cameras
Marc Carné Herrera
Advisors: Xavier Giró-i-Nieto and Cathal Gurrin

2. Acknowledgements
2
Petia Radeva Maite Garolera
Albert Gil Josep Pujal

3. Outline
➔ Introduction
➔ Contributions
◆ Annotation tool for visual memorability
◆ EgoMemNet: visual memorability adaptation to egocentric images
◆ Visual memorability and physiological signals
➔ Conclusions
3

4. Outline
➔ Introduction
➔ Contributions
◆ Annotation tool for visual memorability
◆ EgoMemNet: visual memorability adaptation to egocentric images
◆ Visual memorability and physiological signals
➔ Conclusions
4

5. “Brain is designed to forget in order to survive”
● Lifelogger → person that captures his daily life in order to create a virtual and
digital memory.
● Wearable cameras → capture first person vision.
● Big data → 1.400 - 2.000 images/day.
● Challenge → retrieval!
5
Introduction

6. “Brain is designed to forget in order to survive”
● Lifelogger → person that captures his daily life in order to create a virtual and
digital memory.
● Wearable cameras → capture first person vision.
● Big data → 1.400 - 2.000 images/day.
● Challenge → retrieval!
6
Introduction
What we
want to
remember?

7. ● Cognitive therapy → Alzheimer patients, reminiscence therapy.
7
Motivation

8. 8
Image set
Relevant images
with low level
feature from a
CNN
Relevant images with object detection, faces detction… (based on content)

9. Visual memorability
9
[Isola, CVPR 2011]

10. Visual memorability
10
[Isola, CVPR 2011]
More memorable Less memorable

11. Domain adaptation
11
Human-taken Egocentric
[Khosla, ICCV 2015]

12. Outline
➔ Introduction
➔ Contributions
◆ Annotation tool for visual memorability
◆ EgoMemNet: visual memorability adaptation to egocentric images
◆ Visual memorability and physiological signals
➔ Conclusions
12

13. Why an annotation tool?
13
Convolutional
Neural Network
Input
(image)
Output
(label)
Image + label to train the model

14. ● Inspired by MIT research work [1]
● Visual memory game:
○ Simple task → press ‘d’ when a repeated image is found
○ Duration: 9 minutes
○ Output: text file with detections
14
[1] Understanding and Predicting Image Memorability at a Large
Scale, A. Khosla, A. S. Raju, A. Torralba and A. Oliva. ICCV 2015
Annotation tool for visual memorability
UTEgocentric
Insight Center for
Data Analytics

15. Annotation
15
[Khosla, ICCV 2015]

16. Annotation tool
16

17. ● Docker:
○ Container with an operating system and software required.
○ Always run the same in any environment.
● Simple implementation → dockerfile
17
Annotation tool implementation
Why to use
a Docker?
First docker
implementation
in GPI for
research

18. 18
● Memorability score → [0,1]
● Result:
○ Dataset → 50 annotated images (25 users)
Annotation tool results

19. Outline
➔ Introduction
➔ Contributions
◆ Annotation tool for visual memorability
◆ EgoMemNet: visual memorability adaptation to egocentric images
◆ Visual memorability and physiological signals
➔ Contributions
19

20. Convolutional neural network: definition
● Automatic learning paradigm based by how human brain works
● Neuron interconnection that work together to generate an output stimulus or activation
20

21. 21
Convolutional neural network: layers
Convolutional layer Fully connected layer

22. ● MemNet → CNN for memorability prediction
○ 5 conv layers + 2 fully connected layers + linear regression
22
EgoMemNet: visual memorability adaptation to egocentric images
MemNet CNN
[Koshla, ICCV 2015]
1
Structure: AlexNet

23. CNN fine-tuning
23
MemNet [Koshla, ICCV 2015]Insight dataset
(egocentric dataset)
EgoMemNet
1

24. ● No augmentation
● Spatial data augmentation → common method
● Temporal data augmentation → egocentric feature
24
Data augmentation strategies
Spatial data augmentation Temporal data augmentation

25. 25
Quantitative results
Spearman’s rank correlation
Compute the similarity between
positions between two different
ranked lists.
Memorability rank Ground truth rank

26. 26
Quantitative results
Spearman’s rank correlation

27. 27
Qualitatives results

28. 28
Memorability maps
● Heat maps that highlight most memorable regions.
● Methods:
○ Grid-and-forward → obtain a memorability score per patch
○ EgoMemNet → fully convolutional version
Grid-and-forward EgoMemNet

29. Memorability maps: grid-and-forward pass
29

30. Memorability maps: EgoMemNet
30
[Zhou, CVPR 2016]

31. 31
Memorability vs. saliency maps
Original image Saliency map
(SalNet CNN)
[Pan, CVPR 2016]
Memorability map
(EgoMemNet* CNN)
In green, parts shared between saliency and memorability
maps.
In blue, memorability regions non-salient.
In red, salient regions non-memorability
Binarized maps with
learned threshold

32. Outline
➔ Introduction
➔ Contributions
◆ Annotation tool for visual memorability
◆ EgoMemNet: visual memorability adaptation to egocentric images
◆ Visual memorability and physiological signals
➔ Conclusions
32

33. The Insight dataset
● Multimodal homemade
dataset:
○ Images
○ Memorability score
○ Heart rate value (during
image acquisition)
○ Galvanic skin response
(during image
acquisition)
33
Publicly available!

34. Heart rate correlation
34
Memory scores quantized in 8 bins
Mean heart rate in
the bin

35. Galvanic skin response
35
Memory scores quantized in 8 bins
Mean GSR in the
bin

36. Physiological signals for memorability prediction
36
SNAP !

37. Detect snap points
● Prior approach → efficient capture without image processing
37
Linear
Regression

38. Adding physiological signals for memorability prediction
● Post approach
38
Linear
Regression
EgoMemNet
score

39. New feature: EEG signals
● EEG → electroencephalographic signals
● Hands free visual memory game
39

40. EEG data extraction
40
Peak at 400 ms
P3@Pz → average 350-600 ms

41. 41
P3@PzPeak 400ms

42. Outline
➔ Introduction
➔ Contributions
◆ Annotation tool for visual memorability
◆ EgoMemNet: visual memorability adaptation to egocentric images
◆ Visual memorability and physiological signals
➔ Conclusions
42

43. Conclusions
● New annotation tool allows to create novel dataset for egocentric
memorability.
● Egocentric (first person vision) dataset containing 50 annotated images.
● EgoMemNet, a model adapted for memorability prediction to egocentric
images, presents a perform over MemNet, a convolutional neural network
model trained with human-taken images.
● Physiological signals for memorability prediction.
43

44. Extended abstract
Carné-Herrera M, Giró-i-Nieto X, Gurrin C. EgoMemNet: Visual Memorability Adaptation to Egocentric
Images. Las Vegas, NV, USA: 4th Workshop on Egocentric (First-Person) Vision, CVPR 2016;
44

45. Spotlight
Full spotlight in youtube!
https://www.youtube.com/watch?v=qwM5NNW37YE

46. Poster presentation
464646

47. 47
Open Research
Dataset Model Annotation
tool
http://imatge-upc.github.io/memory-2016-fpv/

48. 48
Hope this presentation has been
memorable!
Thanks for your attention!