Manually selecting subsets of photos from large collections in order to present them to friends or colleagues or to print them as photo books can be a tedious task. Today, fully automatic approaches are at hand for supporting users. They make use of pixel information extracted from the images, analyze contextual information such as capture time and focal aperture, or use both to determine a proper subset of photos. However, these approaches miss the most important factor in the photo selection process: the user. The goal of our approach is to consider individual interests. By recording and analyzing gaze information from the user's viewing photo collections, we obtain information on user's interests and use this information in the creation of personal photo selections. In a controlled experiment with 33 participants, we show that the selections can be significantly improved over a baseline approach by up to 22% when taking individual viewing behavior into account. We also obtained significantly better results for photos taken at an event participants were involved in compared with photos from another event.

1. 1
Smart Photo Selection:
Interpret Gaze as Personal Interest
Tina Walber1
, Ansgar Scherp2,3
, Steffen Staab1
1 Institute WeST, University of Koblenz, Germany
2 Kiel University, Germany
3 Leibniz Information Center for Economics, Kiel, Germany

2. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 2
Managment of Digital Photos
● Its a mess!
● We take a lot of photos
● Manually selecting photos is cumbersome
● Like to have photo selections for
– Sharing photos online
– Creating photo products like photo books
– Creating presentations

3. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 3
State of the Art: Automatic
Creation of Photo Selections
● Content-based approaches
– Analysis of low-level features
● Context-based approaches
– Analysis of context information
● What about individual aestetics, personal
preferences, user interests, ….?

4. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 4
Interpret Gaze as Personal Interest
● Gaze delivers information on user's interest
● Useful for creating individual photo selections?
● Principal approach
– Merely observe what users are doing anyway
– Do not ask to perfom additional tasks

5. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 5
● Starting from $99

6. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 6
Research Questions
1. Is there a need for individual photo selections?
2. Does a gaze-based selection outperform
selections based on content and context analysis
when comparing to those created manually?
3. Does the personal interest in a viewed photo set
have an impact on the obtained selection results?

7. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 7
Experiment Setup
Photo Viewing
Task:
„get an overview“
Step 1
Recording of the
eye tracking data

8. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 8
Photo Viewing
32 pages with 9 photos each

9. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 9
Experiment Setup
Photo Viewing
Task:
„get an overview“
Step 1
Photo Selection
Task: „select photos
for your private photo
collection“
Step 2
Recording of the
eye tracking data
Creation of
Ground Truth
Sm

10. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 10
Manual Selection
Creator:LibreOffice 3.5
LanguageLevel:2

11. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 11
Collection CA Collection CB
162 photos 126 photos
Experiment Data Set C = CA CB
∩
Two Data Sets and
Two User Groups
Institute A Institute B
Home collectionHome collection
Foreign collection
● Taken during social events of the research institutes

12. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 12
Participants
● 33 participants (12 of them female)
● 21 associated to Institute A, 12 to Institute B
● Aged between 25 and 62 (Ø 33.5 ± 9.57)
● 20 graduate students, 4 postdocs,
9 other professions

13. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 13
Overview Analysis and Evaluation
Se
Collection C
Gaze
Based
Selection
Calculation
of Precision
P
Manual
Selection
Content and
Context Based
Selection
Sb+e
Sb
Ground Truth
Sm

14. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 14
Baseline Measures
# Name Description
1 concentration
Time
Photo was taken with other photos in a
short period of time
2 sharpness Sharpness score from related work
3 numberOfFaces Number of faces
4 faceGaussian Size and position of faces
5 personsPopula
rity
Popularity of the depicted persons
6 faceArea Areas in pixels covered by faces
Selection of photos based on:
Calculated for each photo

15. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 15
Eye Tracking Data
●
Fixations and saccades
● Analysed gaze data with eye tracking measures
Creator:LibreOffice 3.5
LanguageLevel:2
● Viewing duration / page:
M = 12.6 s
● Number of fixations /
photo: M = 3.25

16. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 16
Eye Tracking Measures
# Name Description
7 fixated Was the photo was fixated?
Creator:LibreOffice 3.5
LanguageLevel:2

17. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 17
Eye Tracking Measures
# Name Description
7 fixated Was the photo was fixated?
8 fixationCount Counts the number of fixations
Creator:LibreOffice 3.5
LanguageLevel:2

18. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 18
Eye Tracking Measures
# Name Description
7 fixated Was the photo was fixated?
8 fixationCount Counts the number of fixations
9 fixationDuration Sum of duration of all fixations
Creator:LibreOffice 3.5
LanguageLevel:2

19. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 19
Eye Tracking Measures
# Name Description
7 fixated Was the photo was fixated?
8 fixationCount Counts the number of fixations
9 fixationDuration Sum of duration of all fixations
10 firstFixationDuration Duration of the first fixation
11 lastFixationDuration Duration of the last fixation
12 avgFixationDuration Average fixation duration
13 maxVisitDuration Maximum visit length
14 meanVisitDuration Average visit length
15 visitCount Number of visits
16 saccLength Mean length of the saccades
17 pupilMax Maximum pupil diameter
18 pupilMaxChange Maximum pupil diameter change
19 pupilAvg Average pupil diameter

20. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 20
Combination of Measures
● Using a model learned from logistic regression
● Assigns each image a probability of being
selected
● 30 random splits for training and test data

21. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 21
1. Is there a need for individual
photo selections?
1 21 41 61 81 101121141161181201221241261281
0
5
10
15
20
25
30
Photo with the highest number of selections
Photos with no selections
Photos in data set C
10 40 70 100 130 160 190 220 250 280
SelectionFrequency
● Manually created photo selections are diverse

22. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 22
2. Evaluation of the Photo Selections
PrecisionP
Sb Sb+e Se
*
*
Random
Selection
P = 0.428P = 0.365 P = 0.426
● Improvement of 17% over baseline

23. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 23
3. Impact of personal interest?
PrecisionP
Results for Sb+e
Foreign Collection Home Collection
P = 0.446P = 0.404
*

24. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 24
Conclusion
● Photo selection behavior is individual
● Gaze helps capture personal preferences
● Results are better for photos with personal interest
● Might work even better for real personal photos
● Potential application in photo book authoring
Thank you for your attention!