3. Research results
Research so far has indicated
that visitors spend little time
looking at artworks:
• Hein, 1998
• Smith, 2001
• Worths, 2003 (‘grazing’)
• Viewing Project, IMA 2012
4. Observation vs. eye tracking
• Use of observational rubric is
labor-intensive and not always
precise
• Eye tracking technology has
the potential of being more
precise and less time-
consuming
• More recent development of
less intrusive devices (non
head mounted)
Photo from Milekic MW 2010
5. Sparks! grant objectives
• Gauge the practicality of
using such devices in a
museum gallery setting.
• Assess the ability of current
eye tracking technology to
reveal what visitors are
looking at and for how long.
• Explore the potential use of
this equipment in a practical
setting (e.g. VTS discussion)
7. Device initial testing
• Eye tracker range
limitations.
• Too much variation in
height when the person is
standing.
• For the experiments the
viewer has to be seated,
with the tracker placed in a
fixed position between the
tracker and the painting.
9. Experiment 1: objectives
• Distinguish when the
participant looks
inside/outside of the
painting.
• Measure time spent looking
inside/outside of the
painting.
• Track where looking inside
the field of the artwork.
Calibration performed once
10. The device was installed on a cart between the work of art and
the seated participant and calibrated to the first participant.
12. Experiment 1: part 1
• 22 participants were asked to look in
and outside the painting for 1 minute.
• First 10 participants could not adjust
their chair position to optimize eye
tracking, while the next 12 were asked
to do so.
• Participants’ gazes (inside the field of
the painting) were tracked by 2
research assistants with stopwatches.
• The times were averaged and
compared to the time tracked by the
device
13. Experiment 1: part 2
• A subset of participants (8 of
the 22) were asked to look
(over a period of 60 seconds)
at 6 different areas of the
work for 10 seconds each in
sequence prompted by a
research assistant.
• Tracker data was logged in the
same manner as the previous
experiment.
14. Relative quantity of valid gaze data
Missing data for >25% of session time for 6 participants
20
10
difference (% of session time)
0
-10
-20
-30
-40
-50
-60
-70
-80
Fixed seat position Adjusted seat position
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
15. Relative quantity of valid gaze data
Poor performance for 4 of 6 participants at low eye level (<50”)
20
10
difference (% of session time)
0
-10
-20
-30
-40
-50
-60
-70
-80
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
18. Comparison against
manual measurement
Fixed seat position Adjusted seat position
Within 5% 0 5
Within 10% 1 7
• Allowing the participant to adjust the seat position produces
better results
• We were still hoping for better accuracy
20. Gap Handling
Raw 100ms 500ms 1s
Within 5% (fixed) 0 1 1 1
Within 10% (fixed) 1 3 3 4
Within 5% (adjusted) 5 5 8 7
Within 10% (adjusted) 7 7 8 8
• Applying a gap algorithm appears to improve
results with a 500ms threshold
• Accuracy was within 10% for two thirds of
participants when allowing the seat position to
be adjusted and using the algorithm
24. Experiment 1: Summary
• Device was not able to
continuously track the gaze
of a seated viewer.
• An attempt to improve
results by handling gaps in
the data were successful, but
only to a degree.
• Vertical gaze location was not
accurate for uncalibrated
viewers.
26. Experiment 2: objectives
& methodology
• Measure whether the
device could be more precise
when calibrated for each
participant
• We repeated experiment 1
(part 1 and 2) with 12
participants but calibrated the
devices individually
• This second experiment was set
in a lab, where the exact size of
the painting was reproduced on a
board
28. Gaze Duration
Delta Max. calibration Glasses Seated eye
(% of session time) “score” elevation
7.719 6.70 did without glasses 49
15.159 2.76 yes 51.5
6.306 13.90 no 54
6.728 6.13 yes 52
5.164 3.30 no 51.25
10.071 6.78 no 48
0.007 5.13 no 53.75
6.082 4.07 did without glasses 47.75
4.572 4.36 yes 50.5
2.732 10.10 no 52
12.240 12.15 yes 49.5
3.790 3.77 no 51
29. Gaze Duration
Accuracy Raw data 100ms threshold 500ms threshold
Within 2% 1 4 6
Within 5% 4 7 8
Within 10% 9 12 12
• An improvement over the first experiment
• One third of participants were in the 5-10% range
32. Comparisons
Average error Max. calibration “score” Glasses
(degrees of FOV)
0.96 6.70 no
1.60 2.76 yes
1.72 13.90 no
2.47 6.13 yes
0.88 3.30 no
1.61 6.78 no
0.90 5.13 no
1.24 4.07 no
1.18 4.36 yes
1.76 10.10 no
3.00 12.15 yes
2.22 3.77 no
33. Experiment 2: Summary
• Applying the gap handling
algorithm brought all sessions
within 10% of the manual
measurement
• Gaze duration results were
better than in the uncalibrated
study, but still not what we
hoped for
• Gaze location results were also
better than in the uncalibrated
study, but not as accurate as
expected
34. Future Work
• Experiment 3 will
make use of the
tracker during a VTS
session
• We will evaluate
whether the data
recorded assists in
understanding what
VTS participants look
Photo from PAAM.org
at during a session
Visitor engagement with works of art has been the subject of numerous studies in the course of the last 30 years.
Data shows that visitors spend little time at individual exhibit components and seldom read labels (Hein, 1998)Studies at the MET found a mean time of less than 30 seconds viewing an object (Smith, 2001)Visitors wander slowly past many artworks spending only seconds looking at a particular one (“grazing” – Worts, 2003)Time spent looking at an artwork between 4 and 31 seconds (median time) covering a number of installations (IMA’s Viewing Project Report, 2012)
Techniques used by researchers for measuring visitor gaze are based on observational rubrics.These techniques are labor intensive and unable to tell us at exactly what the visitor is looking.Eye tracking technology can be useful in this respect, especially more recently with the development of less intrusive (as opposed to a head-mounted device) eye tracking devices that can be used to track gaze from afar.
Given the recent developments in eye tracking technology and its ability to track visitors gaze less obtrusively, the Indianapolis Museum of Art has been awarded in the spring of 2011 an IMLS grant with the aim of:1) Gauging the practicality of using such devices in a gallery setting.2) Determining the ability of current eye tracking technology to measure precisely how long and what people are looking at.3)Explore the potential use of this equipment in a practical setting like a VTS session or as a way to activate content relevant to a work of art.These issues were to be addressed and tested in the context of 3 separate experiments to be conducted at the IMA between July 2011 and June 2012.
In determining which hardware technology to use for our research, head mounted trackers were ruled out because one of the project goals is to determine whether an eye tracking system can be used to detect the gaze of a general museum visitor. Of the non head-mounted models available, the EyeTech Digital Systems VT2 eye tracker was chosen to be used in the experiments (http://www.eyetechds.com/). How does it work? During operation, the eye tracker emits IR light toward the viewer and captures a series of images (called frames) with a camera. Each frame is analyzed by the tracker to determine the position of the reflections of the IR light on the surface of the eyes. The tracker reports a set of data related to each frame, the calculated gaze point relative to the points used for calibration (if calibrated), and other parameters derived during the calculation.
In our discussion with the representatives from EyeTech and experience during development, we found that the tracker has a series of limitations when it comes to its range. The tracker has an ideal viewing range of 25 inches from thefront panel. It is also unable to detect the eyes if they are looking too far past the left or right edges of the device or toohigh above the device. What this meant for our experiments was that detecting the eyes of a person with arbitrary height when standing was not going to be reliable, as adjustments to the position and/or angle of the tracker would be required.Our experimentation was therefore restricted to a scenario in which the viewer is seated, with the tracker placed in a fixed position between the tracker and the painting.
The piece chosen was Edward Hopper’s Hotel Lobby due to:It being on display for the grant period.It being suitable for a VTS discussion or to trigger audio content.There being noticeable representational details that could be used to focus one’s attention.
Understanding the device’s effectiveness when non-calibrated for each individual user:Distinguish when the participant looks inside/outside of the painting.Measure time spent looking inside/outside of the painting.Track where looking inside the field of the artwork.
Overall, invalid frames surpassed the amount of time spent looking away by 20% of the session time for 6 of the participants. While this was the case during phase 1 (participants 1-10) for 4 people (in fact no valid frames were reported during two of these sessions), an improvement can be seen in phase 2 (participants 11-22), where the seat position was adjusted.
The duration represented by invalid frames surpassed the amount of time spent looking away by over 40% of the session time for 4 of the 6 participants whose eye level was below 50” from the ground. The tracker was not able to register any valid data for two of these participants in phase 1, and two participants in phase 2 had difficulty getting into a good position. There did not appear to be a correlation to the quantity of valid data for seated eye levels between 50 and 53.75 inches (the highest seated eye level in the study).
With 500ms threshold, 9 sessions are within 5% (was 5)Also 11 sessions are within 10% (was 7)Increasing to 1s introduces error
The distribution of points within a cluster is approx 5%This corresponds to roughly 1.5% of the field of view, which is the region within which the eye moves involuntarily while fixating
Clustering was performed without removing outliers
As issues of accuracy in determining exactly where and for how long the participant is looking have emerged from experiment 1, it was decided to repeat the procedure for experiment 2 but this time we calibrated the device for each participant (12 in total) to see if the results would be more accurate.Unlike the first experiment which took place in the gallery, this second experiment was set in one of the IMA Adult Lecture room, where the exact size of the painting was reproduced on a white board. The use of the board as opposed to a painting reduced the chance of distraction during this test of accuracy, and more exact reference points allowed for higher precision in evaluation.
