The Role of Eye Tracking in User Experience Research (Download at: http://www.usercentric.com) Webinar presented on behalf of the Human Factors and Ergonomics Society (HFES) December 11, 2009 By Gavin Lew, Managing Director of User Centric Back to 2009 Publications Index Gavin Lew, Managing Director of User Centric, presented a Webinar on behalf of the Human Factors and Ergonomics Society (HFES) on December 11, 2009. During this Webinar, The Role of Eye Tracking in User Experience Research, Gavin described eye tracking measures, methods, and applications to user experience research. User Centric is well known for its eye tracking research through the widely-published works of both Gavin Lew and Associate Director, Aga Bojko, a leading expert in the field. The HFES launched a series of Webinars in 2009; in 2010 there will be webinars on a bi-monthly basis. Webinar summary: The ability to track eye movements for scientific research has existed for generations. Recent technological advances in this area have made both data collection and analysis easy. Thus, what was once an extremely cumbersome and time consuming research endeavor, now is a very intriguing research technique—with the visual outputs sometimes being perceived as having more value than the data itself. With the growing number of eye tracking system installations in both academic and commercial research centers, a discussion of the role of eye tracking in user experience research is critical. Download the Full Presentation The presentation is available for free download in PDF format to the right (8 MB, Adobe Reader required). About the Human Factors and Ergonomics Society The Human Factors and Ergonomics Society was founded in 1957. Its mission is to promote the discovery and exchange of knowledge concerning the characteristics of human beings that are applicable to the design of systems and devices. The Society's 5,000+ members advocate systematic use of human factors knowledge to achieve compatibility in the design of interactive systems of people, machines, and environments to ensure their effectiveness, safety, and ease of performance. Through its 21 technical groups, 35 local chapters and 31 student chapters, the Society encourages education and training for those entering the human factors and ergonomics profession and for those who conceive, design, develop, manufacture, test, manage, and participate in systems.

1. The Role of Eye Tracking in
User Experience Research
HFES Webinar Series
Prepared by User Centric, Inc.
Gavin Lew
Managing Director
glew@usercentric.com
Presentation Delivered
11 December 2009

2. Outline
The Basics
Eye Tracking and UX Research
The Case for Quantitative Analysis
Ceilings and Floors
Method case study on packages
Q&A
2

3. Eye Tracking
The Basics
3

4. Eye Tracking: The Basics
Basics of Eye Movements
Eye tracking is a
research technique
that captures eye
behavior in response
to a visual stimulus
“Eye-mind
hypothesis”: where
people look is where
they focus their
attention
Saccadic eye
movements (most Saccade (line)
common) consist of: Fixation (circle)
– Fixations
– Saccades
4

5. Eye Tracking: The Basics
Sample Eye Measures
Quantitative Measures Meaning
Informativeness of an area / user interest in the
# fixations on an area
area
Info clarity / info density / info processing
Fixation length
demands
# fixations before target
Time to 1st target fixation Layout effectiveness / search demands
Scanpath complexity
% users fixating on an area
Order of 1st fixation Prominence / perceived importance of an area
# visits to area
Cognitive processing demands / user mental
Pupil diameter
workload / emotion
5

6. Eye Tracking: The Basics
Eye Trackers
Eye tracker determines the position of one or both eyes multiple
times (30 – 1000+) per second
Commonly used eye trackers differ in physical form, setup
procedures, and tracking methodology:
6

7. Eye Tracking: The Basics
The Tobii Eye Tracker
Our eye tracker: Tobii 1750
– Integrated into a 17”
computer monitor
– No restraints, freedom of
head movement
– Binocular tracking
– Sampling rate 50 Hz
– Quick and automatic
calibration
7

8. Eye Tracking: The Basics
Lab Setup
Moderator’s station to view Face Tobii 1750 remote eye-tracking
eye gaze in real time and camera system integrated into a 17”
control eye tracking software monitor (set to 1024 x 768 px)
One-way
mirror
Moderator Participant
8

9. Eye Tracking
Eye Tracking and UX Research
9

10. Eye Tracking: Eye Tracking and UX Research
Three Dimensions of Feedback
How people evaluate objects
− Commercials
− Packages
− Online advertisements
− Products
− …
© December 11, 2009 10

11. Eye Tracking: Eye Tracking and UX Research
Attitude
What users “say” ...
Influencers
– Social status
– Emotion
– Coolness / Hip
Reveal
– Feature importance
– Purchase intent
© December 11, 2009 11

12. Eye Tracking: Eye Tracking and UX Research
Behavior
What users actually “do”
Ultimately, behaviors are
what we wish to shape
Give users context, a task
and stimuli and then
– Observe what users do
Behavior drives usage
© December 11, 2009 12

13. Eye Tracking: Eye Tracking and UX Research
Attention
What users “focus” on
What happens inside the head
Sometimes users are unaware
Often attention measured by
eye tracking
© December 11, 2009 13

14. Eye Tracking: Eye Tracking and UX Research
Eye Tracking in UX Research
14

15. Eye Tracking: Eye Tracking and UX Research
How Can Eye Tracking Help Usability Testing?
TASK: Find a branch near you
Page with no
gaze. With
something hard
to find.
That’s
impossible.
15

16. Eye Tracking: Eye Tracking and UX Research
How Can Eye Tracking Help Usability Testing?
ET FINDINGS:
(1) Average # fixations before the target
link was found: 112
(2) The top right corner of the page
attracted initial & the most fixations.
Ah, quant data…
16

17. Eye Tracking: Eye Tracking and UX Research
How Can Eye Tracking Help Usability Testing?
TASK: Find a list of indoor climbing walls.
So… she only
considered the
first two results?
17

18. Eye Tracking: Eye Tracking and UX Research
How Can Eye Tracking Help Usability Testing?
ET FINDING: She looked at (considered) more
results than just the top 2.
18

19. Eye Tracking: Eye Tracking and UX Research
How Can Eye Tracking Help Usability Testing?
Typically in usability testing, we
collect:
– Behavioral measures (e.g.,
clicks, time)
– User self-report (e.g., RTA)
Eye tracking can:
– Support and illustrate UT
findings
– Help determine user
expectations
– Augment usability findings
by filling in the gaps
between
• Observable events
• User comments
19

20. Eye Tracking: Eye Tracking and UX Research
How Can Usability Testing Help Eye Tracking?
Different eye movement patterns
produced by the same person
looking at the same picture…
Why are they
all so different?
Yarbus, A. L. Eye Movements and Vision. Plenum. New York. 1967
20

21. Eye Tracking: Eye Tracking and UX Research
How Can Usability Testing Help Eye Tracking?
TASK: Estimate
people’s ages.
Different eye movement patterns
produced by the same person looking
at the same picture…
but in a different context!
TASK: Estimate the
family’s material
circumstances.
Yarbus, A. L. Eye Movements and Vision. Plenum. New York. 1967
21

22. Eye Tracking: Eye Tracking and UX Research
How Can Usability Testing Help Eye Tracking?
TASK: Find the museum hours.
22

23. Eye Tracking: Eye Tracking and UX Research
How Can Usability Testing Help Eye Tracking?
ET FINDINGS:
(1) All users looked at the image.
(2) Time spent looking at the image:
30% of all time spent on page.
Great!
Image got lots
of attention!
23

24. Eye Tracking: Eye Tracking and UX Research
How Can Usability Testing Help Eye Tracking?
Actual target link
So this is
bad…
24

25. Eye Tracking: Eye Tracking and UX Research
Usability Testing and Eye Tracking
Usability Eye Tracking
Provides context for…
Testing
reveals factors
reveals (mostly) that contribute
outcomes of Illustrates and helps understand to the outcomes
interaction
ANSWERS PROVIDES
PRACTICAL MORE
QUESTIONS DETAILED
ANALYSIS
25

26. Eye Tracking: Eye Tracking and UX Research
Eye Tracking In Isolation Yields Uncertainty
Eye tracking has limited
applicability when used in
isolation
– A fixation on a face may
indicate recognition, liking,
dislike, or confusion
– More fixations may indicate
interest or inefficient search
Instead combine with
attitudinal and behavioral
probes
– Learn the “why”
– Just knowing where people
look is often insufficient
26

27. Eye Tracking: Eye Tracking and UX Research
Context is Key for Eye Tracking
Is this good or bad?
Were users told to:
– Signup for Newsletters?
– Learn about services?
– Read publications?
– Look for a job?
Or worse, they were not given
any context at all
Always ask about
the context / task!
© December 11, 2009 – Proprietary and Confidential 27

28. Eye Tracking
The Case for Quantitative Analysis
28

29. Eye Tracking: The Case for Quant Analysis
Heatmap: Classic Eye Tracking Output
With thousands of data
points to analyze,
interpretation should be
based on data…
© December 11, 2009 – Proprietary and Confidential 29

30. Eye Tracking: The Case for Quant Analysis
Visualizations Facilitate, Not Interpret
© December 11, 2009 – Proprietary and Confidential 30

31. Eye Tracking: The Case for Quant Analysis
Graphs are Visualizations too… Show Me Data
© December 11, 2009 – Proprietary and Confidential 31

32. Eye Tracking
The Case for Quantitative Analysis
• Case Study
32

33. Eye Tracking: The Case for Quant Analysis
Selecting One Design
Concept 1 Concept 3
Existing Homepage
Concept 2 Concept 4
© December 11, 2009 – Proprietary and Confidential 33

34. Eye Tracking: The Case for Quant Analysis
Free View: First Impressions
Client identified key business
goals
MY ACCOUNT
Between-groups design
– Each group saw a different
design PERSONAL
/BUSINESS PROMO
1st impressions task:
– You are looking for a new
wireless provider and you
decided to check what
Verizon Wireless has to
offer...
Homepage was shown for 10
sec.
© December 11, 2009 – Proprietary and Confidential 34

35. Eye Tracking: The Case for Quant Analysis
Which Won?
CONCEPT 1 CONCEPT 2 CONCEPT 3 CONCEPT 4 EXISTING
Remember the
measures? What
is the heatmap
showing?!?!
% of users who
fixated?
Fixation length?
# of fixations?
Eye Tracking: The Basics
Sample Eye Measures
Quantitative Measures Meaning CONCEPT 1 CONCEPT 2 CONCEPT 3 CONCEPT 4 EXISTING
Informativeness of an area / user interest in the
# fixations on an area
area
Info clarity / info density / info processing
Fixation length
demands
# fixations before target
Time to 1st target fixation Layout effectiveness / search demands
Scanpath complexity
% users fixating on an area
Order of 1st fixation Prominence / perceived importance of an area
# visits to area
Pupil diameter
Cognitive processing demands / user mental Oh, wait. Maybe
workload / emotion
this will help
36
© December 11, 2009 – Proprietary and Confidential 35

36. Eye Tracking: The Case for Quant Analysis
You Must Know the Measure!
% users who fixated each area
CONCEPT 1 CONCEPT 2 CONCEPT 3 CONCEPT 4 EXISTING
Order in which the areas were first visited
CONCEPT 1 CONCEPT 2 CONCEPT 3 CONCEPT 4 EXISTING
© December 11, 2009 – Proprietary and Confidential 36

37. Eye Tracking: The Case for Quant Analysis
Quantitative Results
% users who fixated each area
CONCEPT 1 CONCEPT 2 CONCEPT 3 CONCEPT 4 EXISTING
A C A A
C C
B B
B B A
C B
A C
KEY AREAS OF INTEREST
AREA A: My Account 100% 79% 77% 64% 68%
AREA B: Promo 100% 100% 100% 100% 100%
AREA C: Personal/Business 100% 85% 80% 100% 52%
Order in which the areas were first visited
CONCEPT 1 CONCEPT 2 CONCEPT 3 CONCEPT 4 EXISTING
KEY AREAS OF INTEREST
AREA A: My Account 2nd >5th > 5th 3rd 4th
AREA B: Promo 3rd 1st 4th 2nd 1st
AREA C: Personal/Business 1st 2nd 1st 1st 3rd
© December 11, 2009 – Proprietary and Confidential 37

38. Eye Tracking: The Case for Quant Analysis
Quantitative Results
# of fixations on an area (out of ~31 on average)
CONCEPT 1 CONCEPT 2 CONCEPT 3 CONCEPT 4 EXISTING
KEY AREAS OF INTEREST
AREA A: My Account 3.8 3.4 2.3 2.2 2.3
AREA B: Promo 8.2 8.0 5.5 6.5 8.3
AREA C: Personal/Business 8.2 4.2 3.5 6.9 3.2
TOTAL 20.2 15.6 11.3 15.6 13.8
© December 11, 2009 – Proprietary and Confidential 38

39. Eye Tracking: The Case for Quant Analysis
Bottom Line
Link measures to research
question
MY ACCOUNT
When looking at data, avoid
the allure of reading tea
leaves PERSONAL
/BUSINESS PROMO
Ask:
– What is the context/task?
– What is the measure?
– Where is the quant?
© December 11, 2009 – Proprietary and Confidential 39

40. Eye Tracking
Ceilings and Floors
40

41. Eye Tracking: Ceilings and Floors
Commercial aired in 2003
FYI: Commercial is for a cable TV, Internet and telephone provider
41

42. Eye Tracking: Ceilings and Floors
What Did You See?
Services offered
Internet web page
Money in a piggy bank
Let’s see what Participant #7 saw…
42

43. Eye Tracking: Ceilings and Floors
Participant #7’s Gaze Replay
43

44. Eye Tracking: Ceilings and Floors
What Did Participant #7 See?
Key words
Watched pig
Even followed the mouse!
In testing, we asked about the commercial…
– Love it
– I want the deal
So, you go and get it!
– Literally. Participant asked to open a browser and off they went
– They even entered the URL…
44

45. Eye Tracking: Ceilings and Floors
TV Commercial to Web “Handshake”
Empirical weblog data:
– 20% clicked the pig
45

46. Eye Tracking: Ceilings and Floors
Results
Attitude
– Viewers liked the ad
– Some even “loved it” and wanted to get the “deal”
Behavior
– In the study, 23% clicked (empirically, 20% click)
Note how Eye
Attention Tracking was not
used in isolation…
– 70% looked
• But, the pig spins around, so some glances will occur
– Setting a 150ms threshold (look at pig, read, etc.)
• 40% looked
• Relative to ad spaces on site
46

47. Eye Tracking: Ceilings and Floors
Ceilings and Floors
Not everything you design gets attention (there is a “ceiling”)
Designers and marketers tend to assume that if you build it, 100%
will see it
What would you say about these results?
We said:
– Not bad. Your ceiling was 40% and you are getting 20%.
– To improve click-thru, improve attention.
• If you do not attend, you will not click.
Eye Tracking is useful answer: Do user NOT CLICK because
– They looked, but the value proposition <> click (marketing).
– They never looked (design).
47

48. Eye Tracking
Comparing Search Results: Bing vs. Google
48

49. Eye Tracking: Bing vs. Google
Comparing Two Search Results Pages
Research objective:
– Compare the distribution of attention on equivalent areas of Bing
and Google
– Assess how much attention is captured by elements that are
unique to Bing
Participants (N=21) completed search tasks for each engine
– Two informational (e.g., "Learn about eating healthy")
– Two transactional (e.g., "Book a last minute vacation“)
49

50. Eye Tracking: Bing vs. Google
Gaze Duration for One Task
© December 11, 2009 – Proprietary and Confidential 50

51. Eye Tracking: Bing vs. Google
Results: Similarities
Google and Bing did not differ in
amount of attention on the organic
search results
– In each search, all participants
looked at the organic search
results, spending an average of 7
seconds in that area
Attention on the sponsored links
located above the organic results was
similarly high for both Bing and
Google
– Over 90% of participants looked in
that area during each search.
51

52. Eye Tracking: Bing vs. Google
Results: Differences
Sponsored links on the right attracted
more attention
– Bing (~42% of participants per
search) than they did on Google
(~25% of participants per search)
– Participants who fixated on these
links spent approximately 2.5
seconds looking at the area during
transactional searches and 2
seconds during informational
searches
52

53. Eye Tracking
Method Case Study on Packages
53

54. Eye Tracking: Method
Method Case Study
Study objectives:
– Determine how the client’s packaging
compares to competitors’ packaging in
Marketing
terms of noticeability and visual
engagement / interest
– Assess the findability and clarity of the
information on the client’s package as
compared to the information on the Usability
competitors’ packages
Wearable eye trackers vs. remote eye trackers
54

55. Eye Tracking: Method
A Walk Down The Aisle…
Macro-level:
– Participants saw pictures of
aisles
– Five picture frames per
store (2 s per frame)
– Each showed the entire
prepaid phones display
– However, the display was in
a different position in each
frame to simulate the
changing view of a moving
customer (in a “freeze
frame” way) and minimize
the effect of product position
in the picture
55

56. Eye Tracking: Method
Product View
Micro-level: Front Back
– Participants saw pictures of
individual packages
– Given tasks to answer to find
information on the package
itself
– Participants were able to
“flip” package around by
clicking
Q3: Can you
Q2: What is
The answer is: browse the The answer is:
the brand of
______ Web on this ______
this phone?
phone?
56

57. Eye Tracking: Method
Actual Package Experience
Constructed shelves
Participants were different from
the macro- and micro-level
study
– Far view
– Up close
– Manipulation
– Selection decision
– Discussion
57

58. Eye Tracking: Method
Measures
Memory
– Free recall of products and brands
– Recognition exercises of products and brands
Preference
– First impressions from afar
– Likes and dislikes
Performance tasks
– Accuracy of finding answers to tasks
– Efficiency in time to find answers (excluded incorrect answers)
Eye tracking (computed based on package area)
– % who looked (noticeability)
– # of fixations (visual engagement)
58

59. Eye Tracking
Q&A
59

60. Q &A
Final Takeaways
Eye Tracking should be combined with other UX techniques
– Avoid using ET in isolation
Context matters
– One can change a heatmap with just four words!
When looking at a heatmap, ask:
– What was the task or context?
– What is the heatmap showing?
• % who fixated, gaze duration
• # of fixations, etc.
• Each measure reveals a different story…
With so much data, where is the quantitative analysis?
60

61. Q &A
Q & A?
Questions?
Additional references:
– Me: glew@usercentric.com
– Peer-reviewed journal articles, presentations,
white papers on www.usercentric.com
– ET case study of package labels by Aga Bojko
in Tullis and Albert’s book, Measuring The
User Experience
– ET global case study in Bob Schumacher’s
book, The Handbook of Global Research
• www.globaluserresearch.com
• www.elsevierdirect.com
61