This document summarizes a study on the effects of different visual display options in a web-based survey. The study tested four versions that varied the location of the Next and Previous buttons and the presentation of response options as single or double columns. Participants completed the survey while their eye movements were tracked. Results showed higher satisfaction ratings for versions that placed the Previous button on the left and presented response options in double columns. Eye tracking data provided insights into how participants viewed the different display elements.

1. Display
Ma*ers:
A
Test
of
Visual
Display
Op6ons
in
a
Web-­‐Based
Survey
Jennifer
C.
Romano
Bergstrom1,
Jennifer
M.
Chen1,
Timothy
R.
Gilbert2
&
Ma*
Jans1
1
Center
for
Survey
Measurement
2
Demographic
Surveys
Division
U.S.
Census
Bureau
AAPOR
66th
Annual
Conference
May
13,
2011

2. Current
Survey
Environment
• Increasing
number
of
surveys
online
• Design
considera6ons
– Naviga6on
methods
– Presenta6on
of
response
op6ons
2

3. Current
Survey
Environment
• Increasing
number
of
surveys
online
• Design
considera6ons
– Naviga6on
methods
– Presenta6on
of
response
op6ons
3

4. Background
on
Next
and
Previous
• Next
should
be
on
the
leU
– Reduces
the
amount
of
6me
to
move
cursor
to
primary
naviga6on
bu*on
(Couper,
2008)
– Frequency
of
use
(Dillman
et
al.,
2009;
Faulkner,
1998;
Koyani
et
al.,
2004;
Wroblewski,
2008)
4

5. Background
on
Next
and
Previous
• Previous
should
be
on
the
leU
– Web
applica6on
order
– Everyday
devices
– Logical
reading
order
5

6. Background
on
Next
and
Previous
• Previous
should
be
on
the
leU
– Web
applica6on
order
– Everyday
devices
– Logical
reading
order
6

7. Background
on
Next
and
Previous
• Previous
should
be
below
Next
– Bu*ons
can
be
closer
(Couper
et
al.,
2011;
Wroblewski,
2008)
7

8. Background
on
Long
Lists
• One
column
– Visually
appear
to
belong
to
one
group
– When
there
are
two
columns,
2nd
one
may
not
be
seen
(Smyth
et
al.,
1997)
• Two
columns:
Double
banked
– No
scrolling
– See
all
op6ons
at
once
– Appears
shorter
8

9. Measuring
“Best”
Design
• Typical:
In
the
Field
– Drop-­‐off
rates
– Keystrokes
– Survey
comple6on
6mes
• Our
Study:
In
the
Lab
– User
sa6sfac6on
– Eye-­‐tracking
data
– Usability
metrics
9

10. Usability
• The
extent
to
which
a
product
can
be
used
by
specified
users
to
achieve
specified
goals
with
effec6veness,
efficiency,
and
sa6sfac6on.
ISO/
TR
16982:2002
• For
web-­‐based
surveys,
the
design
must
– Meet
respondents’
needs
– Facilitate
easy
comple6on
– Provide
a
sa6sfying
experience
– Reduce
respondent
burden
– Produce
high-­‐quality
data
10

11. Na6onal
Survey
of
College
Graduates
(NSCG)
• Collects
educa6on
and
job
informa6on
• Respondents
have
Bachelor’s
degree
• Was
available
in
PAPI
and
CATI
• Usability
study
for
a
web-­‐based
self-­‐
administered
instrument
11

12. Method
• Lab-­‐based
usability
study
• TA
read
introduc6on
and
leU
le*er
on
desk
• Separate
rooms
• R
read
le*er
and
logged
in
to
survey
• Think
Aloud
(Olmsted-­‐Hawala
et
al.,
2010)
• Eye
Tracking
• Sa6sfac6on
Ques6onnaire
• Debriefing
12

13. Par6cipants
Gender
N
Age
N
Educa.on
N
Male
14
<
30
8
Bachelor’s
21
Female
16
31-­‐45
7
Master’s
6
46-­‐60
10
Ph.D.
3
>
60
5
Mean:
46
13

14. Eye-­‐Tracking
Apparatus
14

15. Ques6ons
Eye
Tracking
Can
Answer
• Do
respondents
look
at
Next
and
Previous?
• What
do
they
look
at
first?
• Is
it
distrac6ng
when
Previous
is
located
in
a
par6cular
place
on
the
screen?
• How
long
does
it
take
respondents
to
see
the
Next
bu*on?
• Does
presenta6on
of
long
lists
affect
what
users
look
at
on
the
list?
15

16. Previous
and
Next
Bu*ons
16

17. One
Column
vs.
Two
Columns
17

18. 4
Versions
N_P1
N_P2
Next
bu*on
on
leU,
Next
bu*on
on
leU,
1-­‐column
job
code
2-­‐column
job
code
PN1
PN2
Previous
bu*on
on
leU,
Previous
bu*on
on
leU,
1-­‐column
job
code
2-­‐column
job
code
18

19. Results:
Sa6sfac6on
I
*
p
<
0.0001
19

20. 8.5
Results:
Sa6sfac6on
II
8.5
Mean
Sa.sfac.on
Mean
Sa.sfac.on
8
8
Ra.ng
7.5
Ra.ng
7.5
7
7
6.5
6.5
6
6
Mean
N_P
PN
Mean
N_P
PN
Overall
reac6on
to
the
survey:
Informa6on
displayed
on
the
screens:
terrible
–
wonderful.
p
<
0.05.
inadequate
–
adequate.
p
=
0.07.
8.5
8.5
Mean
Sa.sfac.on
Mean
Sa.sfac.on
8
8
Ra.ng
7.5
Ra.ng
7.5
7
7
6.5
6.5
6
6
Mean
N_P
PN
Mean
N_P
PN
Arrangement
of
informa6on
on
the
screens:
Forward
naviga6on:
illogical
–
logical.
p
=
0.19.
impossible
–
easy.
p
=
0.13.
20

21. Eye
Tracking:
Next
/
Previous
21

22. Eye
Tracking:
Previous
/
Next
22

23. Eye
Tracking:
N_P
vs.
PN
• Par6cipants
looked
at
Previous
and
Next
in
PN
condi6ons
• Many
par6cipants
looked
at
Previous
in
the
N_P
condi6ons
– Consistent
with
Couper
et
al.
(2011):
Previous
gets
used
more
when
it
is
on
the
right
23

24. Eye
Tracking:
Time
to
First
Fixa6on
8
7.5
7
6.5
Seconds
6
PN
5.5
N_P
5
4.5
4
Next
Previous
Mean
6me
to
first
look
at
the
naviga6on
bu*on
24

25. N_P
vs.
PN:
Respondent
Debriefing
• N_P
version
– Counterintui6ve
– Don’t
like
the
“bu*ons
being
flipped.”
– Next
on
the
leU
is
“really
irrita6ng.”
– Order
is
“opposite
of
what
most
people
would
design.”
• PN
version
– “Pre*y
standard,
like
what
you
typically
see.”
– The
loca6on
is
“logical.”
25

26. 1
Column
vs.
2
Column
26

27. Time
to
First
Fixa6on
25
20
15
Seconds
1
col
*
p
<
0.01
10
2
col
5
0
First
half
of
list
Second
half
of
list
27

28. Total
Number
of
Fixa6ons
40
35
30
Number
of
Fixa.ons
25
20
1
col
15
2
col
10
5
0
First
half
of
list
Second
half
of
list
28

29. Time
to
Complete
Item
120
100
80
Seconds
60
1
col
2
col
40
20
0
Mean
Min
Max
29

30. 1
Col.
vs.
2
Col.:
Debriefing
• 25
had
a
preference
– 6
preferred
one
column
• They
had
received
the
one-­‐column
version
– 19
preferred
2
columns
• 7
had
received
the
one-­‐column
version
• Prefer
not
to
scroll
• Want
to
see
and
compare
everything
at
once
• It
is
easier
to
“look
through,”
to
scan,
to
read
• Re
one
column,
“How
long
is
this
list
going
to
be?”
30

31. Conclusions
• Par6cipants
were
more
sa6sfied
when
Previous
was
on
the
leU.
• Par6cipants
preferred
the
long
lists
in
two
columns.
• Par6cipants
looked
at
the
first
half
of
the
list
sooner
than
the
second
half
when
in
one
column.
• Par6cipants
looked
at
the
second
half
of
the
list
more
when
it
was
in
two
columns.
31

32. Bigger
Picture:
Recap
on
Next
and
Previous
• Next
should
be
on
the
leU
– Reduces
the
amount
of
6me
to
move
cursor
to
primary
naviga6on
bu*on
– Tab
order
– Frequency
of
use
• Previous
should
be
on
the
leU
– Web
applica6on
order
– Everyday
devices
– Logical
reading
order
– People
are
more
sa6sfied
– It
takes
longer
to
first
look
at
Previous
when
on
the
right
32

33. Bigger
Picture:
Recap
on
Long
Lists
• One
column
– Visually
appear
to
belong
to
one
group
• Two
columns:
Double
banked
– No
scrolling
– See
all
op6ons
at
once
– Appears
shorter
– Second
column
may
not
be
seen
– People
look
at
the
second
half
more
– People
look
at
the
first
half
sooner
when
it
is
in
one
column
– People
prefer
two
columns
33

34. Future
Direc6ons
• This
is
just
a
small
nugget.
• N_P
vs.
P_N
study
in
progress
– Same
layout
– No
skip
pa*erns
– Efficiency
measure
• Long
list
of
items
condi6on
– Which
items
do
people
pick?
– Alphabe6zed
vs.
random
order
34

35. Thank
you!
For
more
informa6on,
please
contact
Jennifer
Romano
Bergstrom
Jennifer.C.Romano@gmail.com
Jennifer.Romano@census.gov
Twi*er:
@romanocog
35