The visual design of surveys and other types of online data collection tools impacts how users perceive, understand and navigate the instrument as well as the responses they provide. Two key considerations that impact how users experience online data collection tools are the device they are using (e.g., smartphone, tablet, computer) and the method of interaction (e.g., website, app or both). When designing online data collection tools, creating a common user experience across different devices and methods of interaction is important to create a consistent user experience and to minimize measurement differences. In this talk, we will compare the user experience across four different combinations of device and method of interaction of a survey: (1) desktop PC-website, (2) smartphone-app, (3) tablet-app, and (4) tablet-website. Through performance and eye-tracking data, we identify UX elements that must be unified across all devices as well as elements that might need customization for difference devices or methods of interaction.

1. Unifying the User Experience of a
Survey Across Multiple Devices
Jennifer Romano Bergstrom, David Hawkins, Jon Strohl
December 12, 2013
MoDevEast | McLean, VA
@forsmarshgroup @romanocog

2. About this talk
1. Usability and User Experience Intro
2. Web Survey Design
3. Implications from Eye Tracking on a Mobile and Desktop
Tool
@forsmarshgroup @romanocog
2

3. What is Usability?
“the extent to which
a product can be
used by specified
users to achieve
specified goals with
effectiveness,
efficiency, and
satisfaction in a
specified context of
use.” ISO 9241-11
@forsmarshgroup @romanocog
3

4. Usability vs. User Experience (UX)
Whitney’s 5 Es of Usability
Peter’s User Experience
Honeycomb
@forsmarshgroup @romanocog
4
The 5 Es to Understanding Users (W. Quesenbery): http://www.wqusability.com/articles/getting-started.html
User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php

5. What People do on the Web
@forsmarshgroup @romanocog
5
Krug, S. Don’t Make Me Think

6. UX Design Failures
•
•
•
Poor planning
“It’s all about me.” (Redish: filing cabinets)
Human cognitive limitations
• Memory and Perception
• (activity)
• Primacy
• Recency
• Chunking
• Patterns
@forsmarshgroup @romanocog
6

7. Patterns
@forsmarshgroup @romanocog
7

8. Patterns
@forsmarshgroup @romanocog
8

9. Patterns
@forsmarshgroup @romanocog
9

10. Patterns
@forsmarshgroup @romanocog
10

11. Mental models and repeating behaviors
@forsmarshgroup @romanocog
11

12. Mental models and repeating behaviors
@forsmarshgroup @romanocog
12

13. Web Survey Design

14. Why is Design Important?
• No interviewer present to correct/advise
• Visual presentation affects responses
• While the Internet provides many ways to enhance surveys,
design tools may be misused
• Respondents extract meaning from how question and
response options are displayed
• Design may distract from or interfere with responses
• Design may affect data quality
@forsmarshgroup @romanocog
14

15. Why is Design Important?
@forsmarshgroup @romanocog
15
http://www.cc.gatech.edu/gvu/user_surveys/

16. Why is Design Important?
Many surveys are long (> 30min)
Long surveys have higher nonresponse rates
Length affects quality
Respondents are more tech savvy today and use multiple
technologies
• It is not just about reducing respondent burden and
nonresponse
• We must increase engagement
• High-quality design = trust in the designer
•
•
•
•
@forsmarshgroup @romanocog
16
Adams & Darwin, 1982; Dillman et al., 1993;
Haberlein & Baumgartner, 1978

17. @forsmarshgroup @romanocog
17
http://www.pewinternet.org/Static-Pages/TrendData-(Adults)/Device-Ownership.aspx

18. @forsmarshgroup @romanocog
18
Nielsen: The Cross-Platform Report, Quarter 2,
2012-US

19. Check Boxes in drop-down menus
@forsmarshgroup @romanocog
19

20. Radio Buttons on Mobile
@forsmarshgroup @romanocog
20

21. Useful Error Messages
@forsmarshgroup @romanocog
21

22. Error Messages Across Devices
@forsmarshgroup @romanocog
22

23. Consistency Across Devices
• Predictable
– User can anticipate what the system will do
• Dependable
– System fulfills user’s expectations
• Habit-forming
– System encourages behavior
• Transferable
– Habits in one context can transfer to another
• Natural
– Consistent with user’s knowledge
@forsmarshgroup @romanocog
23

24. Implications from Eye Tracking on a
Mobile and Desktop Tool

25. UX Testing Methodology
Regardless of the platform that the tool was tested on, participants completed tasks in this
order on Device 1, 2, and 3.
Device 1
Account
Registration
Tutorial
Homepage
Impressions
@forsmarshgroup @romanocog
25
Device 2
Device 3
Log in
Log in
1st entry
3rd entry
2nd entry
Check
Entries

26. What is beautiful is usable!
Two-thirds of participants (24 of 36) had positive feedback about the appearance of the homepage.
16
Initial Impressions of Homepage*
14
12
•
•
•
“I like the colors, and my name is up there so it’s
directed to me.”
“It’s very clean and visually pleasing. The color
choices- aqua and blue- it’s very calming.”
“It seems like icons are very big and clear on what
they are.”
10
8
6
4
2
0
Colorful/Attractive
Clean/Straightforward
@forsmarshgroup @romanocog
*Verbal responses to Moderator’s Guide Q4: Without making any clicks, can you tell me what are your initial impressions of the homepage?
26
Tractinsky, N., Katz, A. S., & Ikar, D. (2000). What is beautiful is usable. Interacting with computers, 13(2), 127-145.

27. Logical entry order
The account creation process had a logical flow from one question to the next.
•
•
•
“The questions were simple and were easy to answer.”
“I was able to figure everything out with no trouble.”
“It’s very easy and straightforward to fill out.”
40
Logical Flow Ratings*
Number of Responses
35
30
25
<=2 2
1&
20
3
3
4&
>=4 5
15
10
5
0
Participant Responses*
@forsmarshgroup @romanocog
*Device 1 Satisfaction Questionnaire Q4: Please rate the flow from one question to the next when entering your demographic and household
information. 1= not logical at all to 5= extremely logical
27

28. Entering is straightforward and easy.
The TV entry process was “not difficult at all” or “only slightly difficult.”
Entries were “moderately to extremely accurate” in capturing TV viewing behavior.
5
Average Ratings of TV Entry Process
4
3
2
1
0
TV Entry Difficulty*
•
•
•
TV Entry Accuracy**
“It’s easy to click with your finger, and the prompts are
easy to follow.”
“There were direct questions and direct options.”
“The screen is short; it's a not a multi-page app. I can
enter it all in one.”
Tablet app
Heat Maps: After being instructed to enter a TV show, fixations were
concentrated on the ‘Enter Viewing’ icon, suggesting that the
functionality of that icon is very clear.
@forsmarshgroup @romanocog
28
Android app
*Moderator’s Guide Q13 & Q25: On a scale of 1 to 5, how difficult was it to input your TV viewing entry, and why? 1 = not difficult at all to 5=
extremely difficult
**Device 2 & 3 Satisfaction Questionnaire Q6: Please rate how accurately this app/site, when used on this device, captures your actual TV
viewing . 1= not accurate at all to 5= extremely accurate

29. Unified experience from paper to mobile/desktop
Prototype Grid
Order of Entry
Paper
Website/App
1. Name of Show
2. Station Name
3. Station
Name/Number
4. Viewing Type
29
2. Name of Show
3. Channel Number
@forsmarshgroup @romanocog
1. Viewing Type
4. Time

30. Issue: Icons are not clickable and do not match user expectations.
Logo should link to the homepage, similar to the way
logos work on websites.
“I’m going to click on it, but it's not doing anything.
That is annoying.”
• “Do these not work?”
•
Homepage on the iPad
website does not display
the clickable links below
each icon.
Clicks along the bottom of the PC homepage showing that participants expected the icons to be working links.
UX Best Practice: Reduce the amount of non-clickable real estate on the homepage. Focus on
making icons and images clickable. This will help the homepage feel more intuitive as users expect
icons and images to function as active links.
@forsmarshgroup @romanocog
30

31. Issue: Error messages do not promote recovery.
Gaze Plot: After getting an error
message, the participant had to
search all over the screen to find
the missing field.
•
•
“How do I advance to the next screen?”
“It seems like it's stuck on the screen.”
UX Best Practice: Specifically explain the error, and place the messages near where the error
occurs so the user can quickly fix the error and move on.
@forsmarshgroup @romanocog
31
M

32. Issue: The location of the ‘Next’ Button is not intuitive.
Intuitive ‘Next’ button location
Non-intuitive ‘Next’ button location
• “Where and how you click is a bit counter-intuitive. [It’s] not super
obvious which button to click to get to next sections.”
• “I feel like the ‘Next’ should be at the bottom and not the top.”
UX Best Practice: Users expect consistency and for navigation buttons to be close to the last entry
field.* Place all ‘Next’ and navigation buttons near the last entry fields.
@forsmarshgroup @romanocog
* Couper, M.P., Baker, R., & Mechling, J. (2011). Placement and Design of Navigation Buttons in Web Surveys. Survey Practice,
4(1).
32

33. Issue: The ‘Back’ button confuses participants.
•
•
•
“I was expecting ‘Next,’ but I
guess I have to push
‘Back.’”
“It was hard to get
confidence that I was
proceeding in the right way.”
“It didn't seem to flow, if you
will—you had to go back and
forth.”
UX Best Practice: Make the location of the ‘Next’ button consistent across all devices by placing it
in the lower right corner. This will assure users that their selections will be saved and promote a
sense of linear progression.
@forsmarshgroup @romanocog
33

34. Issue: The iPad website keyboard blocks the screen.
10
Tablet Web Users: Least Favorite Device
Number of iPad Web Participants
9
8
7
6
5
4
3
2
1
0
Tablet
PC
Smart Phone
UX Best Practice: Design the layout of the page such that important functions are always visible
while the keyboard is open.
@forsmarshgroup @romanocog
34

35. Issue: Workload is high when searching for entries.
UX Best Practice: Follow users’ mental model: Arrange previous entries with the most recent at the
top.
@forsmarshgroup @romanocog
35

36. Assess the UX across devices
•
•
•
•
Not necessarily the same UX across all devices
Different issues occur on different devices
Match user expectations and product objectives
Conduct UX testing
@forsmarshgroup @romanocog
36

37. Thank you!
•
•
•
Twitter: @forsmarshgroup
LinkedIn: http://www.linkedin.com/company/fors-marsh-group
Blog: www.forsmarshgroup.com/index.php/blog
Jennifer Romano Bergstrom
@romanocog
jbergstrom@forsmarshgroup.com
MoDevEast