On a scale of 1-10, how much do you love this workshop? Ok, hopefully that is an obviously bad question, both because it hasn't happened yet and because it has some bias baked right in. But take a quick look around all the surveys floating out in the world, and they often don't seem much better. Surveys can be a powerful tool for a UX researcher, but many of us haven't learned how to get the most out of them. In this workshop we'll cover: Best use cases for surveys (and when to avoid them) An overview of question types Guidelines for writing effective, unbiased survey questions Tips to increase overall engagement and participation Hands on practice crafting surveys Basic survey analysis

1. Hello, Triangle!

2. Effective use of surveys in UX

3. @MandaLaceyS
Survey Overview

4. Survey
“ a systematic method for gathering
information from a sample of
entities for the purpose of
constructing quantitative
descriptors of the attributes of the
larger population fo which the
entities are members”
-The Interwebs

5. IRL UX Surveys*
“ a way to get a whole lot of
responses to a series of questions to
help inform decisions”

6. Surveying is
the process.
A
questionnaire
is the
document with
questions.

7. Qualitative
Quantitative

8. Qualitative
Quantitative

9. Data that is
descriptive,
but not
necessarily
measurable.
Why?
How?
Where?
Qualitative
Data that can be
measured with
counts/
numbers.
How many?
How much?
What size?
What rating?
Quantitative
v.

10. GET 👏 YOU 👏 A 👏 GIRL👏 WHO
👏 CAN 👏 DO👏 BOTH!!!

11. When to use surveys

12. Assess
✨expressed✨
preferences,
attitudes, and
intentions
at scale.
Survey Best
Uses

13. Comparing those
expressed
beliefs over time
or getting a scale
of how widely
shared.
Survey Best
Uses

14. …When you can
rely on self-
reported
information.
Survey Best Uses

15. …everything else.
Worst Uses

16. Worst Uses

17. ❌
• When you want to
identify quanti
fi
able
trends
• When you have already
identi
fi
ed assumptions
and want to get a
sense of scale
✅ v.
• When you need to
observe behavior
• When you need to
explore reasonings
behind decisions
• When you *could* get
the data another
way

18. Start with the goal

19. Identify:
• What do you want to know?
• Why do you want to know that?
• What decisions will you make?
• How will you know you have enough/right
kind of data?

20. EXERCISE

21. ⏲ 10 minutes
Imagine you work at an animal rescue nonpro
fi
t.
What sort of things might you want to know?
Why would that be useful?
What decisions might you make?

22. Share time.

23. @MandaLaceyS
Define who to include

25. You don’t
need formal
personas to
de
fi
ne targets.

26. At a
minimum,
de
fi
ne who
you don’t
want to talk
to.

27. Make it representative.
😬

28. Image Credit Caroline Jarret
Remember biases and the
“Zone of Indifference”

29. Determine sample size

30. Sample size:
how many people
to include/the
number of
responses you
want to get

31. Sample size =
(z) ² x d x (1-d)
(e) ²
OMG
MATH?!?!

32. Sample size =
(z) ² x d x (1-d)
(e) ²

33. Sample size =
(z) ² x d x (1-d)
(e) ²
e = margin of error/con
fi
dence
interval = how much error you feel
comfortable baking in to your study
(expressed as plus or minus
fi
gure)

34. Sample size =
(z) ² x d x (1-d)
(e) ²
d = standard deviation = estimated
variation of responses (expressed
as a decimal)

35. Sample size =
(z) ² x d x (1-d)
(e) ²
d = standard deviation = probably
just put .5 because if you knew you
might not need to run a survey

36. Sample size =
(z) ² x d x (1-d)
(e) ²
z = z score = how many standard
deviations you are away from the
mean (look it up based on
con
fi
dence level)

38. PRO TIPS
1) Decrease confidence
level
2) Increase margin of error
3) Realize it’s probably ok
not to be statistically
significant in a UX context.
4) Be pragmatic.
PRO TIPS

39. Lunch break

40. @MandaLaceyS
Common Question Types

41. @MandaLaceyS
Closed

42. Use when:
• You have a well-de
fi
ned
set of options
• There really can only be
one answer.
Multiple Choice
Select One/Radio button/Dropdown

43. Use when:
• You have a well-de
fi
ned
set of options
• There are many
potential answers.
Multiple Choice
Select Many/Checkbox

44. Use when:
• You have some sort of
linear rating that makes
sense, such as how
much someone agrees/
disagrees.
Rating Scale

45. Use when:
• You want to have
participants rate several
items on the same scale.
Matrix**
** Please use sparingly. People hate these.

46. Use when:
• You have a well-de
fi
ned
list of items to prioritize.
Ranking Scale

47. @MandaLaceyS
Open Ended

48. Use when:
• You don’t have well-
de
fi
ned potential
responses.
• You want to gather
qualitative feedback.
• You’re OK to spend
some time tagging/
analyzing data.
Text Box

49. @MandaLaceyS
Craft effective
survey questions

50. Let me tell
you a secret.

51. Most surveys
suck.

52. Set your goal
Set a specific, targeted goal,
including hypotheses when you can.

53. Make sure to think
through…
• What you want to learn, including
decisions you want to make with
this effort and what sort of data you’ll
need to make those decisions
• What other data you might need to
tie this data to
• What visualizations or stats you
might want to represent
• What can you cut/learn somewhere
else?

54. Ask precise questions.
Which snack is most delicious and nutritious?
Apple
Cookie
Chips
Carrot
Brownie
Cheese

55. Ask precise questions.
Which snack is most delicious and nutritious?
Apple
Cookie
Chips
Carrot
Brownie
Cheese
Probably most nutritious.

56. Ask precise questions.
Which snack is most delicious and nutritious?
But most delicious?
Apple
Cookie
Chips
Carrot
Brownie
Cheese

57. How often do you take a shower?
Every day
Every other day
Every 2-3 days
Every 3-4 days
Every 5-6 days
Once a week or less
With precise answers.

58. How often do you take a shower?
Every day
Every other day
Every 2-3 days
Every 3-4 days
Every 5-6 days
Once a week or less
What if I shower every 3 days?
With precise answers.

59. How often do you take a shower?
Every day
Every other day
Every 2-3 days
Every 3-4 days
Every 5-6 days
Once a week or less
(Promise, I showered today)
With precise answers.

60. How often do you exercise?
Not that much
Sometimes
Often
Pretty often
Really often
Super often
With precise answers.

61. How often do you exercise?
Not that much
Sometimes
Often
Pretty often
Really often
Super often
WTF does often mean?
With precise answers.

62. How often do you exercise?
Every day
A few times a week
Once a week
BETTER
A few times a month
Once a month or less
I don’t exercise
With precise answers.

63. Which of the following best describes you?
Sassy
Classy
Bada**-y
Make sure there are
appropriate answers.

64. Because what if…

66. Make it easy to answer.
What is the exact date and time that you
bought your last Apple device?
I have no clue

67. Make it easy to answer.
Have you bought an Apple device in the last
month?
Yes
No

68. But. Not too easy.
Do you want the incentive?
Yes
No

69. Try something like this instead.
Which of the following have you purchased in
the last month?
Mobile phone
Tablet
Television
Laptop
Desktop
None of the above

70. You can also have multiple “right” answers.
How many Apple devices do you own?
0
1
2
3
4
5 or more

71. You can also have multiple “right” answers.
Maybe I want to talk to anyone who has 3 or more.
How many Apple devices do you own?
0
1
2
3
4
5 or more

72. EXERCISE

73. ⏲ 10 minutes
Thinking of that example animal rescue nonpro
fi
t
core goal you determined earlier.
1) Identify the right kind of questions to answer
those goal
s
2) Write 3 questions that map to the core
respondent needs; understand,
fi
nd, decide, and
plac
e

74. Share time.

75. Let’s talk scales…

76. Use when:
• You want to see how
much someone
agrees with a
statement
• The responses don’t need
to be polar opposites.
Likert Scales
Measures agreement with a statement

77. Use when:
• The pair of objectives
are truly opposite.
• Respondents will be
able to easily
understand the
continuum.
Semantic Scales
Measures response on a continuum

78. Use when:
• There aren’t
necessarily two
opposite ends to
measure.
Unipolar Scales
Measures response on a non-polar scale

79. You can be
fl
exible!

80. Either way…
• Include a neutral point
and equal negative
and positive responses
• Aim for 5-7 options
• Label each option

81. Either way…
• Consider rephrasing to
a neutral (how much
do you agree or
disagree?)
• Have a NA option
• Include a “Please
explain…”

82. EXERCISE

83. ⏲ 5 minutes
Let’s write appropriate scaling questions for the
following contexts:
1) Understanding perceived ease of us
e
2) Asking for a participants’ level of agreemen
t
3) Gauging importance of a particular feature

84. Share time.

85. ⏲ 5 minutes
Let’s write appropriate scaling questions for the
following contexts:
1) Understanding perceived ease of us
e
2) Asking for a participants’ level of agreemen
t
3) Gauging importance of a particular feature

86. Adapted from Caroline Jarrett, adapted from “The Psychology of Survey Response”
In summary…

87. PRO TIPS
PRO TIPS

88. Introduce yourself.

89. Provide contextual instruction.

90. Simplify language.

91. Sequence Matters
Prioritize.
Order Meticulously

92. Break things up carefully.

93. Sequence Matters
Prioritize.

94. Branch with care.

95. Branch with care.
Show progress.

96. Ask why?

97. EXERCISE

98. ⏲ 5 minutes
Think about those questions
…
How would you order them?
Is there anything you could cut
?
What instructions should you include?

99. Share time.

100. TEST. PLEASE.

101. Stretch break

102. Choosing tools.

104. Which one is best?
¯_(ツ)_/
¯

105. Finding participants.

106. Recruiting
can be
tough.

107. You
have
options!

108. Existing
users

109. Digital intercepts
@MandaLaceyS

110. Beware the Halo effect.

111. Make your own panel

112. Tool Panel

113. Dedicated
Panel

114. Which one is best?

115. Which one is best?
¯_(ツ)_/
¯

116. Be careful.

117. PRO TIPS
Try including
some gotcha
questions.

118. Put some time into
invitations.

119. @MandaLaceyS
Incentives

120. Money money money
Incentives
Give something to get something.

121. Money money money
What would your users value?

122. Money money money
@MandaLaceyS
OK fine, give them cash.

123. Money money money
@MandaLaceyS
How much?

124. People like feeling important.

125. In short:
1. Set clear expectations.
2. Value people’s time and effort
appropriately.
3. Explore non-cash incentives.
4. Show participants how their feedback
makes an impact.
5. Experiment and switch up incentives.

126. Analyzing Surveys

127. Revisit your goals
The better your goals, the easier it
will be to analyze. Let the decisions
you need to make determine how
you analyze.

128. Dive in…

129. Make a copy!

130. Scrub data

131. Start with basic counts
Start with basic counts

132. Compare

133. Try lots of
different
visualizations.

134. Or cheat!

135. What about those open
responses…

136. Tagging/Coding

137. Organize. Reorganize.

138. Keep organizing.

139. Craft your story

140. Craft your story
Spread the word.

141. Thank you!
@MandaLaceyS
amanda@stockwellstrategy.com

142. Survey Resources
Surveys that Work, Caroline Jarrett, https://
rosenfeldmedia.com/books/surveys-that-work/