Are you looking to expand your research toolkit to include some quantitative methods, such as survey research or A/B testing? Have you been asked to collect some usability metrics, but aren’t sure how best to go about that? Or do you just want to be more aware of all of the UX research possibilities? If your answer to any of those questions is yes, then this session is for you. You may know that without statistics, you won’t know if A is really better than B, if users are truly more satisfied with your new site than with your old one, or which changes to your site have actually impacted conversion rates. However, statistics can also help you figure out how to report satisfaction and other metrics you collect during usability tests. And they’re essential for making sense of the results of quantitative usability tests. This session will focus on the statistical concepts that are most useful for UX researchers. It won’t make you a quant, but it will give you a good grounding in quantitative methods and reporting. (For example, you will learn what a margin of error is, how to report quantitative data collected during a usability test - and how not to - and how many people you really need to fill out a survey.)

1. Statistics for UX Professionals
Jessica Cameron
UX Scotland
Thursday, 14 June 2017
1

2. Presentation title / Footer text 2Photo by Lane Jackman on Unsplash

3. 3
Statistics for UX Professionals
Featuring more mean words
per slide than any other talk
you will see this week

4. 4
In the next 60 minutes, you will learn:
1. Useful statistical concepts, like what a confidence
interval is and how a margin of error is kind of the
same thing as a confidence interval, but not
completely
2. How many people you really need to fill out a survey
3. What kinds of quantitative data may be collected
during a usability test
4. How to report quantitative data collected during a
usability test
5. How not to report quantitative data collected during a
usability test

5. Presentation title / Footer text 5Photo by Roman Mager on Unsplash

6. 6Photo by Carlos Muza on Unsplash
Useful statistical concepts

7. 7
Why use statistics?
• Statistics let us estimate what everyone might do (a
population) by looking at what some people do (a
sample)
• Do not use statistics if you have access to the full
population (for example, all members of a project
team)

8. 8
Key concepts
• Central tendency and spread
• Mean
• Standard deviation
• Outlier
• Confidence interval
• Confidence level
• Margin of error

9. 9
Central tendency and spread
• Central tendency refers to an average, middle, or
typical value
• Spread refers to how spread out or squeezed
together a set of values are (relative to the central
tendency)
• Example (1 = not at all satisfied, 5 = very satisfied):
• Sample 1: 100 people rate a website 1, 100 people rate it 5
• Sample 2: 200 people rate a website 3
• How do the central tendency and spread differ?

10. 10
Mean
• A mean is a measure of central tendency
• The mean is the sum of all measurements divided by
the number of measurements
• Also known as the average or arithmetic mean
• Can be computed in Excel using the AVERAGE
function
• A sample mean is an estimate of the true population
mean

11. 11
How much do you like or not like spaghetti? (n=75)
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

12. 12
MEAN = 5.87
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

13. 13
Standard deviation
• A measure of spread
• To find the standard deviation, find the square root of
the average of the squared differences between each
measurement and the mean
• Or use the STDEV function in Excel
• If the standard deviation is low, the measurements
are grouped close to the mean – and if it is high, they
are further apart
• If data are normally distributed, 95% of
measurements can be expected to fall within 2
standard deviations of the mean

14. 14
Standard deviation

15. 15
How much do you like or not like spaghetti? (n=75)
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

16. 16
Mean = 5.87
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

17. 17
1 standard deviation = 3.29
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

18. 18
2 standard deviations = 6.58
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

19. 19
How much do you like or not like spaghetti?
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

20. 20
Mean = 6.85
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

21. 21
1 standard deviation = 2.58
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

22. 22
2 standard deviations = 2.58
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

23. 24
Outlier
• Any measurement that falls more than 2 standard
deviations from the mean
• May also be an unrealistic measurement (for
example, time spent on a task that is shorter than the
minimum time required by an expert user)
• Might represent valid, if extreme, data – such as an
important subgroup of users
• If 95% of measurements are within 2 standard
deviations of the mean, we would expect 5% of the
sample to be outliers

24. 25
Any outliers here?
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti
What should we do with them?

25. 27
Confidence interval
• A measure of spread and sample size
• A confidence interval is a range of values that we can
say with a certain level of confidence contains the
true population mean
• This true population mean is the one we would find if
we were able to have all possible users perform and/
or rate a task
• Use the CONFIDENCE.T function in Excel
• (Use alpha = 0.05, which corresponds to a 95% level
of confidence)

26. 28
95% CI (n=75): 6.26, 7.45
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

27. 29
What determines the size of the
interval?
• Standard deviation: The larger the standard deviation,
the more spread out data are and the larger the
margin of error will be
• Sample size: The larger the sample size, the more
reliable measurements are and the smaller the
confidence interval will be

28. 30
95% CI (n=75): 6.26, 7.45
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

29. 31
95% CI (n=375): 6.59, 7.11
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

30. 32
What determines the size of the
interval?
• Standard deviation: The larger the standard deviation,
the more spread out data are and the larger the
margin of error will be
• Sample size: The larger the sample size, the more
reliable measurements are and the smaller the
confidence interval will be
• Confidence level: The higher the level of confidence
need, the larger the confidence interval will have to
be

31. 33
Confidence level
• Confidence intervals are usually reported with a 95%
level of confidence
• That means that if you repeated a survey 100 times,
you would expect the mean response you get to fall
within your confidence interval 95 times
• You can be 95% confident that the true population
mean is within this interval as well
• 90% and 99% are also commonly accepted – and the
higher the level of confidence, the larger the resulting
confidence interval will be

32. 34
95% CI (n=375): 6.59, 7.11
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

33. 35
95% CI (n=375): 6.51, 7.20
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

34. 38
Confidence interval – how to report
Wrong:
• We are 95% confident that the population mean is 6.85
Right:
• We can say with 95% confidence that the true population
mean lies between 6.51 and 7.20
What is the difference between a confidence interval and
a margin of error?

35. 40
Margin of error
• A margin of error is half of a confidence interval
• Excel actually gives us the margin of error, which we
then use to compute the confidence interval
• A margin of error is usually half of a 95% confidence
interval, with the confidence level implied
• (That means that it is easier to talk about margin of
error than confidence interval)
• Margin of error is often used for poll numbers or
preferences, and is expressed in percentage points

36. 41
Margin of error notes
• The ‘error’ refers to random sampling error
• It does not account for other errors:
• Systematic sampling errors
• Errors in survey design (e.g., biased or unclear questions)
• Each measurement has a margin of error associated
with it
• Poll shows Trump at 17% and a bag of hammers at 81%, with
a +/- 3 point margin of error
• Trump could really get anywhere between 14 and 20% of the
vote, and the bag of hammers could get anywhere between
78% and 84% of the vote

37. 44
Recap of key concepts
• Central tendency and spread
• Mean
• Standard deviation
• Outlier
• Confidence interval
• Confidence level
• Margin of error

38. 45
Statistics for surveys
Photo by Ranier Ridao on Unsplash

39. 46
How many survey participants?
(Short answer)
400

40. 47
How many survey participants?
(Long answer)
• It depends on how confident you want to be in the
findings (e.g., what size confidence interval are you
comfortable with?)
• It depends on who your audience is
• It depends on the size of the population you are
drawing your sample from

41. 48
How many participants?
(Long answer)
• Examples of population sizes:
• Everyone in the UK = 60 million
• Topshop Oxford Street shoppers = 200,000 per week
• Topshop.com shoppers = 4.5 million per week
• Topshop employees = 10,000
• A ‘large’ population is anything over about 20,000
• Most surveys of existing (or potential) customers will
draw from a large population

42. 49
Confidence level = 95%
+/- 5
Margin of error
+/- 3 +/- 1
Population size
100 80 92 99
500 217 341 475
1,000 278 517 906
10,000 370 965 4,899
20,000 377 1,014 6,489
50,000 382 1,045 8,057
100,000 383 1,056 8,762
500,000 384 1,065 9,423
1,000,000 384 1,066 9,512

43. 50
95% CI (n=75): 6.26, 7.45
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

44. 51
95% CI (n=375): 6.59, 7.11
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

45. 52
Collecting vs. comparing data
• Surveys typically collect satisfaction data for an
existing website at a given time
• For that, mean plus margin of error (or confidence
interval) is an effective way of communicating results
• But what if you are comparing two sets of data?
• Satisfaction ratings before and after a launch
• Conversion rates on versions A and B of a site
• Then you will need a proper statistical test

46. 53
Use a t-test
• To compare two samples, use a two sample t-test
• A t-test will tell you if the populations that those
samples come from are the same or different (to a
certain level of confidence)
• It does this by considering the difference between the
means relative to the size of the variance, or spread
• Not enough to see if confidence levels overlap
• That would be an overly conservative test
• So instead of eyeballing the data, run a t-test

47. 54
How to run a t-test
• Use the T.TEST function in Excel – need to know
arrays, tails and type
• The two arrays are the two data sets
• The number of tails depends on what you are predicting
• For type, you can generally assume equal variances (type =
2)
• Result is a probability that the samples come from the
same population
• If it is less than .05, you can say that the populations
are different

48. 55
Mean = 6.85
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

49. 57
T-test (N=150): p = .043
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

50. 58Photo by Sandeep Swarnkar on Unsplash

51. 59
Rules of thumb
• At a 95% confidence level:
• A sample size of 400 will yield a margin of error of +/- 5%
• A sample size of 1,000 will yield a margin of error of +/- 3%
• If you are serious enough about your research to use
the recommended number of participants, then you
should take your data seriously
• Report central tendency and spread
• Look for statistically significant results

52. Statistics for user testing
60Photo by Rob Hampson on Unsplash

53. 61
Data collected during usability tests
Qualitative data:
• Observations of ease or difficulty participants have
completing tasks
• What we think about when we think about usability
testing
Quantitative data:
• Success/ failure/ disaster rates (effectiveness)
• Task completion time (efficiency)
• User surveys (satisfaction)

54. 62
Satisfaction data collected during
usability tests
• Do you collect satisfaction ratings (or any other scale
ratings) from usability test participants?
• How many people do you usually run during a single
usability test?
• How do you report those data?
• How should you report those data?

55. 63
Asking about satisfaction
• Questions asked during usability testing can tell us a
great deal about the experience that our participants
are having on a website
• They can tell us basically nothing about the
experience that all users have on a website

56. 64
Remember confidence intervals?
• Your sample size is 6
• Three testers (50%) say they would use the website
again
• You can say with 95% confidence that the true
percentage of the population who would use this
website again is somewhere between…
• 10% and 90%

57. 65
So be careful
• Be careful not only with what you choose to ask…
• But how you choose to report the data
• Graphs and charts can be very compelling…
• But so can qualitative data
https://finickypenguin.wordpress.com/category/nerd-
humor/page/3/

58. 66
Takeaways
• Reporting confidence intervals along with means
gives everyone more confidence in your findings
• Try to get 400 people to fill out your surveys
• You can use Excel to run simple statistical tests, but if
you need something more complicated try R or Stata
• Statistics are not the answer to everything…
• But if you choose to use them, they can expand your
UX toolkit nicely

59. Presentation title / Footer text 67Photo by Morvanic Lee on Unsplash
Jessica Cameron
@jessscameron
User Vision
@UserVision