1. RESEARCH IN PRACTICE
Findings usability professionals should know about
Kath Straub, Usability.org

2. 2

3. Knowing the research
• obviates design debates
makes it unnecessary

4. Knowing the research
• obviates design debates
• makes us more effective designers

5. Research Question:
Does who we study influence
what we find?

7. Findings
If (Berkeley)preschoolers read about nutrition during snack time,
they are more likely to eat more veggies. They will also understand
more about why veggies are important (including details that were
not in the reading.)
Practitioner takeaways
"
• When research findings don’t make sense, be dubious.
Citation:
Gripshover,
S.
J.,
&
Markman,
E.
M.
(2013).
Teaching
Young
Children
a
Theory
of
NutriHon:
Conceptual
Change
and
the
PotenHal
for
Increased
Vegetable
ConsumpHon.
Psychological
Science,
24(8),
1541–1553.
doi:
10.1177/0956797612474827

8. Research Question:
How many users do we need to
test to be confident about our results?

10. Insight 1
If you don’t test anybody,
you don’t learn anything

11. Insight 2
You hit diminishing returns on finding
new usability problems at about the 5th user test

12. AssumpHons:
#
problems
found:
N(1-­‐(1-­‐λ)i)
N=
Total
#
of
usability
problems
λ
=
probability
of
finding
the
average
usability
problem
when
running
a
single,
average
subject
or
using
a
single,
average
Evaluator.
For
this
curve
λ
=
33%.
I
=
#
of
parHcipants
or
evaluators

13. But,
it
depends.

14. Findings
If you make assumptions about the % of problems an individual test
participant or evaluator will find, you can model when you will reach
diminishing returns for testing/evaluating.
Practitioner Takeaways
• If the “takeaway” is that simple, you should probably re-evaluate
for yourself.
Citation: Faulkner, L. (2003). Beyond the five-user assumption: benefits of increased sample sizes in usability testing.
Behavior Research Methods, Instruments, & Computers : A Journal of the Psychonomic Society, Inc, 35(3), 379–83.

16. 5 participants 10 participants 15 participants

17. Findings
With 5 participants, you find 85% the problems on average, but the
range is 55%-98%. When you test 10, its 94%/82%. For 15, its
97% /90%.
Practitioner Takeaways
• Target to test 10-12 participants, not 5.
Citation: Faulkner, L. (2003). Beyond the five-user assumption: benefits of increased sample sizes in usability testing.
Behavior Research Methods, Instruments, & Computers : A Journal of the Psychonomic Society, Inc, 35(3), 379–83.

18. Research Question:
Does how we test influence
the feedback we get?

19. or

20. Findings
When you show more than one design to a usability testing
participant, they can make contrastive comments.
Testing multiple designs leads to more detailed, comparative and
negative comments from participants
Takeaways
• Show participants more than one design during usability tests
and allow them to compare/contrast the user experiences.
Citation: Tohidi, Maryam., Buxton, William., Baecker, Ronald., and Sellen, Abigail. . (2006) Getting the Right Design and
the Design Right: Testing Many is Better Than One CHI 2006 Proceedings. Usability Methods. April 22-27, 2006.
Montreal, Quebec, Canada.

22. Research Question:
Does how we test influence
the feedback we get-MOBILE?

24. Findings
Treadmill tests and controlled walking tasks yield similar
performance in mobile evaluations.
Stopping/restarting is not ecologically valid on a treadmill.
Takeaways
Controlled walking tasks provide a better in-context evaluation.
• Feedback given
Citation: Barnard, L., Yi, J. S., Jacko, J. A., & Sears, A. (2005). An empirical comparison of use-in-motion evaluation
scenarios for mobile computing devices. International Journal of Human-Computer Studies, 62(4), 487–520. doi:10.1016/
j.ijhcs.2004.12.002

25. Research Question:
Does how we get feedback
influence our design creativity?

27. Findings
Parallel prototyping leads to more effective designs and greater
designer self efficacy
Takeaways
• Feedback given on a single design tends to lock designers into a
path or set. Create/Request feedback on more than one design
• Parallel prototyping promotes comparison without
defensiveness: Forces designers to think of alternative designs so
you don’t get attached to one early in the design process
Citation: Dow, S., Glassco, A., & Kass, J. (2011). The effect of parallel prototyping on design performance, learning, and
self-efficacy. ACM Conference on …, (September), 10. doi:10.1111/j.1432-2277.2009.00960.x

28. Research Question:
How should we align labels?

29. 29

30. 30
Fixations on "
whitespace

32. Findings
People liked left- and right-aligned forms best and rate them easiest
to fill out.
People were most efficient (fewest fixations) on top-aligned forms.
People looked longest at top- and in-field aligned forms.
In-field alignment was rated most difficult.
Flow was rated least appealing.
Practitioner takeaways:
This really doesn’t matter as much as we want to argue about it.
But, consider what people are doing and using when you pick.
"
Citation: Bojko,
A.
A.,
&
Schumacher,
R.
M.
(n.d.).
Eye
Tracking
and
Usability
TesHng
in
Form
Layout
EvaluaHon,
1–13.
"

34. Research Question:
How much instruction should we give
in forms?

35. 35

36. 36

37. 37
No instructions
Examples
Description
Description
+ Examples

38. 38
No instructions
Examples
Description
Description
+ Examples
Findings
Participants made the most errors with no instructions.
Descriptions resulted in fewer errors than examples.
Giving descriptions and examples doesn’t help (significantly),
but it doesn’t hurt either.
Practitioner takeaways:
Format examples are better than nothing.
Descriptions work better than examples.
Citation: Bargas-Avila, J., Orsini,S., Piosczyk, H., Urwyler, D., Opwis, K. (2011) Enhancing online forms: Use format
specifications for fields with format restrictions to help respondents. Interacting with Computers, 23(1).

39. Research Question:
How should you present instructions?

41. Visible
instrucHons
Hidden
Behind
Bueon
Hidden
Behind
Link

42. Findings - How to present instructions
• “Expert” users and older users interact with instructions equally
frequently whether they are hidden or not.
• Low literacy participants tended not to interact with instructions
even when they need to (They didn’t help much, when they did.)
• Links draw more clicks that instruction icons.
Practitioner Takeaways
• Provide visible instructions.
• If you must hide them, use links rather than icons to let users
know they exist.
"
Reference: Alton,
N.,
Rinn,
C.
,
Summers,
K.,
and
Straub,
K.,
(Forthcoming)
Using
Eye-­‐Tracking
and
Form
CompleHon
Data
to
OpHmize
Form
InstrucHons.
IEEE
–
IPPC
14.
Piesburgh,
PA
USA.

43. Top
of
SecHon
Above
Field
Right
of
Field

44. Findings - Where to put instructions
• Low literacy people and older participants made significantly
more errors and missed more fields in the accordion
presentation.
• Participants tended not to read page headers in the wizard.
Practitioner Takeaways
• Place instructions above text input field.
"
Reference: Alton,
N.,
Rinn,
C.
,
Summers,
K.,
and
Straub,
K.,
(Forthcoming)
Using
Eye-­‐Tracking
and
Form
CompleHon
Data
to
OpHmize
Form
InstrucHons.
IEEE
–
IPPC
14.
Piesburgh,
PA
USA.

45. Research Question:
How should you present long forms?

48. Findings - How to segment and present forms
• Low literacy users and older participants, and even expert users
(to a lesser extent) miss more items in accordion designs.
• Wizards help lower literacy and older individuals.
Practitioner Takeaways
• Avoid accordion presentations for presenting long forms.
"
Reference: Alton,
N.,
Rinn,
C.
,
Summers,
K.,
and
Straub,
K.,
(Forthcoming)
Using
Eye-­‐Tracking
and
Form
CompleHon
Data
to
OpHmize
Form
InstrucHons.
IEEE
–
IPPC
14.
Piesburgh,
PA
USA.

49. Research Question:
Does the hamburger menu work?

52. Findings
• Mobile users click more frequently on an icon that says MENU.
Practitioner Takeaways
• Use words when you can.
"
Reference:
Test 1: http://exisweb.net/mobile-menu-abtest
Test 2: http://exisweb.net/menu-eats-hamburger

53. Research Question:
Can plain language be credible and
concise?

54.
The
trial
court
erred
in
giving
flawed
essenHal
elements
instrucHons
to
the
jury
and
thereby
denied
the
defendant
due
process
and
fundamental
fairness
since
it
is
error
to
give
the
jury,
within
the
essenHal
elements
instrucHons,
one
statement
containing
more
than
one
essenHal
element
of
the
crime
and
requiring
of
the
jury
simple
and
singular
assent
or
denial
of
that
compound
proposiHon,
fully
capable
of
disjuncHve
answer,
which
if
found
pursuant
to
the
evidence
adduced
would
exculpate
the
defendant.
The
trial
judge
erred
by
instrucHng
the
jury
to
affirm
or
deny
a
single
quesHon.
That
quesHon
included
all
the
major
elements
of
the
crime.
By
joining
all
the
major
elements,
the
judge
prevented
the
jury
from
acquinng
the
defendant
even
if
they
found
him
innocent
of
a
major
element
.This
error
denied
him
his
due-­‐
process
rights.

55.
The
trial
court
erred
in
giving
flawed
essenHal
elements
instrucHons
to
the
jury
and
thereby
denied
the
defendant
due
process
and
fundamental
fairness
since
it
is
error
to
give
the
jury,
within
the
essenHal
elements
instrucHons,
one
statement
containing
more
than
one
essenHal
element
of
the
crime
and
requiring
of
the
jury
simple
and
singular
assent
or
denial
of
that
compound
proposiHon,
fully
capable
of
disjuncHve
answer,
which
if
found
pursuant
to
the
evidence
adduced
would
exculpate
the
defendant.
The
trial
judge
erred
by
instrucHng
the
jury
to
affirm
or
deny
a
single
quesHon.
That
quesHon
included
all
the
major
elements
of
the
crime.
By
joining
all
the
major
elements,
the
judge
prevented
the
jury
from
acquinng
the
defendant
even
if
they
found
him
innocent
of
a
major
element
.This
error
denied
him
his
due-­‐
process
rights.
Findings
• The plain language lawyer was rated easy to understand,
trustworthy, logical and specific and concise by other lawyers.
• Lawyers and non-lawyer thought clients, juries and peers would
understand him better
• Lawyers would be more satisfied with the plain language lawyer
as their lawyer
Practitioner Takeaways
• Plain language enhances credibility, even for professionals.
"
Reference:
Straub,
K.,
Mahaffey,
S.
and
Cheek,
A.
(Forthcoming).
Even
lawyers
want
to
understand
their
lawyers:
New
evidence
showing
plain
language
increases
lawyers’
credibility.
To
be
presented
at
Clarity
2014,
Antwerp,
Belgium.

56. Research Question:
Is infinite scrolling evil?

57. Homework:
hep://www.slideshare.net/danmckinley/design-­‐for-­‐conHnuous-­‐experimentaHon

58. Homework:
hep://www.slideshare.net/danmckinley/design-­‐for-­‐conHnuous-­‐experimentaHon

59. Research Question:
Are infographics all that?

60. Aqer
this
recent
elecHon,
has
the
ALP
party
become
the
second
biggest
or
second
smallest
party?

65. Standard
accurate
(abstract)
Standard
inaccurate
Non-standard
accurate
Non-standard
inaccurate

66. Findings
• Graphic designers prefer non-standard and pictorial
visualizations.
• Laypeople prefer the abstract/accurate visualizations.
• Standard and abstract visualizations reduce interpretive
response times.
Practitioner Takeaways
• If understanding your visuals are important to your story, test
them
Reference Petrie, H., & Power, C. (2012). Quispel, A., & Maes, A. (2014). Would you prefer pie or cupcakes? Preferences
for data visualization designs of professionals and laypeople in graphic design. Journal of Visual Languages & Computing,
25(2), 107–116.

67. Research Question:
Do UXers fixate (on things
users don’t care about)?

73. Findings
• There is some overlap but also some non-overlap in the issues
users focus on and the issues UXers focus on
Practitioner Takeaways
• Be skeptical of your intuitions.
"
Reference:
Petrie,
H.,
&
Power,
C.
(2012).
What
do
users
really
care
about?
In
Proceedings
of
the
2012
ACM
annual
conference
on
Human
Factors
in
Compu@ng
Systems
-­‐
CHI
’12
(p.
2107).
ACM
Press.
doi:10.1145/2207676.2208363

74. Research Question:
Are those cool new UX research
methods something we should be
planning for?

75. JUST
BECAUSE
ITS
SEXY
DATA,
DOESN’T
MEAN
ITS
USEFUL

76. JUST
BECAUSE
ITS
SEXY
DATA,
DOESN’T
MEAN
ITS
USEFUL
Findings
• (Dead) Salmon respond differentially to human emotions
Practitioner Takeaways
• Be skeptical, particularly of new/emerging/sexy/neuroscienc-y
researchy methods.
"
Reference
Bennee,
Baird,
Miller,
and
Wolford
(2009)
Neural
correlates
of
interspecies
perspecHve
taking
in
the
post-­‐
mortem
AtlanHc
Salmon:
An
argument
for
mulHple
comparisons
correcHon.