Rolf Molich owns and manages DialogDesign, a small Danish usability consultancy that he founded in 1993. Rolf conceived and coordinated the Comparative Usability Evaluation studies in which more than 100 professional usability teams tested or reviewed the same applications. Rolf was a principal investigator in the Nielsen Norman Group’s large-scale usability test of 20 US e-commerce websites, involving more than 60 users. He has worked with usability since 1984 and wrote the best-selling Danish book User Friendly Computer Systems, of which roughly 30,000 copies have been sold. The book is now available in English, with the title Usable Web Design. Rolf is also the co-inventor of the heuristic evaluation method (with Jakob Nielsen). Rolf is an experienced speaker. His 24 tutorials at the Nielsen Norman Group World Tour attracted roughly 1,000 participants. The overall average participant evaluation of his tutorials was 4.46 on a five-point scale.

1. Myths about usability testing Copies of Slides
”Five users will find 85% of the usability problems”
- and other myths about usability testing
November 9, 2012
Rolf Molich, DialogDesign (www.DialogDesign.dk)
This document and its content is Copyright ©2012 Rolf Molich and UCD UK Limited.
Supporters Sponsors Organiser
1
UCD 2012, London Rolf Molich, DialogDesign Page 1

2. Myths about usability testing Copies of Slides
Dogma 1
► Usability professionals must be highly usable
Reports, presentations, responses to comments, etc.
3
When you point your finger at someone, remember that three fingers point at yourself
UCD 2012, London Rolf Molich, DialogDesign Page 2

3. Myths about usability testing Copies of Slides
Sweep before
your own door
before you sweep
the doorsteps of
your neighbors
Excerpt from Usability Test Report - CUE-6, Team G
UCD 2012, London Rolf Molich, DialogDesign Page 3

4. Myths about usability testing Copies of Slides
Excerpt from Usability Test Report - CUE-6, Team G
Thank you for your attention! Any questions?
Contact: Rolf Molich, DialogDesign, molich@DialogDesign.dk
Picture credit: Rob Gonsalves – ”The Sun Sets Sail”
UCD 2012, London Rolf Molich, DialogDesign Page 4

5. Myths about usability testing Copies of Slides
Dogma 2
► Data trumps opinion at any time
9
A Good Old American Bake-off
UCD 2012, London Rolf Molich, DialogDesign Page 5

6. Myths about usability testing Copies of Slides
A Usability Bake-off
CUE-9:
19 Atlanta +16 Chemnitz
Professionals
5 Videos
>100 Different
Findings
UCD 2012, London Rolf Molich, DialogDesign Page 6

7. Myths about usability testing Copies of Slides
Data to Fight
Unsubstantiated
Myths and Voodoo
-- CUE motto
Please use a few minutes to mark your position on each of the above 5 statements
UCD 2012, London Rolf Molich, DialogDesign Page 7

8. Myths about usability testing Copies of Slides
Five users are enough to catch
1
85% of the usability problems in
practically any product
Agree
Both agree & disagree
Disagree
Truth or myth?
Five users are enough to catch
1
85% of the usability problems in
practically any product
Agree
Both agree & disagree
Disagree
Truth or myth?
UCD 2012, London Rolf Molich, DialogDesign Page 8

9. Myths about usability testing Copies of Slides
”The parabola of optimism” (Jared Spool)
Real data: CUE studies
CUE = Comparative Usability Evaluation
► 10-17 professional teams test the same website.
► In all CUE-studies, more than 60% of problems
were uniquely reported.
► In CUE-4, more than 300 issues were reported.
18
UCD 2012, London Rolf Molich, DialogDesign Page 9

10. Myths about usability testing Copies of Slides
Five users are
enough to drive a
useful iterative
cycle
The main goal of a usability test is
2
to discover usability problems
Agree
Both agree & disagree
Disagree
Truth or myth?
UCD 2012, London Rolf Molich, DialogDesign Page 10

11. Myths about usability testing Copies of Slides
The main goal of a usability test is
2
to discover usability problems
Agree
Both agree & disagree
Disagree
Truth or myth?
Real cowboy
programmers don’t
need no stinkin’
usability test
-- Anonymous
UCD 2012, London Rolf Molich, DialogDesign Page 11

12. Myths about usability testing Copies of Slides
Goals of usability test
1. Find problems so they can be corrected
2. Demonstrate convincingly to skeptical
stakeholders (developers, management)
that problems exist and that there are
efficient ways to find and correct them
23
Expert reviews provide results that
3
are as reliable as those from
usability tests
Agree
Both agree & disagree
Disagree
Truth or myth?
UCD 2012, London Rolf Molich, DialogDesign Page 12

13. Myths about usability testing Copies of Slides
Expert reviews provide results that
3
are as reliable as those from
usability tests
Agree
Both agree & disagree
Disagree
Truth or myth?
Expert review vs. Utest
Data from CUE-studies, most notably CUE-4,
consistently shows
► Expert reviews find as many problems as
usability testing
► Expert reviews overlook problems –
but so do usability tests
► Expert reviews are cheaper than usability tests
These findings apply only when expert reviews
are carried out by Experts
26
UCD 2012, London Rolf Molich, DialogDesign Page 13

14. Myths about usability testing Copies of Slides
Beware!
Expert reviews can be political dynamite in
organizations that are immature with respect to
usability.
Before doing an expert review, consider what you
will answer an upset stakeholder or developer
who asks you:
► “Why are your opinions better than mine?”
27
At least 25% of the comments in a
4
usability test report should be
positive
Agree
Both agree & disagree
Disagree
Truth or myth?
UCD 2012, London Rolf Molich, DialogDesign Page 14

15. Myths about usability testing Copies of Slides
At least 25% of the comments in a
4
usability test report should be
positive
Agree
Both agree & disagree
Disagree
Truth or myth?
Positive Findings
1. Make it easier for you to sell inconvenient truths
2. Ensure that the development team doesn’t
remove a feature that users actually liked
30
UCD 2012, London Rolf Molich, DialogDesign Page 15

16. Myths about usability testing Copies of Slides
Positive Findings
”A spoonful of sugar helps the medicine go down”
Mary Poppins
31
Even
developers
have feelings
UCD 2012, London Rolf Molich, DialogDesign Page 16

17. Myths about usability testing Copies of Slides
Required by UPA CPC
Positive findings are required by the
UPA Code of Professional Conduct
► The CPC requires you to "accurately report
both the positive and negative feedback".
► If you can prove that there is nothing positive
to say, then you have fulfilled the CPC.
I am not sure how I would prove this.
33
When you have a hammer in your hand, everything looks like a nail
-- The Law of the Instrument, Abraham Kaplan
UCD 2012, London Rolf Molich, DialogDesign Page 17

18. Myths about usability testing Copies of Slides
Real World Data
Number of reported positive findings for each CUE-9 team
50%
30%
10%
0 0 1 1 2 2 2 2 2 3 3 4 4 5 8 10
35
Usability testing can be conducted
5
by anyone
Agree
Both agree & disagree
Disagree
Truth or myth?
UCD 2012, London Rolf Molich, DialogDesign Page 18

19. Myths about usability testing Copies of Slides
Usability testing can be conducted
5
by anyone
Agree
Both agree & disagree
Disagree
Truth or myth?
Quality testing can be conducted
5
by anyone
Agree
Both agree & disagree
Disagree
Truth or myth?
UCD 2012, London Rolf Molich, DialogDesign Page 19

20. Myths about usability testing Copies of Slides
If you
don’t care
about quality,
everything else
is trivial
-- Gerald M. Weinberg
10 Assessments
1 professional passed with flying colors
2 professionals had serious deviations but no critical deviations from
generally accepted good practice in usability testing.
5 professionals had both critial deviations and serious deviations
2 professionals did so badly that the assessor recommended that the
client should not pay for the service, or that the test should be
repeated at no cost for the client.
A "serious deviation" from good practice may skew important results.
A "critical deviation" from good practice will skew
important test results seriously.
40
UCD 2012, London Rolf Molich, DialogDesign Page 20

21. Myths about usability testing Copies of Slides
Client Actions
Actions taken by clients to prevent future quality problems:
► Develop and publish usable guidelines for good usability testing.
The guidelines are based on observed user behaviour.
► Quality Assurance as part of procurement
“Do a test while we observe”
Is their understanding of a good usability test the same as ours?
► Make assessments available to teams and insist that they respond.
Fortunately, most usability professionals take pride in doing things right
41
Certification
► The German UPA is currently working on a
professional certification scheme for
►Foundation level (“Do You Speak Usability?”)
►Usability Engineer
►Usability Tester
► Certification procedure:
►40 multiple choice questions
►Relevant deliverable (advanced levels only)
42
UCD 2012, London Rolf Molich, DialogDesign Page 21

22. Myths about usability testing Copies of Slides
Did this presentation change some
6
of your views of usability testing?
Yes
Don’t know
No
Evaluation
Takeaways
► As usability professionals we must set a good example for
the world
► Our products, presentations, doings must be usable and
useful: Correct, relevant, brief, constructive
► Five users are enough to drive a useful iterative cycle.
Never claim that you can find all usability problems
► Expert reviews are valuable – but politically challenging
► Include positive findings
44
UCD 2012, London Rolf Molich, DialogDesign Page 22