The document summarizes a presentation on myths and dogmas related to usability testing. It discusses several commonly held beliefs and presents data from previous studies to evaluate whether they are truths or myths. Some of the myths addressed include the idea that five users are enough to find most usability problems, that expert reviews are as reliable as usability tests, and that usability testing can be conducted by anyone. The presentation aims to dispel unsupported notions and emphasize the importance of using empirical data to inform best practices in usability work.

1. World Usability Day Wroclaw 2011
”Five users will find 85% of
the usability problems” - Rolf Molich
and other myths about DialogDesign
usability testing

2. Dogma 1
► Usability professionals must be highly usable
Reports, presentations, responses to comments, etc.
2

3. Źdźbło w oku
bliźniego widzisz,
a belki w swoim
nie dostrzegasz
Sweep before your own door
before you sweep
the doorsteps of your neighbors

4. When you point your finger at someone, remember that three fingers point at yourself

5. Dogma 2
► Data trumps opinion at any time
5

6. A Good, Ol’ American Bake-off

7. A Usability Bake-off

8. CUE-9:
19 Atlanta +16 Chemnitz
Professionals
5 Videos
>100 Different
Findings

9. Data to Fight
Unsubstantiated
Myths and Voodoo
-- CUE motto

10. Excerpt from Usability Test Report - CUE-6, Team C

11. Excerpt from Usability Test Report - CUE-6, Team G

12. Excerpt from Usability Test Report - CUE-6, Team G

13. Five users are enough to catch
1
85% of the usability problems in
practically any product
Agree
Both agree & disagree
Disagree
Truth or myth?

14. Five users are enough to catch
1
85% of the usability problems in
practically any product
Agree
Both agree & disagree
Disagree
Truth or myth?

15. ”The parabola of optimism” (Jared Spool)

16. Real data: CUE studies
CUE = Comparative Usability Evaluation
► 10-17 professional teams test the same website.
► In all CUE-studies, more than 60% of problems
were uniquely reported.
► In CUE-4, more than 300 issues were reported.
16

17. The main goal of a usability test is
2
to discover usability problems
Agree
Both agree & disagree
Disagree
Truth or myth?

18. The main goal of a usability test is
2
to discover usability problems
Agree
Both agree & disagree
Disagree
Truth or myth?

19. Real cowboy
programmers don’t
need no stinkin’
usability test
-- Anonymous

20. Goals of usability test
1. Find problems so they can be corrected
2. Demonstrate convincingly to skeptical
stakeholders (developers, management)
that problems exist and that there are
efficient ways to find and correct them
20

21. Expert reviews provide results that
3
are as reliable as those from
usability tests
Agree
Both agree & disagree
Disagree
Truth or myth?

22. Expert reviews provide results that
3
are as reliable as those from
usability tests
Agree
Both agree & disagree
Disagree
Truth or myth?

23. Expert review vs. Utest
Data from CUE-studies, most notably CUE-4,
consistently shows
► Expert reviews find as many problems as
usability testing
► Expert reviews overlook problems –
but so do usability tests
► Expert reviews are cheaper than usability tests
These findings apply only when expert reviews
are carried out by Experts
23

24. Eye-tracking shows us what users
4
see, so it reveals important issues
that usability testing without it can't
Agree
Both agree & disagree
Disagree
Truth or myth?

25. Eye-tracking shows us what users
4
see, so it reveals important issues
that usability testing without it can't
Agree
Both agree & disagree
Disagree
Truth or myth?

26. Real data: Eye tracking
► CUE-6 tested Enterprise.com, a large
American car rental website
► One team (team G) used eye tracking while
the other 12 teams used conventional
evaluation methods
► Team G was from a renowned promoter of eye
tracking
► Team G did not report any important findings
that were not reported by other teams
26

27. At least 25% of the comments in a
5
usability test report should be
positive
Agree
Both agree & disagree
Disagree
Truth or myth?

28. At least 25% of the comments in a
5
usability test report should be
positive
Agree
Both agree & disagree
Disagree
Truth or myth?

29. Positive Findings
1. Make it easier for you to sell inconvenient truths
2. Ensure that the development team doesn’t
remove a feature that users actually liked
29

30. Positive Findings
„Łyżka cukru pozwoli przełknąć ci lekarstwo”
”A spoonful of sugar helps the medicine go down”
Mary Poppins
30

31. Even
developers
have feelings

32. Required by UPA CPC
Positive findings are required by the
UPA Code of Professional Conduct
► The CPC requires you to "accurately report
both the positive and negative feedback".
► If you can prove that there is nothing positive
to say, then you have fulfilled the CPC.
I am not sure how I would prove this.
32

33. When you have a hammer in your hand, everything looks like a nail
-- The Law of the Instrument, Abraham Kaplan

34. Real World Data
Number of reported positive findings for each CUE-9 team
50%
30%
10%
0 0 1 1 2 2 2 2 2 3 3 4 4 5 8 10
34

35. Usability testing can be conducted
6
by anyone
Agree
Both agree & disagree
Disagree
Truth or myth?

36. Usability testing can be conducted
6
by anyone
Agree
Both agree & disagree
Disagree
Truth or myth?

37. Quality testing can be conducted
6
by anyone
Agree
Both agree & disagree
Disagree
Truth or myth?

38. If you
don’t care
about quality,
everything else
is trivial
-- Gerald M. Weinberg

39. 10 Assessments
1 professional passed with flying colors
2 professionals had serious deviations but no critical deviations from
generally accepted good practice in usability testing.
5 professionals had both critial deviations and serious deviations
2 professionals did so badly that the assessor recommended that the
client should not pay for the service, or that the test should be
repeated at no cost for the client.
A "serious deviation" from good practice may skew important results.
A "critical deviation" from good practice will skew
important test results seriously.
39

40. Client Actions
Actions taken by clients to prevent future quality problems:
► Develop and publish usable guidelines for good usability testing
► Quality Assurance as part of procurement
“Do a test while we observe”
Is their understanding of a good usability test the same as ours?
► Make assessments available to teams and insist that they respond.
Fortunately, most usability professionals take pride in doing things right
40

41. You can’t
control what
you don’t
measure
-- Tom DeMarco

42. UTA Checklist
Seven out of currently about 70 checkpoints:
► Do tasks contain “humorous” elements?
► Do tasks contain hidden clues?
► Is the moderator sufficiently curious?
► Does the moderator provide hidden clues?
► Does the moderator talk too much?
► If both a moderator and a note taker attend the test,
are their notes consolidated after the test?
► Can all findings in the test report be traced back to
actual events during the test?
42

43. Certification
► UPA-I is currently working with the German
UPA on a professional certification scheme
► Our friends at Human Factors International
(HFI) have proven that there is some interest
in professional certification in the market, for
example from employers
43

44. A burning question
► Are there sufficiently many interesting things
that we agree on in our field to make
certification meaningful?
44

45. “My usability testing skills are
7
above average”
Agree
Both agree & disagree
Disagree
Evaluation

46. Dziękuję.
Życzę miłego
dnia!

47. Did this presentation change some
8
of your views of usability testing?
Yes
Don’t know
No
Evaluation

48. Takeaways
► As usability professionals we must set a good example for
the world
► Our products, presentations, doings must be usable and
useful: Correct, relevant, brief, constructive
► Five users are enough to drive a useful iterative cycle.
Never claim that you can find all usability problems
► Expert reviews are valuable – but politically challenging
► Remember the positive findings 
48