Usability testing involves:
1. Recruiting and testing target users on a product or system
2. Analyzing the results to identify any usability problems
3. Reporting findings and recommendations to stakeholders
2. Rebecca Destello
• Restaurant manager
• Front-end web developer
• User researcher
• Affiliate lecturer, UW HCDE
• Chair of Puget Sound PSSIGCHI
• Manager, Research & Insights,
Atlas Informatics
R. Destello | Usability Testing
3. Atlas Recall is a searchable photographic memory
for your digital life — if you see it, you can search
it and share with anyone, from anywhere.
atlas.co
R. Destello | Usability Testing
5. R. Destello | Usability Testing
Tonight’s lecture
What you’ll need to know
to do this as a professional
6. R. Destello | Usability Testing
Why do we do
usability testing?
7. Because there is a science to how people use things.
R. Destello | Usability Testing
8. usability testing:
the flow
R. Destello | Usability Testing
Identify
problems in the
system
Create a study
Recruit and
schedule target
users
Test with
participants
Analyze +
Synthesize
Report out
Evangelize
findings!
Repeat
9. The one book you need:
Handbook
of Usability Testing
by Jeffery Rubin and Dana Chisnell
R. Destello | Usability Testing
11. • Identify key stakeholders
• Scope and priorities
• Identify & recruit target users
• Establish a timeline
• Identify usability concerns
planning a usability study
R. Destello | Usability Testing
12. It is critical to get key players involved up-front
• Project sponsors
• Product owners
• People who will have to act on the findings
identify key stakeholders
R. Destello | Usability Testing
13. R. Destello | Usability Testing
How do we
generate support
for usability testing?
14. 1. Start running studies!
2. Get people to observations
3. Always be prepared with talking points; include
findings from other studies that made a critical
difference
4. Be a good team player - be willing to compromise
and negotiate
5. Map usability planning to the company and product(s)
5 steps to getting buy-in
R. Destello | Usability Testing
15. Think about:
• What is the political landscape? How much buy-in do
you have from the stakeholders?
• Has usability been done before?
• What questions are you trying to answer?
• What is your budget?
• What time constraints are you working with?
scope and priorities
R. Destello | Usability Testing
16. Tips for identifying areas of concern:
• Ask your stakeholders
• What do you want to learn from this test?
• What are your greatest fears about this product?
• Are there different points of view in the group about this
issue?
• What are you willing to change as a result of this test?
• Mine customer feedback
• Perform a task analysis
• Evaluate the product (heuristic evaluation or cog. walkthrough)
identifying usability concerns
R. Destello | Usability Testing
18. • Find your target user by asking your client to add
a survey to their website
• Use Craigslist (with caution)
• Ping your social networks
• Go where your users might congregate
And, write a good screener…
tips to find your target user
R. Destello | Usability Testing
19. The screening questionnaire needs to be
• Specific, inclusive of all necessary
criteria
• Contain boundary conditions
• Contain exclusion criteria
May need to stand alone
• May be passed off to others, firm,
colleague
Can be lengthy
• 10 pages
• 28 questions
screeners
R. Destello | Usability Testing
20. Tips for writing a good screener
• Ask the elimination questions first
• Eliminate conflicts of interest
• Screen for experience
• Recruit based on behavior and attitudes
• Screen out “professional” users
• Write a good screener (bit.ly/uxmattersrecruiting)
screeners
R. Destello | Usability Testing
21. usability test participant incentives
Why should you give an incentive?
• Participants are helping you by giving up their time
• Participants will leave with a positive attitude
• Participants will tell their friends (larger pool of
participants for future studies)
• Best way to get qualified, representative participants
R. Destello | Usability Testing
22. usability test participant incentives
What should the incentive be?
• Incentives are determined based on the client /
budget
• Example:
– Microsoft gives out free software
– Boeing gives out movie tickets
– Chocolate (at the very least)
How much should an incentive be?
• Based on budget and target audience
R. Destello | Usability Testing
23. Your timeline will be determined by
• The complexity of the product
• The broadness of the scope of the test
• What set up will be needed (facilities, travel, special equip, etc.)
• Level of difficulty in recruiting the target users
• # of participants needed
• How formal the report out will need to be
• Related deadlines (design, dev, and even holidays!)
establish a timeline
R. Destello | Usability Testing
25. what’s in the usability study?
ü Product or assets to test
ü Participants (5 works if doing iterative testing, if only 1 user group)
ü Moderator script with scenarios and tasks
ü Pre/Post task questionnaires
ü Checklists for study setup
ü Participant documents (Informed Consent, NDAs, Questionnaires)
ü Data collection tools
ü Participant incentives
R. Destello | Usability Testing
26. • Greet and make participant comfortable
• Explain study, setup, and complete forms
• Explain Thinking Aloud Protocol
• Remind them they are not being tested
• Begin tasks
• Wrap-up
• Questionnaire, if applicable
• Final Comments/Questions
• Thank them profusely
• Provide gratuity
• Parking validation, if applicable
testing framework
R. Destello | Usability Testing
27. what about scheduling?
Allow time during the study:
• Time for test (30 minutes - 1 hour)
• Time in between (1 hour between participants)
– Setup for the next study, check in with observers, clean up
notes, start examining data while it’s fresh
R. Destello | Usability Testing
E X A M P L E
28. stay organized with a checklist
Why use a checklist?
• Checklists are good, reusable tools for any usability study
• Checklists outline all things that must be completed before,
during, and after study
• Helps ensure study consistency and increases internal validity of
the study from day 1!
Key Takeaway: Even if you don’t think you’ll need it, create a checklist of
all the things you need to do for study (you’ll be surprised what you forget!)
R. Destello | Usability Testing
29. participant documents
Before Testing
• Pre-Test Questionnaire (optional)
• Informed Consent/NDA (required)
During Testing
• Post-task Questionnaires
After Testing
• Post-Test Questionnaire
R. Destello | Usability Testing
30. R. Destello | Usability Testing
What’s the role of
the moderator?
31. so what does the “moderator” do?
ü Interacts with the participant
ü Ensures the participant’s comfort
ü Sets the pace and tone of the study
ü Administers scenarios, tasks, and questionnaires
ü Handles prompting and working with the participant to
understand the reasons behind problems.
ü Provides clarifications so note taker can capture data
ü Takes care of the participant!
R. Destello | Usability Testing
32. why is moderating challenging?
• Goal is unbiased feedback
• Participants need to be comfortable, but the
test situation is often unsettling
• You need to know what the participant is
thinking, but you can’t read their minds
R. Destello | Usability Testing
33. intervention: what are they doing?
When and how to intervene during a test:
• In general, it’s preferable to let participants decide
how much time to work on a task
• Some exceptions
– If they are getting very frustrated
– If you feel that you’ve got data on the task at hand and you
want to keep going and get to other tasks
– If they’ve wandered so far down a bunny hole
they are unable to proceed with the remaining tasks
R. Destello | Usability Testing
34. prompting: guidelines
When prompting, you want to…
• Get the user to elaborate on something that is unclear
Without…
• Introducing bias
• Influencing the user’s behavior
• Taking over as the expert or primary speaker
• Unnerving the user through implication that she
did something wrong
R. Destello | Usability Testing
35. prompting: techniques
Provide Acknowledgment Tokens at natural intervals
• Mm hmm, I see…..
Focus on Questions
• How do you feel about that?
• Tell me what you are thinking…
• Tell me more about that…
Repeat their word or phrase back to them as a question
• They say: That was confusing / You say: “confusing…?”
Make the alternative explicit
• Was this task easy or difficult?
R. Destello | Usability Testing
36. Keep in mind…
• People cannot reliably reflect on their own cognition
• Without video to playback questions like “what were you
thinking when…” or “why do you think you…” you will find
you often have missing / bad data
• Participants will struggle to come up with a believable
rationale while being unaware of the true cause for their
actions
prompting: words of warning
R. Destello | Usability Testing
37. participant questions
Participants sometimes ask the moderator for help. Turn their
question into a question.
How would you respond to the following?
• Is that right? Did I do it right?
• Am I supposed to keep going?
• What should I do now?
• Should I click here?
• What is this?
R. Destello | Usability Testing
38. The most important thing to remember
about moderating is this:
Being in a study is stressful for the participant – their
comfort level and state of mind are vital. When people
are stressed, their behavior/memory are impaired.
R. Destello | Usability Testing
39. Things to keep in mind while moderating:
• Be patient
• Be excited about the process of discovery
• Be “small” in relationship with the participant
• Be as neutral as possible
• Be professional
• Be caring to the participant’s needs
Important: The newer you are, the less you should talk.
moderate with care
R. Destello | Usability Testing
40. q Does the moderator use relaxed body language?
q Does the moderator encourages/discourages responses?
q Does the moderator follow-up (when allowed) and dig into ambiguous
statements or promising topics or ideas
q Does the moderator have any verbal or behavioral tics that would be
distracting?
q Does the moderator provide excessive reinforcements?
q Is the moderator aware of expressions of support, sympathy, joy, anger, and
other emotions?
q Does the moderator recover from miss-steps well.
q Does the moderator make the interviewee comfortable during the initial
briefing and questions.
q Does the moderator clearly explain the limits of his/her ability to help the
participant?
q Does the moderator follow protocols for prompting?
q Was the moderator sensitive to cultural/org. issues?
CHECKYOURSELF
beforeyouwreckyourself
R. Destello | Usability Testing – Moderator Checklist credit: Chauncey Wilson
41. moderator script: the essentials
Prepares participants for the session
• Describes what to expect (how long, what they’ll be doing, if they’ll be observed, if
they’ll be videotaped)
• Gives think-aloud instructions
• Lets them know their rights
Why is the script important?
• Ensures that the facilitator gives the same information to all the participants
• Stresses that the study is a test of the product, not of the participant
R. Destello | Usability Testing
42. moderator script: the essentials
How to write unbiased, non-leading usability tasks (at the
10,000 ft. level):
First think: What is the user naturally trying to do at this step? Then –
• Make them short (hard to remember long tasks)
• Write tasks in the user’s natural language
• Avoid jargon
• Don’t use words found in the system
R. Destello | Usability Testing
43. For a successful study…
• Make sure to know what data you are looking for (what questions are
you trying to answer?)
• Define success criteria for each task
• Pilot the study at least once
• Do a quick “clean up” of notes after each session
• Have a plan for effective note-taking so you can spend less time
analyzing data
other considerations
R. Destello | Usability Testing
45. Data you can capture during the study:
• Pathways
• Verbal commentary
• Think Aloud Protocol (TAP)
• Time on task (only when not using TAP)
• Task completion
• And more
data collection: available data
Data can be gathered using:
• Handwriting
• Document
• Spreadsheet
• Usability software
R. Destello | Usability Testing
46. data collection: DOs
Do capture:
• Observations; what you see the participant do
• Quotes; what you hear the participant say
• Details; what specifically caused issues or
prevented task completion
R. Destello | Usability Testing
47. data collection: DON’Ts
Don’t capture:
• Why you think they are doing what they are doing (if
you aren’t sure, ask the participant)
• Broad generalizations that don’t contain a
description - “She had a lot of trouble finding the link”
won’t be helpful when you’re analyzing findings
R. Destello | Usability Testing
48. Use codes to note
instances of issues
that you’re tracking
Example:
F = failure
E = error message
H = goes to help
Keep an issues
matrix and add to it
after each
participant.
data collection: note-taking tools
R. Destello | Usability Testing
53. • Word of mouth and email
• Executive summary
• Formal report
• Presentation
How you choose to communicate results can depend upon
your audience, the culture, timeline, and the potential
“Freak Out Factor”
communicating results
R. Destello | Usability Testing
54. • Know your user (reader) – what’s important to them?
• Accentuate the positive, but deliver the news
• Report the facts (don’t overreach)
• Be thoughtful about word choice
(opportunity vs. issue)
• When in doubt, recommend further testing
• Check grammar and spelling
communicating results: tips
R. Destello | Usability Testing
56. In groups of 2, using a phone 1 person act as the participant, the other as the
note taker and go to Nordstrom.com and complete the following tasks:
1. You’d like to see the hot new items for this season. Where on the site
would you browse what’s trending this fall for yourself?
2. You want to buy a new shirt for your 5 year old niece. Where would you
find a shirt that might work for her?
Focus on problems in navigation and filter use.
in-class exercise: Navigation
R. Destello | Usability Testing
57. Don’t forget!
• Take notes (Where did the participant have trouble?
What did they say? What worked?)
• Use prompts to keep them talking: “What are you
thinking?” “Say more.” “Is that what you expected?”
• Don’t say much and listen closely
in-class exercise: Navigation
R. Destello | Usability Testing
58. in-class exercise: Navigation
R. Destello | Usability Testing
In groups of 2, using a phone 1 person act as the participant, the other as the
note taker and go to Nordstrom.com and complete the following tasks:
1. You’d like to see the hot new items for this season. Where on the site
would you browse what’s trending this fall for yourself?
SWITCH ROLES
2. You want to buy a new shirt for your 5 year old niece. Where would you
find a shirt that might work for her?