1. 1
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 1
Usability Testing for
Usable Accessibility
U
Kate Walser
Whitney Quesenbery

2. 2
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 2
Hi
Kate
Interaction design, usability
testing and slipping
accessibility under the radar.
Whitney
Users’ stories, plain language,
discovered accessibility as a
right in elections
What about you?

3. 3
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 3
What do you want to learn today?
What do you think usability
testing is?
Does anyone in your
organization do any sort of
usability testing?
What stands in the way of
doing usability?

4. 4
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 4
Here’s what we planned
Introduction to usability testing
Demo of a usability test
Planning a usability test
Take a deep breath
First practice test
Debrief and moderating skills
Break
Testing with people with disabilities
Practice tests
Debrief and applying what you learned

5. 5
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 5
What is usability (testing)?1

6. 6
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 6
Usability is about people

7. 7
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 7
Different types of
people…
doing different
tasks…
have different
usability needs
depending on
their goals
and tasks
For this person efficiency and effectiveness are
the most important dimensions.
For this person, easy to learn and error
tolerant are the most important dimensions.

8. 8
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 8
Helpful models for thinking about usability
A’s Ability, aptitude, attitude
(what users bring to the site)
E’s Efficient, effective, engaging, error-tolerant, easy to learn
(dimensions of usability)
Use Useful, usable (operable), desirable
(how we define success)

9. 9
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 9
Usability testing lets you see behavior
Observe
 People like the audience using the
site rather than asking them about it
 How easily they meet their goals
 What causes confusion or problems
Inform design
 Lets you confirm or challenge
assumptions about the design.
 Gives you the opportunity to fix
problems.
Informal usability testing of voting
materials at the Farmer's Market in
Olympia, Washington

10. 10
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 10
How easily can we learn from users?
We learn from users
by watching them use the
site or other materials.
 Don’t explain or demo.
 Watch what they do.
 Listen to their comments.
 Take their problems
seriously.
Dona Vitale conducting a usability test
in Chicago, 2005

11. 11
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 11
A usability test lab
A formal lab includes a room where
the participants work, and a
separate room for observers to
watch.
Sessions are often recorded on
video
Michigan State University Usability & Accessibility Center
http://usability.msu.edu/default.asp

12. 12
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 12
Many options
Usability testing
can happen in
different forums,
even farmers
markets…
…and hallways, using
devices or even paper
and post-it notes as
shown here at General
Services
Administration and
Open University

13. 13
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 13
Usability testing with people with disabilities
How will a person with disabilities use the product?
 Assistive technology use and settings
 Other resources they use
How usable is the product by people with disabilities?
 Content, forms without surrounding visual context, proximity
 Navigation
 Truly accessible vs. just meets the standards
How engaging and effective is your content strategy?
 Images only vs. engaging context + images
 Search engine readiness (headings, link labels)
 Great content flow

14. 14
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 14
Demo2

15. 15
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 15
Your goal
You want to a group from Access to go
fishing at a local park.
Find a park that has a wheelchair
accessible fishing pier.
What are the admission fees?

16. 16
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 16
Planning a usability test3

17. 17
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 17
Begin with your goals
Success − for you
and your users
Your site visitors'
goals (and their reality)
• Answer people's questions about
your topic
• Have people complete
a transaction correctly
without calling
• Get more people to fill out
a form correctly without calling
Your
organization's
goals

18. 18
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 18
Location and context: formal informal
Recruiting: defined opportunistic
Activities: instructed tasks free tasks
Questions: structured unstructured
Data collection: observation task/data only
Results quantitative qualitative
The purpose of the test influences the plan

19. 19
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 19
Think about what senses are most involved in the UX
Example Component Senses Include people
who are…
Website,
Mobile
- Visual interface
- Links, menus
- Forms
- Vision
- Way to navigate
(mobility, speech)
- Blind
- Vision impaired
- Mobility impaired
Phone - Speech and
audio feedback
- Hearing
- Speech (or way to
indicate option)
- Deaf
- Hearing impaired
- Speech challenges
Multimedia - Visual interface
- Sound
- Way to select
options
- Vision
- Hearing
- Way to navigate
(mobility, speech)
- Blind
- Vision impaired
- Deaf
- Hearing impaired
- Speech challenges

20. 20
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 20
What you need for a usability test
What
 The material you want to test
 Scenarios, if it’s not just an exploratory “try it out” type test
Where
 A quiet room (Maybe. We know of very successful
usability testing in an open marketplace, a hangar
at an air show, the chemotherapy center at the NIH
Clinical Center.)
Who
 Moderator
 Observer/note-taker
 Users: 3 − 6 people, one at a time

21. 21
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 21
The moderator runs the session
Impartial, unbiased, observing
No teaching!
Listen and watch
Open-ended questions: Why?
How? What were you doing?
Moderator roles:
Flight attendant:
Ensuring safety and comfort
Scientist:
Planning,
maintaining objectivity,
managing data

22. 22
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 22
Observers and note-takers
Watch (and listen) quietly for whether the user
 Is confused by anything, or has to re-read information
 Has to correct mistakes, or backtrack to correct naviation
Take good notes
 Write down what you see and hear. Don’t “translate” or put words in
the user’s mouth
 Be specific. Not "he's confused." But "he said he doesn't know what
APR means.”
 Don't infer the user's reasons for doing something.
 Don't solve problems while taking notes. That's for later.
It will take all your concentration to note what is happening.

23. 23
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 23
Briefing the user
Thanks for trying out this […]. Your doing this will help improve this
material.
 Note: If you never use the words "test" or "evaluate" with the user,
you will not have to say "We are not testing you."
You can stop anytime.
Your involvement will be confidential.
If you get stuck or confused, say so.
Please let me know what you are thinking as you use this …

24. 24
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 24
Different ways of having people try out your site
Ask participants for their own scenarios.
 Have you used this site? What for? Please show me how you did that
with this site. or Would you be likely to use this site? What for? Please
show me how you would do that with this site.
Give participants tasks you have written.
 Write scenarios that will have people use parts of the site you are
worried about.
Customize scenarios for the participants.
 Write general scenarios, but adapt them to what you know about the
participant, letting them fill in details to make the task more relevant.
Use both of these with their own scenario first.

25. 25
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 25
Use think aloud during − or after the task
Consider asking people to “think aloud” as they work
 What they’re doing
 Why they’re doing it
Assistive technology may make think aloud harder. If so, you can go
back over the material (“retrospective”)
 Ask the participant to walk you through what they did
and why.
 Use the material as a guide for the discussion.
 Ask if anything was confusing or frustrating.

26. 26
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 26
Use simple techniques to help users keep going
If the user says, “hmmm” or “oops” or “I wonder…”
 Say, “What questions do you have right now?”
If you are doing "think aloud" and the user is silent for
10 or 20 seconds (count!)
 Say, “What are you thinking?”
If users stop because they think they're done
or they are stuck (and you think there’s a problem)
 Summarize what you saw the user do.
 Ask "What would you do now?"

27. 27
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 27
How easily can we learn from users?
Usability testing
does not have to be formal,
lengthy, or expensive.
You don’t need
 a formal laboratory
 100s of participants
 special equipment (except for
your voting system)
 special recording systems
Poster from Washington State

28. 28
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 28
Summary: When to do usability tests
Early
Understand users
and their goals
Project
Initiation
Design and
Development
Product
Release
In Use
Middle
Evaluate the site
throughout the
development
process
End
Evaluate the
finished site to
measure its
success
Ongoing
Watch site
usage and
technical
support for
evidence of
problems

29. 29
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 29
Take any opportunity for a quick usability observation
The next time someone asks
you a question about your site,
ask them to use it instead.
Watch as they try to find the
information.

30. 30
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 30
First practice test
25 minutes
4

31. 31
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 31
Plan your practice test
Pick something you want people to do.
Review the script for briefing the participant.
Write the task instructions.
Starting point (desktop, a URL?)
Phrase it in their language
What do you want to watch for?
Remember to thank the participant at the end

32. 32
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 32
Ready? Three rounds
Get into a group of three.
Take turns, so everyone tries each role.
We’ll call time.

33. 33
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 33
How did that go?
What felt awkward?
What worked well?
Anything surprise you?

34. 34
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 34
Moderating skills5

35. 35
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 35
From earlier: The moderator runs the session
Impartial, unbiased, observing
No teaching!
Listen and watch
Open-ended questions: Why?
How? What were you doing?
Moderator roles
Flight attendant
Ensuring safety and comfort
Scientist
Planning,
maintaining objectivity,
managing data

36. 36
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 36
Moderator Responsibilities
Often, but not always, create the test plan and testing scenarios
Welcome and support the participant
 Informative but not revealing
Avoid accidentally influencing results
 Explain the objectives succinctly
 Helpful but not too helpful
 Try not to answer questions that affect outcomes
Observe quietly and avoid distracting participants
 Thoughtful about taking notes
 Ask questions only when they realize participant is at a stopping
point or is ready to talk

37. 37
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 37
Managing the customer
Often, moderators also manage the customer
team whose products will be used in the
sessions.

38. 38
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 38
Prepare the customer team
Define and prioritize objectives
 Address these in scenarios, probe questions, questionnaire
 Determine which are “must haves” and which are optional
Describe to the customer the process, logistics
 Where they will sit (if they will attend)
 What you expect from them (no interruptions, other room, etc.)
Explain how you will handle “edge cases”
 Late or no-show participants
 Technology woes
 Questions that arise during the sessions (and how to get moderator’s
attention to ask)

39. 39
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 39
A good moderator uses a test plan
Creates a test plan to use
during the session
 Manage the usability testing
like a mini-project
 Provide clear schedule with
tasks and deadlines –
moderator, customer, etc.
 Follow up to make sure
everything’s set
Creating a test plan and schedule
for usability testing helps avoid
mistakes and oversights.

40. 40
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 40
A good moderator
Prepares
 Create a test script
 Practice on real participants
 Verify that everything is ready and there are plan B’s
Arrive early – “on time” is “late”
Listen and watch with an open mind
 If you wonder, ask without bias – don’t jump to conclusions
 Story – boy with the black crayons
Respect the participant
 Watch for “trouble” indicators
 Stop as needed due to time, tech issues, or other factors

41. 41
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 41
Expect most participants to be…
Hesitant or nervous
 Are thier skills being tested?
 What if they don’t give the
“right” answer?
Ready to throw their friends and
colleagues under a bus before
themselves!
 Give them an easy way to
provide their feedback
without worrying about
hurting feelings
 Ask what their friends or
colleagues will think
Smiling man points to friend who is covering his face.
Photo credit: “accuse,” Grey World
http://www.flickr.com/photos/greyworld/

42. 42
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 42
Great moderators make participants feel valued and
important
If possible, confirm the session
yourself
 Call or email the participant to
confirm, introduce themselves
 Make sure participant knows
where to go
 Give participant a friendly face
to expect
Memorize the script
Learn negotiation tactics
 Maintain an open mind
 Manage challenges that arise
(Handle surprises without
disruption
A little girl sitting in dentist’s chair is smiling
from ear to ear after learning she has no
cavities.
Photo credit: “No Cavities!” daveparker
http://www.flickr.com/photos/daveparker/

43. 43
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 43
Have a backup plan
Person uses paper printout of screens to describe interaction
and path
Photo credit: “Usability Testing our Paper Prototypes” Jason de Runa
http://www.flickr.com/photos/jderuna/

44. 44
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 44
Summary: Moderators
Prepare and anticipate problems
Observe without influencing outcomes
Make participants feel valued

45. 45
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 45
Break
15 minutes*

46. 46
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 46
Practice tests
45minutes
6

47. 47
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 47
Moderating and observing
 Conduct sessions the same way.
 Ask the participant to “pause” the AT or lower the
volume as they may have thoughts they want to
share.
 Remember that the AT becomes a large component
of the UX
 Avoid interrupting the user as they use the AT
 Remember to watch the cursor, not the mouse.

48. 48
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 48
Get into teams
If you are the moderator, use your practice test.
If you are an observer, think about
What you will watch for
Where to sit so you can see,
but won’t distract the participant.
Challenges observing this person’s
AT or accessibility setup.
Remember: focus on what happens, not analysis
or recommendations

49. 49
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 49
Planning a test with people with
disabilities
15 minutes
7

50. 50
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 50
How did that go?
What felt awkward?
What worked well?
Anything surprise you?

51. 51
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 51
Consider how different AT and disabilities affect
usability test logistics
 Recruiting
 Participants
 Scenarios and handouts
 Incentives
 A place to tests

52. 52
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 52
Recruiting
Allow more time to find participants
 Screener + disabilities
Look for community groups and ask
 Meetup, Facebook, Twitter
Use specialized recruiting services
 KnowbilityAccessWorks portal
Work with outreach groups
 Advocacy groups
 Employment non-profit groups
 Colleges and universities

53. 53
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 53
Outreach Groups
Group Focus Website
American Foundation for the
Blind
Blind, Vision loss http://www.afb.org
Columbia Lighthouse for the
Blind
Blind, Vision loss http://www.clb.org/
Gallaudet University Deaf, Hearing loss http://www.gallaudet.edu/
Governor’s Committee on
Employment of People with
Disabilities
Various Search for your state name +
“governor’s committee on
employment…”
National Federation of the Blind Blind, Vision loss https://nfb.org/
Rehabilitation centers Mobility impairments Search for rehabilitation
centers or programs in your
area

54. 54
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 54
KnowbilityAccessWorks Portal
http://www.knowbility.org/v/service-detail/AccessWorks-Usability-Accessibility-Testing-Portal/3k/
Knowbility’sAccessWorks portal gives users with disabilities a way to sign up to participate in online
tests and explains to usability and marketing professionals that they can use Loop11’s online
testing and recruit users with disabilities.

55. 55
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 55
Decide where, when to conduct testing
Remote
Conduct via online tools
 Accessible screen sharing
software (or) accessible online
usability testing
Moderator may facilitate or
may just review results
later
User uses their own, native,
familiar assistive
technology and setup
In Person
Conduct in a physical location
 Traditional lab
 Some other place
Moderator sits with participant
Be prepared to provide the
assistive technologies or
ask users to bring theirs
(including any
personalized style sheets
for web pages)

56. 56
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 56
In-person: Environment & Setup
Location becomes even more critical
 Is it reachable by public transportation?
 Is the building itself accessible?
 Are the signs to find the room accessible?
Room setup matters
 Is there space to navigate with wheelchairs or walking devices? Is
there space for a service animal to lie down near its human?
 Are tables or desks wheelchair accessible?
 If visually impaired, can the person sit close enough to the monitor to
view the screen? Can you mirror the screen on another monitor for
the moderator to view?
 How close are other workstations if screen readers will be used?

57. 57
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 57
For in-person tests,
explain setup and ask about accommodations
Explain what will happen and ask about specifics
 “We usually provide handouts for users to read that describe the
scenario. What’s the best way to provide that to you?”
Describe the setup, and include accessibility specifics in your
confirmation emails
 “We’re on the 3rd floor. There’s a ramp to the right of the front doors.
The doors automatically open. There’s an elevator bay in the lobby to
get to our floor.”
Ask if they will bring anything
 Service animal (would they like a rest or water stop for the animal?)
If refreshments will be served, ask about dietary restrictions

58. 58
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 58
Remote / Online: Setup & Use
Ensure that the testing and/or screen sharing application are truly
accessible
 Can the user download / access them ahead of time to install any
plugins and test with their chosen assistive technologies?
 Can they turn on / off screen sharing?
 Can they access and use their assistive technology while using the
screen sharing capabilities?
 Can the remote moderator hear and see what they’re doing?
Ask about the best way to interact with the participant
 Explain what will happen and ask about specifics
 “We usually provide the scenarios via the tool. What’s the best way to
provide that to you?”

59. 59
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 59
Schedule
Add time between sessions for
 Setup time
 Moderator break
Dry runs are critical, preferrably with a representative user or else,
with someone trying to reflect the different considerations
 Online: Try downloading / opening the tool, going to the scenario,
talking with the moderator with AT on
 In-person: Try going from closest transportation spot to location, into
building to designated location, taking / using handouts

60. 60
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 60
Tips for Moderating and Observing
Act like you normally would
 Treat them like you would any other colleague or friend
 Ask first before assuming they need help
Learn the AT
 Practice listening to JAWS or other assistive technology while
someone is talkingAsk first before assuming they need help
 Know when the AT will affect how you interact with the participant
Look for general usability problems as well as those made worse by
poor accessibility or interaction with the AT
 Missing information, problems navigating can reveal problems for
everyone
 Look for issues caused by the presentation

61. 61
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 61
Summary: Testing with People with Disabilities
Consider logistics, location, and setup
Allow more time for recruiting and between sessions
Do a dry run and practice observing

62. 62
Usability Testing for Usable Accessibility – Whitney Quesenbery, Kate Walser 62
Handbook of Usability
Testing by Jeff Rubin
and Dana Chisnell
usabilitytestinghowto.
blogspot.com
www.peachpit.com/promotions/promotion.aspx?promo=137602
Rocket Surgery demonstration video
www.usability.gov
A guide to developing usable and useful web sites (from US Dept of HHS)
www.howto.gov/web-content/usability/first-fridays
First Fridays Product Testing Program
Rocket Surgery
Made Easy
by Steve Krug
www.sensible.com
Learn more about usability testing

63. Whitney Quesenbery
WQusability.com
whitney@wqusability.com
908-617-1122
Kate Walser
CX Insights
kwalser@cxinsights.com
571-281-2626