Every study starts with a question. This session at CSUN 2014 started by examining the questions that usability testing can answer. Short case studies illustrate how the right technique will help us know not only what is happening but also why it’s happening. It's an overview of usability testing as a research method, and what you can (and can’t) learn from working with real people as they try to use a web site or other product.
3. 3
Usability testing answers questions, so we have to
understand the question we are asking.
If we want to know:
Which design helps users complete tasks more
effectively?
We are asking a question that is
Comparative, quantitative (errors, time)
(and why they made those mistakes)
If we want to know:
Do voters understand how to mark their ballot?
We are asking a question that is abut
Mental models, variability, qualitative insights
(and how prevalent each model is)
4. 4
Helpful models for thinking about usability
A’s Ability, aptitude, attitude
(what users bring to the interaction)
E’s Efficient, effective, engaging, error-tolerant,
easy to learn
(dimensions of usability)
Use Useful, usable (operable), desirable
(context of use)
5. 5
The As help you think about diverse participants
Attitude
motivation, emotion, risk tolerance, persistence, optimism, tolerance for
frustration
Aptitude
current knowledge, ability to make inferences or innovate solutions,
expertise, habits
Ability
physical and
cognitive attributes
Source: Dana Chisnell "Character Creator" http://www.slideshare.net/danachisnell/character-creator
7. 7
5Es help you plan usability tests
Usability Goal Types of usability testing needed
Evaluate tasks for how accurately they were
completed and how often they produce errors
Effective
Error-tolerant
Easy to learn Control how much instruction is given to test
participants, or recruit participants with different
levels of knowledge
Construct task scenarios to create situations
with potential problems
Engaging User satisfaction surveys to gauge acceptance
review logs for „time on site‟
Efficient Time tasks with realistic tasks and working
versions of the software
8. 8
The recipe for planning a usability test
Location and context: formal informal
Recruiting: defined opportunistic
Activities: instructed tasks free tasks
Questions: structured unstructured
Data collection: observation task/data only
Results quantitative qualitative
9. 9
A few examples of questions
and how we answered them
These examples all come from work on
elections and civic design..but the
principles apply to any project.
10. 10
Comparing two designs
Can a design improve user performance?
Plain vs. traditional language
About this project
45 participants
3 lab setting, 8 days
2 moderators
Instructions for voting
Careful observation
Satisfaction survey
We learned about
Voter accuracy
Order effects
Voter preference
Results
Statistical analysis
Participant preference
11. 11
Comparing two designs
Can a minor difference have a major impact?
Left aligned or right aligned
names
About this project
100 participants
2 locations, 1 day
4 moderators + 6 others
Instructions for voting
Minimal observation
We learned about
Voter preference
Time to vote
Number of errors
Results
Statistical analysis of marked
ballots
12. 12
Testing a new design
What problems might voters have with a new ballot design?
About this project
1 of 5 similar tests
10 participants
1 locations, 1 day
2 moderators + 1 others
Self-defined tasks
Observation
Post-task questionnaire
We learned about
Navigation on the ballot
Instructions
Results
Found pattern of errors for
one interaction
Time on task
13. 13
Exploring new ideas
Can we extend a design to new audiences?
http://anywhereballot.org
http://civicdesigning.org/featured-story/rapid-responsive-radical-the-anywhere-ballot-is-born/
About this project
Research project
Paper prototyping
Low literacy
We learned about
What they understsand
Navigation
Instructions
Interactions
Results
Simplified everything for a
clear, universal design
14. 14
Answers quickly and inexpensively
Can we find participants "where they are?"
http://anywhereballot.org
http://civicdesigning.org/featured-story/rapid-responsive-radical-the-anywhere-ballot-is-born/
About this project
13 participants
Observed and interviewed as
they filled in a new voter
registration form
We learned about
How people filled in the form
Results
Improved voter registration
forms
This project and poster were done by Jenny Greeve, the Design Fellow at the
Washington Department of Elections
15. 15
Learning about comprehension
Can voters explain and act on that understanding?
About this project
20 participants
4 locations, 1 day
8 moderators
Self-defined tasks
Minimal observation
We learned about
Navigation on the ballot
Instructions
Mental models of counting
Whether there‟s a problem
Results
Qualitative insights
16. 16
Gauging opinion
Which accessible voting system do voters prefer?
About this project
496 surveys; multi-disabilit
11 systems
Multiday “voting fair”
They learned about
Voter assessment of usability
Voter comments about
accessibility
Differences by disability
Results
Statistical analysis survey
results
Qualitative analysis of
comments
This was a project of the Oregon Department of Elections
18. 18
Look for a balance of methods by approach and
context of use
19. 19
Early in
Project
Many Users
Few Users
Comparative
benchmarking
Summative testing
Card sorting
Satisfaction surveys
Market research
Iterative evaluation of
prototypes
Evaluation
of Designs
Ethnography
Use the right tool at the right time
Stakeholder
interviews
User observation
Walkthroughs
Expert
Reviews
A-B Testing
20. 20
Look for the 'cadence' of what kind of usability work
you do at different times
Seeing the Elephant: Defragmenting User Research by Lou Rosenfeld
http://alistapart.com/article/seeing-the-elephant-defragmenting-user-research
26. 26
Storytelling for User Experience
with Kevin Brooks
Global UX
with Daniel Szuc
A Web for Everyone
with Sarah Horton
http://rosenfeldmedia.com/books/a-web-for-everyone/
Hinweis der Redaktion
A tool to help you focus on the attributes that matterAccount for what the user brings to the tableIdentify both center and outliers
The 'Anywhere Ballot'Designed through iterative testing with low literacy participants
On April 4th, 2005, citizens with disabilities from all over the state tested voting systems and completed surveys documenting their reactions to each system. A total of 496 surveys were completed.The survey results on the following page will be one of several factors used to help the Oregon Secretary of State decide which voting system or systems to purchase to give citizens with disabilities the opportunity to vote privately and independently. Survey was a modified SUS – 12 questionsI would like to use this voting system in an election. There were too many steps in using this voting system.I thought this voting system was easy to use. The instructions for this voting system were difficult to understand. The buttons or touch screen were easy to use. It was hard to move around the ballot with this system. I think that most people could learn to use this voting system very quickly.I found this voting system awkward or difficult to use. I felt very confident that my vote was cast correctly with this voting system. I would need help each time I used this voting system. The ballot text was easy to read or hear. Casting my vote was easy.