Introduction to usability studies, presented to Baobab Health Trust

Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Usability Studies and Empirical Studies
Harry Hochheiser
University of Pittsburgh
Department of Biomedical Informatics
harryh@pitt.edu
+1 412 648 9300
Attribution-ShareAlike

CC BY-SA

Outline
Usability Studies

Think-Aloud

Summative Studies

Empirical Studies

Beyond Inspections
Inspections won't tell you which problems users will face in
action

Might not identify mental models and confusions

..finding out where things go wrong.

Baobab Health, March 2014

Harry Hochheiser, harryh@pitt.edu
Edit Title
No bright dividing line in process
Design
Fully-functional

Prototype

Paper

Prototype
Release
Usability Inspections
Usability Studies
Empirical User Studies, Case
Studies, Longitudinal Studies,
Acceptance Tests
Low
cost,
low
validity Higher
cost,
validity

Formative Usability Studies: Goals
• Generally, to understand if the proposed design supports completion of
intended tasks
• Be specific -
• Tasks and users
• Define success
• User Satisfaction?
• Do users like the tool?
• What are the important metrics?

Formative Usability Studies: Tasks
• Representative and specific
• What would users do?
• Realistic – given available time and resources
• Appropriate for assessment of goals
• Possibly some user-defined/suggested
• Particularly if participants were informants in earlier
requirements-gathering

Which Tasks?
Bad: Give this a try?

Better: Try to send an email, find a contact, and file a
response

Still better:

Detailed scenario with multiple actions that required coordinated use of
diverse components of an application's functionality
Formative Usability Studies:

Formative Usability Studies: Conditions
• Usability Lab
• Two-way mirrors/separate rooms
• Workspace
• Online?
• Often video and/or audio-recorded
• Screen-capture
• Logs and instrumented software
• Goal: Ecological Validity

Formative Usability Studies:  
Measures
• Key question to answer: “can users complete tasks”?
• Generally, lists of usability problems
• Description of difficulty
• Severity
• Task completion times – depending on methods
• Error rates?
• User Satisfaction
• Quantitative results for measuring success
• Not comparative

Methodology
• Define Scope
• Users complete tasks
• Researchers observe process
• What happens?
• What goes right? What goes wrong?
• Note difficulties, confusions?
• Record – audio/video, screen capture

Participants
• Somewhat representative of likely users
• Willing guinea-pigs
• Need folks who are patient, willing to deal with problems
• Well-motivated
• Compensated
• Eager to use the tool
• Small numbers – repeat until diminishing returns
• How many?

Nielsen – why you only need to test with 5 users

http://www.useit.com/alertbox/20000319.html

Hwang & Salvendy (2010) – maybe need 10 +/- 2
Only 5 users – or maybe not

Two approaches
• Observation
•Subject performs tasks, researchers observe
• Ecological validity, but no insight into users
• “Think aloud”
•User describes mental state and goals

Think-Aloud Protocols
• User describes what they are doing and why as they try to complete a task
• Describe both goals and steps taken to achieve those goals.
• Observe
• Confusions – when steps taken don't lead to expected results
• Misinterpretations – when choices don't lead to expected outcomes
• Goal: identify both micro- and macro-level usability concerns
• Strong similarities with contextual inquiry, but..
• Focus specifically on tool
• Participant encouraged to narrat
• Evaluator generally doesn’t ask questions

Caveats
• Think-aloud is harder than it might sound
• What is the role of the investigator?
• How much feedback to provide?
• Very Little
• What (if anything) do you say when the user runs into problems?
• Not much
• What if it's a system that you built?
• How to identify/describe a usability problem?

Think-Aloud Protocols: A Comparison of Three Think-Aloud
Protocols for use in Testing Data-Dissemination Web Sites for
Usability Olmsted-Hawala, et al. 2010
"... it is recommended that rather than writing a vague statement such as 'we
had participants think aloud,' practitioners need to document their type of
TA protocol more completely, including the kind and frequency of probing.”

Reporting Usability Problems 
adapted from Mack & Montaniz, 1994
• Breakdowns in goal-directed behavior
• Correct action, noticeable effort
• To find
• To execute
• Confused by consequence
• Correct action, confusing outcome
• Incorrect action requires recovery
• Problem tangles
• Qualitative analysis by interface interactions
• Objects and actions
• Higher-level categorization of interface interactions
Gulf of Execution
Gulf of Evaluation

Reporting Usability Problems 
adapted from Mack & Montaniz, 1994
• Inferring possible causes of problems
• Problem reports
• Design-relevant descriptions
• Quantitative analysis of problems by severity

Analysis
• Challenge – identify problems at the right level of granularity?
• When does a series of related difficulties lead to a need for
redesign?
• What if these difficulties come from different tasks?
• When appropriate, relate usability observations back to contextual inquiry or
other earlier investigations
• Does the implementation fail to line up with the needs?
• Perhaps in some unforeseen manner?

Analysis
• Multiple observers
• Calculate agreement metrics?
• Use audio, video, transcripts to illustrate difficulties
• Particularly useful for demonstrating problems to implementation
folks
• Rate problem severity
• Which are show-stoppers and which are nuisances?
• Which require redesign vs. small changes?
• Must prioritize...

Completion – Summative User Studies
• Demonstrate successful execution of system
• With respect to
• Alternative system – even if straw man
• Stated performance goals – Acceptance Tests
• Generally empirical

Completion – Summative 
Studies of systems in use
• Case studies
• Descriptions of individual deployments
• Qualitative
• Longitudinal study of ongoing use
• Collect data regarding impact
• Similar to case studies, but potentially more quantitative.
• Use observations and interviews to see what works?

After system is complete

More realistic conditions?

Acceptance tests

Usability tests aimed at measuring success

Does the tool do what the client wants

• 95% task completion rate within 3 minutes, etc.?

Client has clearer idea – not just “user friendly”
Summative Tests

What: Empirical Studies
• Quantitative measure of some aspect of successful system use
• Task completion time (faster is better)
• Error rate
• Learnability
• Retention
• User satisfaction...
• Quality of output?

Tension in empirical studies
• Metrics that are easy to measure may not be most interesting
• Task completion time
• Error rate
• Great for repetitive data entry tasks, less so for complex tasks
• Analytics, writing...

Empirical User Studies: Goals
• I have two interfaces – A and B.
• Which is better? and how much better?
• Want to determine if there is a measurable, consistent difference in
• Task completion times
• Error rates
• Learnability
• Memorability
• Satisfaction

Running Example: Menu Structures
• Hierarchical Menu structures
• Multiple possibilities for any number of leaf nodes
• Broad/Shallow vs. Narrow/Deep
• which is faster?

Hypothesis
• Testable Theory about the world
• Galileo: The rate at which falling items fall is independent of their weight
• Menus
• Users will be able to find items more quickly with broad/shallow trees
than with narrow/deep trees.
• Often stated as a “null hypothesis” that you expect will be disproven:
• There will be no difference in task performance time between broad/shallow
trees and narrow/deep trees.

Background/Context
• Controlled experiments from cognitive psychology
• State a testable/falsifiable hypothesis
• Identify a small number of independent variables to manipulate
• hold all else constant
• choose dependent variables
• assign users to groups
• collect data
• statistically analyze & model

Other goals
• Strive for
• removal of bias
• replicable results
• Generalizable theory that can inform future work
• or, demonstrable evidence of preference for one design over another.

Empirical User Studies: Tasks
• Use variants of the design to complete some meaningful operation
• Usually relatively close-ended, well-defined
• Relatively clear success/failure

Empirical User Studies: Conditions
• Lab-like?
• Simulated realistic conditions?

Independent Variables
• What are you going to test?
• Condition that is “independent” of results
• independent of user's behaviors
• independent of what you're measuring.
• one of 2 (or 3 or 4) things you're comparing.
• can arise from subjects being classified into groups
• Examples
• Galileo: dropping a feather vs. bowling ball
• Menu structures – broad/shallow vs. narrow/deep

Dependent variable
• Values that hypothesis test
• falling time
• task performance time, etc.
• May have more than one
• Goal: show that changes in independent variable lead to measurable, reliable
changes in dependent variables.
• With multiple independent variables, look for interactions
• Differences between interfaces increase with differences in task
complexity

Controls
• In order to reliably say that independent variables are responsible for
changes in dependent variables, we must control for possible confounds
• Control – keep other possible factors constant for each condition/value of
independent variables
• types of users, contexts, network speeds, computing environments
• confound – uncontrolled factor that could lead to an alternate explanation
for the results
• What happens if you don’t control as much as possible?
• Confounds, not independent variables, may be the cause of changes in
dependent variables.

Examples of Controls
• Galileo:
• windy day vs. not windy?
• Menus
• network speed/delays? (do everything on one machine)
• skills of users? (more on participant selection later)
• font size, display information, etc.?

• Related to controls
• Experimenter can introduce biases that might influence outcomes
• Instructions?
• Choice of participants?
• more on this in a moment
• Protocols
• prepare scripts ahead of time
• Learning Effects?
Bias
Thanks to Jinjuan Feng for figure

Between-Groups vs. Within-Groups
Design
• How do you assign participants to conditions?
• All people do all tasks/cells?
• Within-groups – compare within groups of individuals.
• one group of test participants
• Certain people for certain cells?
• between groups – compare between groups of individuals
• 2 or more groups
• Mixed models

Between Groups
• Pros
• Simpler design
• Avoid learning effect
• Don't have to worry about ordering
• Cons
• may need more participants
• to get enough data for statistical tests
• to avoid influence of some individuals.

Within-Groups
• Pros:
• Can be more powerful statistically
• same person uses each of multiple interfaces
• Fewer Participants
• Cons
• Learning effects require appropriate randomization of tasks/
interfaces
• Fatigue is possible

Mixed Models
• Elements of both
• 3 different interfaces
• Want to compare performance of different groups
• Docs vs. Nurses?
• Each interface a within-subject experiment
• Across professions is between-subjects.

Other Challenges
• Ordering tasks?
• How many?
• Want to avoid fatigue, boredom, and expense of long sessions
• How many users?
• 20 or more?
• Variability among subjects
• May be unforeseen.
• Bi-modal distribution of education or computer experience?
• Training materials
• Run a pilot

Procedure
• Users conduct tasks

• Measure

• record task completion times

• errors

• etc.

• Now what?

• Analyze data to see if there is support for the hypothesis

• alternatively, if the

Hypothesis Testing
• Not about proof or disproof

• Instead, examine data

• Find likelihood that the data occurred randomly if the null
hypothesis is true

• If this is small, say that we have support for the hypothesis

Data, Stats, and R
• Need to talk about

• data distributions

• statistical analyses

• to do hypothesis testing

• Tools:

• R - r-project.org

• R-Studio - rstudio.org

Sampling
• Data sets come from some ideal universe

• all possible task performance times for a given menu selection
task

• Compare two samples with given means and deviations

• Are they really different? Or do they just appear different by
chance?

• Statistical testing gives us a p-value

• probability that differences are random chance

• low values are significant

The key questions
• Given two sets of measurements, or samples, did they come
from the same underlying source or distribution

• x = [29 33 89 56 86 85 7 84 67 78 59 28 10 76 11 12 97 61
66 9 40 95 90 4 31 18 24 48 45 82]

• y = [51 3 10 11 5 90 87 13 64 86 67 98 12 55 56 80 59 63 94
93 25 4 79 52 36 73 99 22 62 2]

• mean(x) = 50.67, sd(x)=31.01

• mean(y) = 51.7, sd(y) = 33.26

• are they from the same distribution?

Boxplot
• Show quartiles

• Are they the same?

“Normal” distributions
• Given mean and standard deviation (measure of variation)

• 95% of area under curve within 2 standard deviations

• If you take many samples from a space

• Their averages will go to a normal distribution

• Statistical testing -> comparison of distributions.

Histograms
Run a subset of a
population, 1000 times

get average of each subset

Normal distribution

Hypothesis testing
• Test probability that there is no difference between two
distributions

• Possible errors

• Type 1 Error: α - reject null hypothesis when it is true

• believe there is a difference when there is none

• False positive

• Type 2 Error: β- accept null when false

• believe no difference when there is

• False Negative

Significance Levels and Errors

• Highly significants ( p <0.001)

• Don't believe there is a difference unless it's really clear

• low chance of false positive – Type 1

• Greater chance of false of false negative /Type 2

• Less significant (p < 0.05)

• More ready to believe there is a difference

• More false positive/type 1 errors

• fewer type 2 errors

• Usually use p=0.05 as cut-off.

Type 1 and Type 2 errors
Type 1 error

reject the null hypothesis when it is, in fact, true

Type 2 error

accept the null hypothesis when it is, in fact, false

Decision
Reality

Statistical Methods -Crash Course
• Comparisons of samples

• t-tests: 2 alternatives to compares

• ANOVA: > 2 alternatives, multiple independent variables

• Correlation

• Regression

t-test
• x = [29 33 89 56 86 85 7 84 67 78 59 28 10 76 11 12 97 61
66 9 40 95 90 4 31 18 24 48 45 82]

• y = [51 3 10 11 5 90 87 13 64 86 67 98 12 55 56 80 59 63 94
93 25 4 79 52 36 73 99 22 62 2]

• t.test(x,y)

Results
Welch Two Sample t-test
data: x and y
t = -0.1245, df = 57.72, p-value = 0.9014
alternative hypothesis: true difference in means is not equal
to 0
95 percent confidence interval:
-17.65522 15.58855
sample estimates:
mean of x mean of y
50.66667 51.70000

xkcd on significance testing

http://xkcd.com/882/

Correlation

Correlation
• Attributing causality

• a correlation does not imply cause and effect

• cause may be due to a third “hidden” variable related to
both other variables

• drawing strong conclusion from small numbers

• unreliable with small groups

Regression
Calculates a line of “best ﬁt”

Use the value of one variable to predict the value of the other

r2=.67, p < 0.01

r=.82

Be careful

http://xkcd.com/552/

User Modeling 
Hourcade, et al. 2004
Predict performance characteristics?

Calculate index of difficulty

similar to MT = a + b log2
(A/W+1)

Linear regression to see how well it fits

Longitudinal use
• Lab studies are artificial
• Many tools used over time.
• use and understanding evolve
• Longitudinal studies look at usage over time
• Expensive, but better data
• Techniques
• Interviews, usability tests with multiple sessions, continuous data
logging, Instrumented software, Diaries

Case Studies
• In-depth work with small number of users
• Multiple sessions
• Describe scenarios
• Illustrate use of tool to accomplish goals
• Good for novel designs, expert users
• Formative evaluation – can be used to gather requirements
• Summative – show validity of idea
• Possibly less compelling than usability evaluations.

Informed Consent
• Research must be done in a way that protects participants

• Principles

• Respect for persons

• Beneficence – minimize possible harms, maximize possible benefits

• Justice – costs and benefits should not be limited to certain
populations

• Institutional Review Board (IRB) – approves experiments
and requires signatures on “informed consent” form.

• Crucial for responsible research

Other Metrics
What if task completion time is not the most important
metric?

Insight?

Automated Usability Testing
Possible for defined criteria

Text complexity?

Accessibility

WCAG

Section 508

Example: wave.webaim.org.

Log File Analysis
• Use clickstream and usage data to study actual use

• Which parts of the system are people using?

• Which are they not using?

• Are they going in circles?

• Are they having problems?

• Rich data, but hard to interpret

• particularly without observations or interviews to
provide context.

Shortcomings of User Studies
What happens in the lab may not be reflected in real use

Deployment/post-mortem, etc.

Case studies, qualitative work

How can we meaningfully evaluate a system in use

… when deployment presents a significant expense...

Introduction to usability studies, presented to Baobab Health Trust

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (14)

Ähnlich wie Introduction to usability studies, presented to Baobab Health Trust

Ähnlich wie Introduction to usability studies, presented to Baobab Health Trust (20)

Mehr von Harry Hochheiser

Mehr von Harry Hochheiser (8)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Introduction to usability studies, presented to Baobab Health Trust