Research workshop study_methods_v5

Dan Berlin – Experience Research Director, @banderlin
Susan Mercer – Senior Experience Researcher, @susanamercer
July 9, 2013
UXPA International 2013 Conference Tutorial
Research Methods Roulette
CHOOSING THE RIGHT METHODS FOR YOUR PROJECT

Today’s Tutorial
 Introductions
 Study goals
 Study types
 Best practices
 Create-a-study exercises
 Discussion about projects at your work
2

Hello, I’m Susan Mercer
 BA and MSc in Geophysics
 19 years in software and web UI and UX design
 Developer
 Designer
 Web Producer
 Product Manager
 Researcher
 MS Human Factors, Bentley University
 Twitter: @susanAmercer
3

Hi! I’m Dan Berlin
4
 BA in psychology from Brandeis University
 Studies focused on visual space perception
 Seven years in technical support
 Sat as a participant for a usability study for a product I was working on
 Realized that user experience (UX) work is the perfect combination of computers and psychology
 Went to Bentley U. to earn an MBA and MS in Human Factors in Information Design
 Two years at an interactive agency performing usability and neuromarketing research
 Then did some freelance UX consulting for about a year
 Two years as an Experience Research Director in Mad*Pow’s Boston office

Introductions
 Name
 Company/school
 Position
 Years experience
 Research interests
 Favorite place visited
5

Study Goals
 Where are you in the project?
 How much time do you have?
 How much of a budget do you have?
 Is there an existing design?
 What are you trying to learn?
7

Study Types
Formative
 Interviews
 Focus Groups
 Collaging
 Ethnography
 Surveys
 Diary Studies
 Card-Sorting
9
Evaluative
 In-Person Usability
 Remote Usability
 Unmoderated Usability
 Desirability Testing
 Eye-Tracking

User Interviews
What is a User Interview:
 A focused one-on-one conversation
 In-person or via telephone
 Familiar format for both participant and interviewer
 Requires little training for researcher
11

User Interviews
When to use user interviews:
 Rich, detailed data is desired
 Exploring early in the project
 Looking for qualitative direction or to understand a new domain
 A “large” sample size is not needed
 Looking to uncover current behaviors, motivations, and desires
 Users are accessible and available for in-depth discussion
12

User Interviews
Types of Interviews:
 Structured Interview
 Set of closed-ended, factual questions similar to a survey
 Frequently done for US Census
 Unstructured Interview
 Questions are open-ended and not asked in a particular order
 Participant mostly guides the conversation
 Valuable for new concept discussions
 Semi-Structured Interview
 Combination of the above
 Most common format
13

User Interviews
14
Pros Cons
Gather rich, detailed data Qualitative data takes longer to analyze
Small sample size yields effective data Questions may not completely consistent
from participant to participant
Can be done remotely via telephone Need to manage talkative and quiet
participants
Good for exploratory, formative
research
Not as efficient when there are many
different demographics with truly
different needs
Fairly cheap to conduct

User Interviews
Interview Format:
 Introduction
 Context for research, research goals
 Warm-Up Questions
 Start easy – confirm/gather a few relevant demographics
 First Question
 Open-ended with lots of room for participant to think
 Body
 Follow their lead
 Ask follow-on questions that align with other questions in the moderator’s guide
 Get more specific as you go
 End with “Magic Wand” or “What Else?” Question
15

User Interviews
What to look for:
 User’s picture of the world (mental model)
 What triggers cause which effects
 Their perception of how things work
 What things are grouped together as similar, or are different?
 Stories of how things usually happen
 Stories of exceptions – when things go really good or bad
 Contradictions in stories
 With same participant
 From one participant to another
 Barriers to use
16

User Interviews
How to write good interview questions:
 Open-Ended Questions
 Who, What, When, Where, Why, How
 Start with Broad Phrases
 Let the participant guide the discussion at first to what they think is important
 Examples:
 “Tell me about how you prepare printed materials for your classes.”
 “How do you know when it is time to refill your prescriptions?”
 “Describe what you do to get ready to go watch a soccer match.”
 “What healthy changes do you want to make or have been making?”
17

User Interviews
How to write good interview questions:
 Follow-up prompts
 I usually follow a broad question with a series of prompts
 These remind me what things to ask about if the participant does not mention them
Example:
 Tell me about sharing the printer and copier with others.
 What works?
 What doesn’t work?
 Do you need an ID, card, or code to use the shared printer/copier?
 Does your printer keep track of how many copies/$ you have left on your “budget”?
 Do you know how much it costs per page to print?
 Is the school actively trying to manage printing costs? If so, how?
 Is the school trying to “go green”?
18

User Interviews
Timing and Resources:
 Writing a Screener: 8-10 hours
 Recruiting: 2-3 weeks
 Writing Interview Guide: 16-24 hours
 Conducting Interviews: 1-2 hours per interview
 About 5 participants per demographic group
 Double your resource estimate if using a separate note taker
 If no note taker, add extra time for review or transcription of notes
 Analysis: 1-3 hours per interview
 Report Preparation: 8-16 hours
19

User Interviews
Additional Resources:
Understanding Your Users: A practical guide to user requirements. Catherine Courage & Kathy
Baxter, Morgan Kaufmann, 2005. Chapter 7
Interviewing Users: How to uncover compelling insights. Steve Portigal, Rosenfeld Media, 2013.
Mental Models: Aligning Design Strategy with Human Behavior. Indi Young, Rosenfeld Media, 2008.
Chapter 7
20

User Interviews Exercise
Your client wants to build a mobile app to manage grocery lists for busy families.
They want something that will be really useful, so they want to understand how
people make their grocery lists.
 Write 5 questions for the start of an interview script (5 mins)
 Divide into pairs and interview your partner (5 mins each)
 Group Discussion (5 mins)
21

Focus Groups
What is a Focus Group:
 A moderated 1-2 hour session with typically 6-8 participants
 Traditional Focus Group = sit around a conference table and have participants
respond to a stimulus; ask a lot of questions
 Modern Focus Group should include more activities
 Gets participants moving around
 Can collect different types of data
 Card sorting, affinity diagramming, role playing, etc.
23

Focus Groups
When to use Focus Groups:
 You want to get open-ended input from a large(r) number of participants
 Early in the project – during idea-generation
 To get user input on project goals and product features
 To understand current user behavior so that your product can fit seamlessly
into users’ lives
 Best if you have some sort of stimulus, anything for them to react to
 Participants are available in a geographic location
24

Focus Groups
25
Pros Cons
Quickly get feedback from a large
number of participants
Mix of participant personalities can be
very hit or miss
Group dynamic can help foster
creativity
Conversation tends to be a lot less
structured
Different activities can be conducted in
one research session
Participants can have a positive or
negative impact on other participants

Focus Groups
Focus Group Format:
 Participant & moderator introductions
 Ask participants to share a personal, but relevant, piece of information
 First Question
 Open-ended, to get a feel for what is important to the participants
 Align the moderator’s guide with the conversation flow
 Be familiar enough the guide so that you can tell what has been already covered in
the course of the conversation
 Include activities to get participants moving around
 But you don’t necessarily have to discuss the results – just collect the data in the
activity and move on
 Don’t forget to take a break!
26

Focus Groups
What to look for:
 Shared mental models amongst participants
 And where their mental models differ
 How participants expect things to work
 And how these align with business & project goals
 Participants’ feelings
 What will delight users and what will make them sad
 Stories of exceptions – when things go really good or bad
 Contradictions in stories
 With same participant & between participants
 Barriers to use
27

Focus Groups
 About 6 participants per group
 Double your resource estimate to account for note taker
 If no note taker, add extra time for review or transcription of notes
28

Focus Groups
Focus Groups: A Practical Guide for Applied Research. Richard A. Krueger & Mary
Anne Casey, SAGE Publications, 2009.
Focus Groups: Theory and Practice. David Stewart, et. al., SAGE Publications, 2007.
29

Ethnography
What is ethnography:
 Observing users in their natural environment while they interact with the
product/interface in question
 Big “E” Ethnography
 “A research strategy that allows researchers to explore and examine the cultures and
societies that are a fundamental part of the human experience” (Murchison)
 Generally involves immersion of the researcher within the culture to be studied
 Longitudinal and can take many years
 Little “e” ethnography
 UX Researchers perform little “e” ethnography
 Typically involves a period of observation followed by an interview
 Focuses on how users use the product/interface and their overall environment
31

Ethnography
When to use ethnography:
 If you feel that the user’s environment is likely to influence how they use the
product
 Examining how visiting home nurses use their laptops when visiting new patients
 To understand current user behavior so that your product can fit seamlessly
into users’ lives
 Seeing how patients set up and use home dialysis machines in an effort to increase
adherence
 The product or use case just doesn’t translate well to a lab setting
 Observing how people use metro ticket machines, particularly when there is a line
behind them in rush hour
 Participants are available in a centralized geographic location
32

Ethnography
33
Pros Cons
See natural interruptions and
interactions with others and other
products
Logistics of setting up interviews in
someone’s home/office
Users will interact with their own items,
including cheat sheets or other “coping
mechanisms”
Safety concerns for participants and
researchers
Can yield “aha” moments when see the
real way something is used vs. the
intended way
Interviewing many people requires a lot
of time
On-site recording logistics

Ethnography
Ethnography Format:
 Natural observation
 Ask participants go about their tasks as they normally would
 Silently observe and take notes
 Specific tasks
 If they have not done so already, have participants do tasks that you would like to
specifically observe
 Interview
 After observing the participant, sit down for a 30-40 minute interview about what
you just observed
34

Ethnography
What to look for:
 What information users look for and where they find it
 Who and what they interact with to get the task done
 The order in which things happen, and whether that is important
 Distractions, barriers, and interruptions that the participants encounter
35

Ethnography
 Should always have a moderator and a note taker/filmer
 Two researchers will make the participant more comfortable
36

Ethnography
Ethnography in UX, Nathanael Boehm, uxmatters.com,
http://www.uxmatters.com/mt/archives/2010/06/ethnography-in-ux.php
Practical Ethnography: A guide to doing ethnography in the private sector, Sam
Ladner, Expected soon. http://www.practicalethnography.com/
Ethnography Essentials: Designing, conducting, and presenting your research,
Julian M. Murchison. Josey-Bass, 2010.
37

Ethnography Exercise
During our break, observe the other tutorial attendees and their relationship to
technical devices
 Before Break – Write down observation points (5 mins)
 After Break – Group discussion (10-15 minutes)
38

Collaging
What is Collaging:
A creative activity to help uncover thoughts and attitudes.
Participants are given photographs, large paper and art supplies and asked to
create a collage about a central theme. They then explain their collage.
 Can be done one-on-one or as a focus group activity
 Can make some participants conscience of their lack of creative skills
 Their storytelling about the college is the key
 Is often followed by a short interview
40

Collaging
When to use collaging:
 Rich, detailed data about attitudes, values, or motivations is desired
 Trying to uncover information that is not at the top of participants’ minds
 To answer “Why?” questions
 Exploring early in the project
 Looking for qualitative direction or to understand a new domain
 When you don’t know exactly what questions to ask
41

Collaging
How to do it:
1. Prepare materials
 Collect a large number of color photographs
 Magazines, online sources, take your own
 Include some that are related to your topic
 Include many that are not related to your topic
 Provide same photographs to all participants
 Provide a blank canvas (post-it flipcharts work well)
 Provide glue sticks, tape, markers, scissors, post-its, etc.
42

Collaging
How to do it:
2. Define focus
 Provide a central theme or question for them to demonstrate by creating
their collage.
 Focus them on your research topic, but do not be too specific or you may
miss interesting results
 Example:
 Goal: Uncover attitudes, values, and current behaviors about ‘Saving Energy’.
 Instruction - “Create a collage about how you save energy”
• Too specific, and assumes that they try to save energy (they may not)
 Better Instruction - “Create a collage about what ‘Saving Energy’ means to you.”
• Found many reasons about why they do not do energy saving behaviors, even
though they feel that they should.
43

Collaging
How to do it:
3. Provide instructions
 Emphasize that creative skills are not important – that is why they have
photographs
 They can use however few or as many as they would like
 They can do whatever they want with photographs
 They can use all supplies however they want.
4. Time Limit
 Provide enough time for them to create something interesting, but not too
long so they can’t be perfect.
 Run a pilot to test timing if you’re not sure
 15 – 30 minutes is usually sufficient
44

Collaging
How to do it:
5. Have them tell their story
 Ask them to explain their collage – what does it mean to them?
 Listen
 Make notes for “why?” follow-on questions
6. Discuss
 Ask follow-on questions about areas of interest
 If in focus group, ask questions to group to get others’ perspectives as well
45

Collaging
46
Pros Cons
Gather rich, detailed data Qualitative data takes longer to analyze
Small sample size required Questions are not consistent from
participant to participant
Good for exploratory research Need to manage talkative and quiet
participants
Good for uncovering values,
motivations, attitudes that are not top-
of-mind – participants may not even be
consciously aware of them
Some participants are not as open to
creative exercises as others
Good seeding activity for focus groups Preparation can be time-consuming
Can help some participants verbalize
thoughts

Collaging
What to look for:
 User’s picture of the world (mental model)
 What triggers cause which effects
 Their perception of how things work
 What things are grouped together as similar, or are different?
 What do they think about these topics?
 Do they agree or disagree?
 With society? With their family/friends? With marketing messages?
 How do these topics relate to their life?
 Why they do things – or why not?
 How do these pictures or topics make them feel?
47

Collaging
 Conducting Collaging Sessions: 1-2 hours per session
 Allow twice as long for discussion as for collage creation
 Use shorter timings and fewer pictures if doing this as part of a focus group
48

Collaging
Collaging: Getting answers to the questions you don’t know to ask. Kyle Soucy,
Smashing Magazine, February 6, 2012.
http://uxdesign.smashingmagazine.com/2012/02/06/collaging-getting-answers-
questions-you-dont-know-ask/
McKay, D., Cunningham, S. J., Thomson, K. Exploring the user experience through
collage. CHINZ ’06 Proceedings of the 7th ACM SIGCHI New Zealand’s chapter’s
international conference on Computer-human interaction: design centered HCI,
p. 109-115.
49

Surveys
What is a Survey:
A structured set of questions to be completed by a participant
 No moderation
 On paper
 Online (more common)
 Usually large numbers of participants
 Familiar format for participant
 Usually easy for participants to complete
51

Surveys
When to use surveys:
 A “large” sample size is desired
 Or, you want to collect the same information from all interview participants
 You know what questions to ask (and what likely answers are)
 Want to verify observations from qualitative methods
 Want to measure prevalence of certain responses within a population
 Need quantitative results to satisfy stakeholders
 Need to collect a lot of data in a short time (online surveys)
52

Surveys
How to write good survey questions:
 Single or Multiple-Choice Questions
 Provide a set number of responses – select one or multiple
 For single-choice, make sure all options are mutually exclusive
 Provide an “out” – “I don’t do <X>”, “I don’t know”, “Other”
 Be wary of using “Other (please specify)” – coding and analyzing responses takes time
 Example:
 What type of mobile phone do you own?
 iPhone (Apple)
 Android (Samsung, HTC, Motorola, LG, etc.)
 Windows (Nokia Lumia, HTC Windows Phone, etc.)
 None of these
 I do not own a mobile phone
53

Surveys
How to write survey questions:
 Rating questions / Likert Scales
 Ask participants to rate something on a scale of 1 to 5
Example:
 What motivates you to improve your health?
54
1
Not at all
motivating
2 3
Neutral
4 5
Very
motivating
My family     
My friends     
I feel better when I’m healthy     
I want to live as long as possible     

Surveys
How to write survey questions:
 Frequency questions
 Make frequency intervals specific so that participants know how to answer accurately
Example:
 How often do you do the following on your mobile phone?
55
Less than
weekly
Once a
week
A few
times a
week
Once a
day
Multiple
times a
day
Send email     
Surf the web     
Share pictures & videos     
Use social media apps     

Surveys
Things to be aware of:
 Response Biases
 Social Desirability bias – participants answer questions based on what they think they
should answer so that they are viewed in a more favorable light
• Provide anonymity, and minimize socially intrusive questions
 Central Tendency bias – participants tend to avoid the middle option
 Extreme Avoidance bias – participants tend to avoid the extreme options
 Acquiescence bias – participants tend to agree with the statement as written
 Response Rates
 Anywhere from 5% to 50%, depending on source of participants
 Run a Pilot test
 Get about 10% of desired responses and look at the data in detail – does it make sense?
56

Surveys
Increasing Your Response Rate:
 Keep it short! (20-25 questions / 20 minutes)
 Make it easy to complete
 Keep number of open-ended questions short
 Include personalized introduction
 Follow up with reminders
 Offer an incentive:
 All completions receive a $5 gift card
 All completions are entered into a drawing for a $100 gift card
57

Surveys
How will you analyze your data?
 Plan this in advance!
 What will you do with the answers to each question?
 If you don’t know…do you really need to ask it?
 How will you graph it?
 Are you collecting data in the right format?
 Example: Age
 How will you “code” open-ended responses?
 Have you allotted enough analysis time?
 What “cross-tabs” do you want?
 By gender? By age group? By participant type?
58

Surveys
59
Pros Cons
Easy to gather a large number of
responses
Need to know what questions to ask
(and possible responses)
Quantitative findings are relatively
easy to analyze
Qualitative findings are subjective and
time-consuming to code and tally
Can be done in a short period of
time (online surveys) In a survey, EVERY WORD COUNTS

Surveys
Survey Format:
 Screener Questions
 If you’re screening out participants to ensure only a certain population completes the
survey, put these questions first
 Interview Questions
 Start more generic, then get more specific
 Group them logically
 Contact Information
 Gather this at the end, and only if you need it for incentive purposes.
 Ask as little as possible
 Thank You Statement
60

Surveys
Recruiting Participants:
 Customer lists
 Newsletter subscribers
 Social Media
 Website intercepts
 Survey Tool User Research Panel (e.g. SurveyMonkey Audience)
 Hire a Recruiter
61

Surveys
Online Survey Tools:
 Survey Monkey http://www.surveymonkey.com/
 Zoomerang http://www.zoomerang.com/
 Survey Gizmo http://www.surveygizmo.com/
 Constant Contact http://www.constantcontact.com
62

Surveys
 Writing a Survey: 8-24 hours
 Conducting Survey: Few days to two weeks
 Depends on recruiting method and incidence rate of qualified participants
 Watch over time and send reminders, adjust recruiting methods if necessary
 Analysis: 2-4 days for 20 question survey
63

Surveys
Understanding Your Users: A practical guide to user requirements. Catherine Courage &
Kathy Baxter, Morgan Kaufmann, 2005. Chapter 7
Designing & Conducting Survey Research: A Comprehensive Guide. Louis M. Rea & Richard A.
Parker, Jossey-Bass, 2005.
64

Surveys Exercise
Revisiting our Interview Exercise:
Your client wants to build a mobile app to manage grocery lists for busy families.
They want something that will be really useful, so they want to understand how
people make their grocery lists.
 Take the questions you wrote for the interview exercise, and change them to
survey questions.
 Write a total of 5-7 survey questions
65

Diary Studies
What is a Diary Study:
 A semi-longitudinal study used gather user behavior over time
 Users are asked to keep a diary for a certain topic
 When they interact with a device/website/etc.
 Health, food, study habits, banking habits, etc.
 Data collection may be structured, unstructured, or a combination
 Can be time-intensive and laborious, but worth the effort
67

Diary Studies
When to use a diary study:
 When a single research session will not truly capture users’ interactions
 When the users’ environment plays a role in how they use the interface
 If there is plenty of time in the project timeline
 Can be used for formative or summative research
 Formative: capture how they currently do something so that something new can be
built that will fill a void
 Summative: capture how they interact with a newly designed product
68

Diary Studies
69
Pros Cons
Can capture users’ data in their
natural setting
Reliant on participants keeping up
with their diaries
Users will be interacting with their
own items Must keep participants on track
Can capture data over time Management of the data collection
mechanism
Must have a lot of time in the project
timeline

Diary Studies
Dairy Study Format:
 Takes place over the course of weeks or months
 Participants are provided with a mechanism with which to collect data and are
asked to submit them at certain intervals
 Mixture of closed and open-ended questions
 Mechanism may be a spreadsheet, an online form, or something more intricate
 Researcher must be diligent about following-up with participants who:
 Do not fill out complete diaries
 Do not submit diaries on check-in dates
70

Diary Studies
You will need to:
 Be organized!
 Plan out how you will track participant progress and check-ins
 Plan out the questions that you want participants to answer over time
 Be prepared to perform interim analysis as the study progresses
 Be creative on how to engage and incent participants
 The data will only be as good as what the participants provide
 Have foresight and be malleable
 Diary studies should go on for weeks or months
 Project trajectories can change – your study may have to as well
71

Diary Studies
What to look for:
 What prompted them to perform a task
 What information participants look for and where they find it
 Who they interact with to get the task done
 Distractions, barriers, and interruptions encountered
 Task ease and satisfaction ratings
72

Diary Studies
 Writing Study Guide: 16-24 hours
 Data Collection: 1-6 months (or more)
 Managing Study: varies greatly, but plan for more than you think
 Could be 40 hours per month, could be 8
 Depends on the study size and breath
 Analysis: 2-3 hours per participant
73

Diary Studies
Observing the User Experience: A Practitioner’s Guide to User Research, Elizabeth Goodman,
Mike Kuniavsky, & Andrea Moed, Morgan Kaufman, 2012.
Diary Studies in HCI & Psychology, Demetrios Karis, UPA Boston 2011 Conference.
http://www.slideshare.net/UPABoston/diary-studies-in-hci-psychology
74

Diary Studies Exercise
Group Discussion:
 Think about your current or past work situations
 What issues have you encountered that would be good for diary studies?
 How would you have captured information?
 What would you hope to learn?
75

Card Sorting
What is Card Sorting:
An organizational activity where participants sort topics into categories and
optionally label them. Provides great input into information architecture, menu
design, and labeling.
 Usually an individual exercise
 In-person or using online tools (more common)
 Both Quantitative and Qualitative
 Online tools provide much quantitative analysis
 However, good analysis requires understanding the domain and interpretation
77

Card Sorting
When to use card sorting:
 To understand how users categorize information
 To understand users’ language for related items
 As input to create a brand new menu hierarchy
 As input to gather feedback about an existing menu hierarchy
 When the business or design team keep arguing about menu options and
wording – get some data!
 Card sorting itself relates how, not why users group items
 Consider pairing with a qualitative method, such as interviews or focus groups
78

Card Sorting
Types of Card Sorts:
 Open Card Sort
 Users sort individual items into categories
 Then users name the categories
 Good as a first-round
 Closed Card Sort
 Users sort individual items into pre-defined categories
 Good as a second-round to test categories determined from open sort
 Good to test where new items should go in an existing hierarchy
79

Card Sorting
80
Pros Cons
Provides good input on how users
think about organization of items
Difficult to test more than one level of
nesting at a time
Easy to conduct online with current
tools
Understand how users categorize
information, but not why
Current online tools make analysis
much easier
Analysis requires insight into content
and interpretation – it is not
straightforward quantitative analysis
Can get large number of participants
easily

Card Sorting
Running a Card Sort:
 Determine content items
 Content or functionality, not both
 Items are unique, but related enough to suggest groupings
 Roughly 30-100 items
 Content should be representative
 Use words that participants understand
 Create cards / Set up online tool
 Run test
 Analyze findings
81

Card Sorting
Questions to ask:
 Rationale for grouping
 Which item is the best example in each group?
 Which items did you have trouble sorting? Why?
 If there are similar groups, how are they different?
82

Card Sorting
How many participants?
 Depends on your goal and number of items
 For broad feedback and 40 cards, 5-8 may be enough if you see convergence
 For detailed feedback and 99 cards, may not see convergence at 30 participants
 Online, it’s easy to do a lot, but do you need a lot?
 30-50 completes should give you as good results as you’re likely to get
 Assuming 5% completion, 50 completes requires 1,000 invitations
83

Card Sorting
Analyzing your card sort:
 Read Donna Spencer’s Book!
 Standardize Labels
 It’s common for people to use very similar, but slightly different names for labels.
 Example:
 “Billing”, “billing”, “billings”, “My bills” all convey the same general topic
 So, change them all to “Billing” for analysis purposes
84

Card Sorting
 Similarity Matrix
 How often 2 cards are grouped together
 Darker color is more often
 Dark triangles define strong groupings
85

Card Sorting
 Category Matrix
 How many times a card is put in a category
86

Card Sorting
 Dendograms or “Tree Maps”
 Show what percentage of people agree with a particular grouping
87

Card Sorting
 Creating content items: 8-10 hours
 Conducting In-Person Card Sorts: 1-2 hours per participant
 Conducting Online Card Sort: up to 1 week, depending on recruitment
methods, incentive, size of card sort
 Analysis: 2-4 days depending on number of cards, complexity of sort
88

Card Sorting
Online Card Sorting Tools:
Optimal Sort by Optimal Workshop
http://www.optimalworkshop.com/optimalsort.htm
Web Sort by UX Punk http://uxpunk.com/websort/
Simple Card Sort http://www.simplecardsort.com/
User Zoom http://www.userzoom.com/software/research-capabilities/card-
sorting/
…and more!
89

Card Sorting
Card Sorting: Designing usable categories. Donna Spencer, Rosenfeld Media, 2009.
Measuring the User Experience: Collecting, analyzing, and presenting usability metrics. Tom
Tullis and Bill Albert, Morgan Kaufmann, 2008. Chapter 9.
90

Card Sorting Exercise
Be a Participant:
 Go to http://www.optimalworkshop.com/optimalsort.htm
 Click on “Try a demo as a participant” (10 mins)
 Be a Researcher:
 Click on “See the results as a UX researcher” and explore (10 mins)
91

Desirability Testing
What is Desirability Testing:
A method for exploring participants’ emotional reactions to different design
options.
 Can be used to select a visual design which most closely aligns with a
company’s branding strategy
 Can be used to evaluate a product design vs. competition to uncover
differences in brand perceptions or user experience
 Can be used to evaluate a single design for emotional response
 In-person or online
 Can be done quickly at the end of another research activity
 Requires little training for researcher
93

When to use desirability testing:
 Different design options have been created
 Qualitative, emotional reactions to visual designs are desired
 Users are accessible (in-person or remote, moderated) and available for
discussion
94

Method:
 Determine set of 50-60 adjectives to use
 Approximately 60% positive / 40% negative connotations
 Adjectives should be easily understood by participants
 Determine designs to be tested
 Visual design compositions
 Live websites or prototypes
 Prepare words
 Cards: Create one card per word (print labels and put on index cards)
 Adjective Sheet: Create one sheet per stimulus with words alphabetized
 Online: Prepare adjective sheet in slide format
95

Method:
 Show design A
 Ask participant to select words which best describe design
 “Small number of words” – provide flexibility on number, but < 7 is ideal
 Ask participant why they selected those specific words
 Repeat for all designs
 Consider randomizing design order by participant
96

Which adjectives best describe the pages you’ve seen?
97
Accessible Dated Hard to Use Simplistic
Appealing Dull Helpful Sophisticated
Approachable Easy to Use Impersonal Sterile
Boring Effective Innovative Stimulating
Busy Efficient Inspiring Straight Forward
Clean Energetic Intimidating Stressful
Clear Engaging Intuitive Time-consuming
Comfortable Enthusiastic Inviting Time-saving
Compelling Exciting Motivating Too Technical
Complex Familiar Organized Trustworthy
Confident Fast Overwhelming Understandable
Confusing Flexible Patronizing Usable
Convenient Fresh Predictable Useful
Creative Friendly Professional Valuable
Cutting Edge Frustrating Reliable

98
Pros Cons
Gather qualitative feedback on
emotional response to designs
Choice of adjectives could introduce
subtle bias
Quick method for testing Generally, relatively small samples
Easily combined with other research
methods
Data analysis is relatively
straightforward

Reporting Formats:
 Word Cloud
 http://www.wordle.net
 Positive / Negative word analysis
 Compare % of Positive / Negative words for each design
 Compare words that are common across designs, and those that are missing
99
25
10
37
4
21
15
0
5
10
15
20
25
30
35
40
Posi ve Nega ve Posi ve Nega ve Posi ve Nega ve
Design A Design B Design C
#ofwords
Word Sen ment for Different Designs

Benedeck, J. & Miner, T. (2002) Measuring Desirability: New methods for evaluating
desirability in a usability lab setting. Proceedings of Usability Professionals Association,
2003, 8-12.
http://www.pagepipe.com/pdf/microsoft-desirability.pdf
Rapid Desirability Testing: A Case Study, Michael Hawley, uxmatters.com,
http://www.uxmatters.com/mt/archives/2010/02/rapid-desirability-testing-a-case-
study.php
Users play cards We keep score Magic Results, presentation by Carol Barnum and Laura
Palmer
http://www.slideshare.net/cbarnum/barnum-palmer-stc-2011-presentation
100

CHOOSING THE RIGHT METHOD(S)
101

How to Choose the Right Method(s)
1. Make sure your research goals are clear
 Know what you want to learn
 Know what you will do with the information you gather
 Know what decisions the business needs to make
2. Know your constraints
 Timeline?
 Resources?
 Budget?
 Access to Users?
3. Create a Methods Chart
 The answers will become clear
102

Methods Chart
103
Method Pros Cons Candidate
Method 1 • List the advantages for
this method for this
specific project
• List the disadvantages
for this method for this
specific project
Yes / No –
Would this method be a
good candidate for this
specific project
Method 2
Method 3
Etc…

Methods Chart Example
 Example Project Goals:
 Have 3 concepts for behavior change application. Which one is the best to develop
for maximum global appeal?
 Client wants answers as quickly as possible
 Client wants large numbers to provide confidence behind decision
 Client wants data collection in 5 countries around the world
 Designers want to know why or why not users chose each concept, to provide
additional design direction
 User population – adults who own a smart phone and want to get healthier
104

105
Method Pros Cons Candidate
Interviews • Good for capturing
motivations for behavior
change
• Can get good qualitative
details on why users
prefer each concept or
not
• Good to explore issues to
fine tune survey
questions
• Can be done via phone
(get broad geographic
sample)
• Can be done quickly
• Small numbers – client
wants large numbers
Y – would be good as a
qualitative method to pair
with a larger quantitative
method
Focus Groups • Good for qualitative
information gathering
• Could generate some
interesting conversations
about behavior change
motivations
• Wouldn’t get as much
detail as interviews
• Concern about group
think when evaluating
concepts
• Concern about not
sharing details of
personal goals in front of
others
N – interviews would be
better for qualitative
Survey • Good for large numbers
• Easily replicated across
different
countries/languages
• Can be done online for
broad geographic
distribution
• Can be done quickly
• Unclear what exact
questions to ask
• Doesn’t provide detailed
insights into qualitative
topics
Y – good paired with
qualitative method

 Example Research Plan:
 Conduct telephone interviews in US with 12-15 participants
 Create online survey in US for 300 participants
 Use international research partners to conduct 12-15 interviews in 4 countries
 Use international research partners to conduct survey for 300 participants in same 4
countries
 Why this approach works:
 Interviews provide input into questions for survey
 Survey provides large numbers; interviews provide in-depth insight
 Both are commonly used methods and easy to replicate in different countries
 Both methods can be done quickly
106

Study Type Exercise A
 Your client makes software for electric utilities websites
 Consumers can view, monitor, and pay their electric bills
 Utility companies want consumers to conserve energy, particularly on high-
demand days
 Consumers want to pay less per month for their electricity bill
 The client wants to understand customer needs and desires better, but
doesn’t know where to start
 You’ve been asked to write a project proposal
1. Define your research goals (what do you want to focus on learning?)
2. Create a methods chart for this project
3. Which method(s) would you recommend and why?
107

Study Type Exercise B
 Your client is a major credit card company
 Their data show that young adults (ages 18-25) are not using credit cards
compared to older generations when they were at that same age
 The client wants to understand why they are not using credit cards so that
they can improve their products and/or market to them better
 You’ve been asked to write a project proposal
1. Define your research goals (what do you want to focus on learning?)
2. Create a methods chart for this project
3. Which method(s) would you recommend and why?
108

ETHICS OF USING HUMAN SUBJECTS
109

Humans as Research Subjects
As a researcher, you are responsible for your participants
 Physical safety
 Do disabled participants need special accommodations?
 Did the participant make it out of the office during the fire drill?
 I better ensure that broken glass is fully cleaned before the next session.
 This medical device uses a needle, I better have a First Aid kit handy.
 Mental well-being
 Is this line of questioning making the participant uncomfortable?
 Am I sitting at an appropriate distance to build rapport, but also to make the
participant comfortable?
110

Ensure to cover the “informed” portion of “informed consent”
 Does your paperwork and session introduction cover the things that
participants need to know?
111
Before starting a session, ensure that participants:
Know the purpose of the study Know the study procedure
Give permission to record audio/video Know that their name will be kept confidential
Are informed of any foreseeable risks
Know that they are welcome to a break whenever
needed
Know that they can withdraw at any time
without penalty
Are reminded of the agreed upon compensation
Are informed of who to contact for
questions
Are offered to receive a copy of the informed consent
form
Know that the moderator is a neutral
observer
Are reminded that we are investigating the stimulus,
not the participant

When working with children, there are special considerations
 Parent/guardian signs the consent form; child signs an assent form
 Consent forms are written in the third-person (“You are participating…”)
 Assent forms are written in the first-person (“I am participating…”)
 Prepare to give children additional breaks – every 30 minutes
 This medical device uses a needle, I better have a First Aid kit handy.
 Decide at what point parents should participate
 At what point does it become co-discovery and is that okay?
 You will treat a 17 year old different than an 8 year old
 Be cognizant of the study protocol to ensure you are consistent in your data capture
112
Source: http://evilstaring.com/2011/06/08/amazingly-evil-child-3/

When in doubt, contact an IRB
 You can hire an Independent (or Institutional) Review Board (IRB) to review your study
methodologies and facilities
 This is especially important if conducting studies on medical devices for FDA approval
 IRBs will review each study methodology to determine where participants may come
into harms way, and how the researcher will mitigate
 Most UX research does not involve much possible harm
 But usability studies are part of the FDA approval process
 IRBs will also inspect the research facilities to ensure proper safety in the lab space and
proper file keeping for ongoing studies
 They will also review all documentation to ensure that foreseeable risks are properly
conveyed to participants
 This includes: screeners, recruitment flyers, consent & assent forms, study guides, etc.
113

Recruiting
Recruiting the right participants for your study is paramount
 Identify the demographics of the potential users as a very first step
 Your first call is to the recruiter to see how much recruiting & incentives will cost
 Agency folks: this is done when scoping the project, not during kick-off
 Prepare a VERY DETAILED screener document that outlines the characteristics of your
desired study participants
115
Screener
Section
Purpose
Study Information • Study name, type, dates, session times, location, contact, and incentive (and who is paying it)
Recruitment
Requirements
• A verbal description of who you are looking for
• 12 total participants
• 6 who prefer Sharktopus
• 6 who prefer Sharknado
Opening Script • The exact words that the recruiter will use to introduce your study
Screening
Questions
• The series of questions that will determine if the respondent meets the criteria for the study
• Should include a question at the end to test the respondent’s articulation (“What is your
favorite website?”)
Invitation
• The exact words that the recruiter will use to invite the respondent to participate in your study
• Should include what the participant should expect and the next steps

Writing Good Screener Questions
 Be specific to get the right participants
 Be as specific as you need to
 Relax constraints where they are not needed
 Example:
 18-25 (2M/2F); 26-45 (4M/4F); 46-65 (4M/4F)
 50% mix Male/Female & even mix of age ranges (18-25, 26-45, 46-65)
 Focus on specific behaviors
 Ask frequency questions, not yes/no questions
 “How many times have you purchased items online in the last 3 months?”
 0-3 (TERMINATE)
 4-8 (Recruit 50%)
 9 or more (Recruit 50%)
117

 Beware of professional participants!
 Don’t make it obvious what answers you are looking for
 How do you currently interact with your pharmacy to order and refill prescriptions?
• In Person
• Phone (talking to a live person)
• Phone (using an automated system)
• Email
• Website
• Mobile application
(TERMINATE if Website is not mentioned)
 Use a few open-ended questions
 NO: “Do you work for XYZ corporation?” (If Yes, TERMINATE)
 YES: “What company do you work for?” (Do not read answers) (If XYZ corporation, TERMINATE)
118

 Always use an “articulateness” question
 The exact question is not important, but instructing the recruiter how to determine
articulateness is.
PLEASE BE VERY ATTENTIVE TO THE RESPONSE FOR THIS QUESTION – IT IS VERY IMPORTANT. This
question is to determine how creative and articulate and outgoing the respondent is. If the
respondent is not able to express themselves, is not articulate, or accent is too heavy to be clearly
understood when answering the question, TERMINATE. Do not in any way imply that termination is
because of the celebrity they chose.
If you had an opportunity to go out to dinner with any person, alive or not, who would it be and briefly
describe why you chose that person.
119

Structure of an Research Session
121
Trust
Comfort
Interview
Build Rapport
Neutral
Observation
Define Social
Interaction Rules
Comfortable
Conversation
Be Accepting
Thanks
Greeting Introduction
Interview /
Evaluation
Wrap-Up

Greeting – Building Rapport
 Be a friendly person
 Smile
 Use their name
 Be a good listener
 Make the other person feel important – and do so sincerely
A person's name is to that person the
sweetest and most important sound in
any language.
- Dale Carnegie

 Small talk – find common ground
 Safe topics: travel to office, traffic, weather, local sports
 Avoid asking direct questions
 Listen and look for shared experiences

 Be empathetic
 Apologize if they had trouble finding the office
 “Oh, it’s raining there? It is here too. I hate rainy days.”
 Show you understand their point of view
 “I can understand that…”
 “I can see that…”
 “That does sound very frustrating…”

Introduction – Define Social Interaction Rules
 Describe the session
 List the activities
 Describe the roles
 Social niceties do not apply
 You’re not emotionally involved in the design/project
 There are no right or wrong answers
 Your job is to get honest opinions

Interview/Evaluation – Be Accepting
 Watch your reactions
 Don’t show surprise
 May make them think that they are giving a wrong
answer
 Don’t overly agree
 May make them think that they are giving the right
answer
 Don’t be negative
 Watch your tone – stay neutral and accepting
 Try not to laugh

Interview/Evaluation – Be Accepting
 Be yourself
 No one is perfectly neutral
 Recover gracefully and move on
 “Perfect” – “That’s the level of detailed feedback we’re looking for.”
 “Interesting!” – “I haven’t heard that perspective yet, tell me more.”
 (something surprising) – “I can understand that.”
 Interject some Rapport-building comments when needed
 Quiet or uncomfortable participants
 “I hate it when that happens.”, “I can imagine that was challenging”, etc.
 Again, showing that you are human like them

Interview/Evaluation – Comfortable Conversation
What is a comfortable conversation?
 Conversational cues and turn-taking are expected
 Acknowledgement tokens – “Uh huh”, etc.
 Encourage the continuation of the other speaker’s talk
 Usually implies that the other speaker’s prior talk is incomplete
Source: 1 Drummond and Hopper, 1993.

Interview/Evaluation – Neutral Observation
 Some may introduce bias
 “Oh!”, “Interesting” – indicating unexpected answer
 “Yes”, “Perfect”, “Great” – indicating agreement
 “Hmmm.”, “Really?” – indicating disagreement
 Notice that tone is key
 Neutral is best
 “Mhmm”, “Uh huh”, “Continue”, “Tell me more”, “OK”
 “Mhmm” or “Uh huh” vs. silence  interviewees saying 31% more phrases. 1
 Body Language
 Head nodding while participant is speaking  interviewees speak 50% longer. 2
Source: [1] Matarazzo et. al., 1964, [2] Matarazzo et al., 1963

Interview/Evaluation – Neutral Observation
 Really listen
 Pay attention – stay in the moment
 Look at the participant
 Take notes if you can
 Be quiet - give them time to say what they need to

3. Interview/Evaluation – Neutral Observation
 Be quiet!
 Most agreements happen immediately. Most people delay before disagreeing.1
 If you don’t respond to their answer, it encourages them to talk more
 People often delay speaking before disagreeing – give them time
 Some people are uncomfortable with silence, so they will keep talking
 “People speak in paragraphs.” (Steve Portugal)
 The best way to stay neutral 
Source: 1 Goodwin and Heritage, 1990.

Classic Books on Moderating
132

Coming Soon!
New book on Moderating
 Donna Tedesco, Fiona Tranquada
 Book coming this Fall
 Follow @ModSurvivalUX
133

Note Taking
Proper planning for taking notes is very underrated
 The organization and thoroughness of your notes will dictate the ease with which you
will create the final report
 Organized, complete notes = easy reporting
 Disorganized, incomplete notes = back to the video you go (ewww!)
 Make your notes grid once your study & moderator’s guide is complete
 Give each question and subquestion its own row
 Put each participant in a new column (or vice-versa, if you like)
 DO NOT put each participant/task in a new worksheet
 Use data validation for quantitative data
 Task ease ratings, task success, multiple choice questions, etc.
 Always include an extra “Why?” cell for data validated cells (to capture qualitative data related to
the question)
135

Note Taking
136
• Do not include the
participant name in your
notes grid
• Include a date/time cell to
best align with the videos
• Visually separate sections
of the study
• Fill data validated cells with
a light color
• Hide columns when have
moved on to the next
participant

Note Taking
137
• Ensure these two
checkboxes are checked
• In a new worksheet, make your lists
• One “question type” per column
• Choose “List”
• Choose the cells that have
your list
• Press the Data Validation button
• Can also be found in the “Format”
menu
How to do Data Validation in Excel

Documentation
 Each study should have:
139
Document What It Has
Recruitment Screener & Schedule
• Defines, in great detail, the demographics of your desired
participants
• Mainly comprised of the questions to be asked of potential
participants and the actions to be taken based on their
responses (continue, terminate, skip to Q4, etc.)
Consent / Assent Forms
• Outlines study, risks, confidentiality, use of information, etc.
• Participant signature indicates agreement to participate
Study & Moderator’s Guide
• The goals and methodology of the study
• The questions that the moderator will ask the participants and
any other session activities
Note Takers Grid
• A spreadsheet that is based on the Study & Moderator’s guide
• Should have a row for each question and a column for each
participant
Findings Document
• The intermediary document that aids in the transfer from raw
notes to final presentation
• One categorized and prioritized finding (not suggestion) per
row
Reporting Template
• If you have a rough idea of the report before starting the
study, you will ensure to collect the proper data
• Report can be based upon the Study & Moderator’s guide, but
should tell the story of the collected data

Documentation
CX Partners website. http://www.cxpartners.co.uk/ux-resources/
Observing the User Experience: A Practitioner’s Guide to User Research, Elizabeth Goodman,
Mike Kuniavsky, & Andrea Moed, Morgan Kaufman, 2012.
A Practical Guide to Usability Testing. Joseph S. Dumas & Janice C. Redish, Intellect. 2009.
140

Report only observable findings
 As researchers, our goal is to report on observable findings – that which
participants said or did
 Don’t say that participants “thought” something or “felt” an emotion
 Unless you state, for example, that “Participants said that they felt sad upon seeing
the interface.”
 Report only what participants said or did/didn’t do
142
Research Reporting Best Practices
Good verbs for reporting participant observations
Said Mentioned
Reported Clicked
Voiced concerns Voiced confusion

143
Preparing the raw findings
 Start with a list of findings in Excel
 “Finding”: Something that you observed the participant(s) do or say that offers insight into
the user experience
 Categorize the findings by severity: positive, high, medium, & low
 If using a 1-5 scale, 1 & 2 = high priority, 3 = medium, 4 & 5 = low priority
 Positive: point out what users liked and what worked on the interface
 High: mission-critical interactions that do not work as designed or prevent task completion
 Medium: problems that would be good to fix, but could wait until a second release
 Low: feature requests and other items that do not greatly affect the user experience
 Subcategorize the findings
 Could be based upon location within the interface
• Use “Global” for findings that affect the entire interface
 Or possibly task number or other categorization, as the project warrants
 Optional: if you want to include the number of times a finding was encountered, include a
column with the participant number
 Will help if you get interrupted, so you don’t double-count
 This will help organize the data so that it can be summarized into a PowerPoint report

144
Sample findings sheet
 Once all the findings are documents in Excel, then you can start building your PPT report

Preparing the report document
1) Recap the study goals and methodology
145

2) Indicate the demographics of study participants
146

3) In the executive summary, replace long lists of findings with sentences that
state how the system performed as a whole
 Focus on findings that are most relevant to business goals
 Should only include key findings – save the details for later in the report
147

Transferring findings – Tone and general guidelines
 The report text should indicate that the study happened in the past (i.e. past
tense)
 The report should be written in the active voice – ensure that the subject of
the sentence performs the action
 No: “The ‘send’ button was quickly found by participants”
 Yes: “Participants quickly found the ‘send’ button”
 Ensure to include contextual verbatim participant quotes, with attribution to
participant number
 NEVER include users’ names in the report… ever
 Can include the list of stakeholders’ names to indicate with whom we spoke
 But, never attribute quotes or findings to a particular stakeholder
148

Transferring findings – Detailed findings
 Include the actual number of participants who experienced a key finding
 E.g.: 9 of 13 participants clicked the “next” button
 Always report numbers of participants in relation to the total number of
participants
 E.g.: 9 of 13 participants or 9/13 participants
 Avoid using percentages – it’s easy for clients to read more into “69% of
participants” than “9 of 13 participants”
 Percentages will often make client forget the context of small, qualitative samples
 Try to void using numerical summary words, such as “some, few, many,
several, etc.”
 Give actual numbers, if possible
149

Transferring findings – Graphs
 Graphs should always have:
 Data labels (outside end)
 Horizontal and vertical (rotated) axis labels
 A title
 A properly labeled legend (if a legend is needed)
 Correct y-axis values
 Starts at 0 for interval data
 Starts at the lowest possible choice for ordinal data (e.g. a 1-5 scale)
 Correct sorting, where appropriate
 If graphing counts of participants who did something specific, include the total count
of participants in the title for context (e.g. “<Title> (n=13)”)
 Use 2-D graphs; avoid using 3-D graphs
150

151
3 3
1
2 2
0
1
2
3
4
5
6
7
1 Very Easy 2 Easy 3 Neutral 4 Difficult 5 Very Difficult
#ofParcipants
Ra ng Scale
Ease of Use Ra ng: Task 2
Title
(Above graph)
Rotated vertical
y-axis label
Proper y-axis
interval
Data label
(outside-end)
Horizontal
X-axis label
Transferring findings – Graphs

Transferring findings – Making suggestions
 When pointing out a potential usability problem, always include a suggestion to fix it
 Before the pages with screenshots of the interface with boxes and arrows, define the
colors of the boxes and arrows
 Green = positive; yellow = low priority; orange = medium; red = high
 When making design suggestions, ensure that they are suggestions and not orders
 No: “Make this box green”
 Yes: “Consider making the button bigger to increase visibility”
 When making design suggestions, make it clear WHAT needs to change, not HOW
(unless it’s very obvious)
 No: “Make this button green”
 Yes: “Consider making this button more visually prominent”
 At the end of the report, include a prioritized list of action items
152

153
From spreadsheet to report
 The goals is to goal from an organized list to a PPT with useful and
informative boxes and arrows

Usability Studies
 The usability study is a primary evaluative UX research technique
 Different types of usability studies for different research goals:
 Formative
 Early in the design process
 Gathers mostly qualitative feedback using think-aloud protocol
 Can test paper prototypes, click-through wireframes, or functional prototypes
 A more informal, conversational moderation style is acceptable
 Not suitable for measuring task times
 Summative
 After development – generally a fully functional application
 Gathers mostly quantitative feedback using pure task observation and recording
 Formal moderation style is required
 Moderator is generally not in the room
 Often used for benchmarking or comparing across versions of software
155

Usability Studies
 All usability studies are TASK BASED
 Give participants tasks to perform on important parts of the interface
 Observe not only how they perform, but also how they react
 Different formats of usability studies for different purposes
 In-person – moderated & lab based
 Remote – moderated over the phone and a screen-share
 Unmoderated – automated via tools such as UserZoom or Loop11
156

Usability Studies
 In-person usability study
 Allows moderators to observe participants’ non-verbal behavior
 Moderators can build a better rapport with the participants, putting participants at
ease and generating richer responses
 Used when representative users can be found in a centralized geographic location
 No technical hurdles for the participants
 Typically 7-10 tasks and 12-15 participants in 1-hour sessions
 Anything can be tested: website, software application, mobile app, physical device,
documents, paper prototypes
157

Usability Studies
 Remote usability study
 Conference call and screen-sharing technology has made it quite easy to conduct
remote usability studies
 Use a service such as WebEx or GoToMeeting
 Conference call, screen-share, and recording
 Can recruit across a much wider geographic area
 Participants must have access to a phone and computer connected to the Internet
 And be comfortable enough with computers to install the screen-share program
 Typically 7-10 tasks with 12-15 participants in 1-hour sessions
 Can only test websites and software interfaces; only “on screen” interfaces
158

Usability Studies
 In-person vs. Remote usability studies
 They are both qualitative research techniques that are conducted with 12-15
participants
 It all comes down to geographic area, budget, and ease of finding participants
 Better to run an in-person study when feasible
 Run a remote usability study if:
 The budget doesn’t allow for travel
 You want a distributed representation of geographic areas
 Participants are hard to find – wider geography gives a bigger pool
 You don’t have or want to pay for lab facilities
159

Usability Studies
 Automated usability study
 Used to gather quantitative usability data
 Users participate from their homes/office
 They are presented with a task to perform on an interface
 After performing the task, the tool asks a few questions, then moves to the next task
 Can perform automated studies with large (statistically significant) numbers of
participants
 Typically 3-5 tasks per study and 100-300 participants
 Typically need a live website to test using automated tools. Some will allow
screenshots
160

Usability Studies
 Rely on Automated usability studies when:
 You need to collect quantitative data
 You only have a few important tasks to investigate
 You can have a wide population perform the study
 Or have a very large list from which to recruit participants
 You need feedback fast (some can get results in days)
 Automated usability study metrics
161
Description What it Measures
Time on task (efficiency) How long did it take users to complete the tasks?
Error/success rate
(effectiveness)
Did users successfully complete the tasks?
Ease of use ratings
(satisfaction)
How easy or difficult did users perceive the task?
SUS, NPS, or other surveys
SUS = System Usability Score
NPS = Net Promoter Score
Clickstreams Where did users click?

Usability Studies
 What makes for a good task?
 Something the participant is likely to do
 Clearly defined outcome and ending
 Provides enough context so that participant understands desired outcome
 Does not use words on the interface or jargon
 Simply worded and concise
 Does not lead the participant
 Examples:
 You want to buy a pair of Converse high-tops for yourself. Go to converse.com, find a
pair that you like, and buy them. Stop when you are asked for your credit card
number.
 You requested a refill of a prescription at cvs.com. You’re running errands near your
CVS. Use your smartphone to see if your prescription is ready yet.
162

Usability Studies
 Tips for In-Person Usability Studies:
 Make recording devices as unobtrusive as possible
 Sit beside and slightly behind participant
 Use a second monitor turned towards you to watch what their on-screen actions, if
needed
 If possible, have a note taker watch and take notes
 If you have observers in the other room, at the end of the session, go into the
observation room to get final questions for participant
163

Usability Studies
 Tips for Remote Usability Studies:
 Email participant a meeting invitation with instructions 24 hours in advance
 Allow at least 10 minutes for participant to login to screen sharing tool
 Be familiar with your screen sharing tool and be able to walk participant through their
login experience
 Observers:
 Ask them to put their phones on mute
 Most tools allow meeting host to mute them if they do not
 One “online meeting” for all sessions vs. individual meetings for each session
164

Usability Studies
 Tips for Automated Usability Studies:
 Prioritize your tasks – ensure you are testing your more important tasks
 Double-check that your rating scales are laid out as expected
 Proof-read your tasks and questions… again and again
 Run a pilot test – ensure the data is telling you what you want to understand
 Think through your analysis after the pilot test
 Make sure you are capturing information in a useful format before you get 300 responses
165

Usability Testing
A Practical Guide to Usability Testing. Joseph S. Dumas & Janice C. Redish, Intellect. 2009.
Beyond the Usability Lab: Conducting large-scale online user experience studies. William
Albert, Donna Tedesco and Tom Tullis, Morgan Kaufmann, 2009.
166

Eye Tracking
What is a Eye Tracking:
 Eye tracking technology allows us to see where participants are looking
 Typically incorporated in the course of a usability study
 “Eye tracking is a tool, it is not a methodology.” – Dante Murphy, Digitas Health
 Can be used for both on and off-screen interfaces
168
Tobii GlassesSMI RED

Eye Tracking
When to use eye tracking:
 An age-old question
 Best used when comparing designs
 Are participants looking at different elements of the design?
 Are participants exhibiting different or inefficient gaze patterns with the
different design?
 Should be one or two tasks during the course of a usability study
 Always present the eye tracking stimuli in the course of performing a task
169

Eye Tracking
170
Pros Cons
Clients love quantitative data Expensive equipment
Can capture additional user behavior
data
No set methodology or analysis standards
Can be paired with other temporal data
capture (such as biometrics)
Tough to nail down ROI on eye tracking
Data analysis is relatively
straightforward
Incremental data

Eye Tracking
Eye Tracking Format (on-screen):
 Moderator should be in another room (“voice of God” technique)
 Moderator in the testing room is distracting
 Present the task on the screen
 Then have the participants look at a “+” in the center of the screen for 5 seconds –
gets them at the same starting point
 When the participant has completed the task, go to the next stimulus in the
eye tracking software
 Eye tracking is best used to compare designs
 Have participants perform the same task on the different designs
171

Eye Tracking
You will need to:
 Break the screen into Areas of Interest (AOIs)
 Headline, main navigation, content, side
navigation, etc.
 Compare metrics between AOIs and between
stimuli
 Don’t let heat maps and gaze plots dictate
your findings
 Export the raw data to Excel and do your own
analysis
172
Areas of Interest

Eye Tracking
Get familiar with the different eye tracking metrics:
173
Description What it Measures
Overall # of fixations Increased overall fixations indicate less efficient search
Fixations per AOI Increased fixations indicate increase noticeability or importance
Overall fixation duration Increased fixation duration indicates confusion or engagement
Time to first fixation on-target
Faster time to first fixation on-target indicates increased
noticeability
Percentage of participants fixating
an area of interest
Higher percentages indicate increased noticeability
Fixation/saccade ratio Higher ratio indicates less searching (more processing)

Eye Tracking
174
Heat Map Gaze Plot
• # of fixations for all
participants
• Order of fixations for one
participant
15
8
5
2
10
15
5
4
0
2
4
6
8
10
12
14
16
Area 1 Area 2 Area 3 Area 4
Area of Interest
#offixaons
# of fixa ons
Design 1
Design 2
Data!
• Comparison of the number
of fixations between AOIs

Eye Tracking
Eye Tracking: A comprehensive guide to methods and measures. Kenneth
Holmqvist, et. al., Oxford University Press, 2011.
Eyetracking Web Usability. Jakob Nielsen & Kara Pernice, New Riders, 2010.
175

Usability Study Exercise A
The US Postal Service redesigned their website earlier this year and wants to make sure
that the https://www.usps.com/ site is optimized for shipping and tracking packages for
the upcoming Winter Holiday season.
1. Create a usability test script for a Formative usability test. Write a scenario, and three
tasks. Include up to 3 additional prompts for more detail.
2. Create a usability test script for a Summative usability test. Write a scenario and three
tasks. Include the list of metrics that you wish to capture and what they will tell you
about the design.
177

Usability Study Exercise B
Amazon.com wants to make sure that their Wish List feature is optimized for the
upcoming Winter Holiday season.
1. Create a usability test script for a Formative usability test. Write a scenario, and three
tasks. Include up to 3 additional prompts for more detail.
2. Create a usability test script for a Summative usability test. Write a scenario and three
tasks. Include the list of metrics that you wish to capture and what they will tell you
about the design.
178

Research workshop study_methods_v5

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Research workshop study_methods_v5

Ähnlich wie Research workshop study_methods_v5 (20)

Mehr von UXPA International

Mehr von UXPA International (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Research workshop study_methods_v5

Hinweis der Redaktion