UI/UX Foundations - Research

UI/UX Foundations:
Research & Analysis
Meg Kurdziolek and Karen Tang

Your Goals
What would you like to learn today?

Our Goals
We want you to…
talk confidently to UX researchers
critically understand research presented to you
conduct basic UX research on your own
have a basis to continue learning about UX research

Activity (setup)
Which of these problems do you feel strongly about?
Pittsburgh public transportation
Food delivery in Pittsburgh
Finding family-friendly activities in Pittsburgh
Pittsburgh public schools

Agenda
09:00 - 09:20 Breakfast and Introductions
09:20 - 09:30 User-Centered Design
09:30 - 10:10 Surveys, Diary Studies, Interviews
10:10 - 10:55 Usability Studies, Field Studies
10:55 - 11:25 A/B Testing, Log Analysis
11:25 - 12:00 Adapting Your Methods
12:00 - 12:30 LUNCH
12:30 - 01:00 Interpreting Your Data
01:00 - 01:40 Special Topics: Dark UX Patterns
01:40 - 01:50 Case Studies
01:50 - 02:00 Group reflections & wrap-up Q&A

User-centered design (n.) - a framework of processes
in which the needs, wants, and limitations of end
users of a product are given extensive attention at
each stage of the design process.
https://en.wikipedia.org/wiki/User-centered_design

Design Process
Refine
BuildLearn

UX Research & Design
Refine Build Learn

Refine Build Learn
Data Collection, Validation, Evaluation

Refine Build Learn
A/B Testing
Contextual Inquiry
Diary Studies
Ethnography
Field Studies
Focus Groups
Hallway Testing
Heuristic Evaluation
Interviews
Lab Testing
Log Analysis
Remote Testing
Think-Aloud
Surveys
User Observation

Need Finding
Reﬁne Build Learn
A/B Testing
Contextual Inquiry
Diary Studies
Ethnography
Field Studies
Focus Groups
Hallway Testing
Interviews
Lab Testing
Log Analysis
Remote Testing
Think-Aloud
Surveys
User Observation

Surveys
http://www.baerpm.com/blog/what-a-customer-survey-can-do-for-your-business/

are good
for learning:
overall impressions
who your users are
(demographics)
outstanding opinions
who’s might participate
in further research
Surveys

Common Survey Example
How likely is it that you’d recommend [brand] to a friend?
Not at
all Likely
Neutral
Very
Likely
1 2 3 4 5 6 7 8 9 10

Net Promoter Score
NPS: “research has shown that your NPS® acts as a leading indicator of
growth. If your organization’s NPS is higher than those of your
competitors, you will likely outperform the market…”
https://www.netpromoter.com/know/

Surveys: Pros and Cons
Beneﬁts
Cheap (in $ and time)
Easy to recruit participants
May receive high response rate
Easy to analyze
Limitations
Limited in type & scope of data
Question interpretation issues
Response bias
Easy to misinterpret or *over*-
interpret results

Silly Survey Example
Do you like to eat lunch alone?
Yes
No

Diary Studies
http://hciresearch4.hcii.cs.cmu.edu/M-HCI/2011/BOA-PlanningTools/

are good for….
day-to-day habits & patterns
when, how, and why they
use your product
reflections on real problems
encountered and how they
were solved
Diary Studies

Diary Studies: Pros and Cons
Beneﬁts
A longitudinal scope of data
Get a look at the mundane,
every-day interactions and
behaviors
Limitations
Costly (in $ and time)
Difficult to recruit
participants (& high attrition)
Relies on self-report

Example Diary Study
Radar: Intellicast vs.
Weather Underground

are good for….
user’s background
their use of technology
their goals and motivations
their pain points
what problems need to be
addressed or solve
Interviews

Interviews: Pros and Cons
Beneﬁts
Cheap (in $)
Can target specific users or be
opportunistic
Can engage with users
personally
Can get the answer to lots of
“why” questions
Limitations
Takes a moderately amount
of time
Results indicate what people
*say* they do (rather than
actual behavior)

Interview Tips
Start broad, then narrow-in.
example: “Overall, how do you think Pittsburgh public
transit compares to other cities?”
Ask clarifying questions, and use their words.
example: “You said the bus system is hard to predict,
could you explain that to me?”
It’s okay to play-dumb. (But be honest.)
example: “I’ve never used public transit here. Can you
tell me how you would find out the schedules and
figure it out how to get downtown from here?”

More Interview Tips
Avoid “Yes/No” questions.
Avoid asking about feelings. Ask about behaviors instead.
Don’t number your questions. Organize by topics you
want to cover. Be prepared to skip around.
Always be prepared to go off-script.
Ask the question, then pause. Don’t rush to fill silence.

Activity - Part 1
Partner with someone who is interested in a different topic than you. You
will interview them on their chosen topic.
It’s your job to explore what the needs are and uncover the main issues,
feelings, thoughts, and pain-points.
Round 1: (5 minutes) Develop your Script
Goal: Individually, develop a rough script that you will use to interview
your partner. Start broad to gather overall impressions, then narrow in
on specific topic areas. Remember, you are trying to understand overall
impression and the biggest pain-points.
Round 2: (20 minutes) Interview (10 minutes each)
Goal: Take turns interviewing each other. Be sure to keep notes.

How do you know when you are
done conducting interviews?
Saturation (n.) - when the same topics (or themes)
keep emerging in your interviews, and conducting
more interviews results in no new themes.
Rule of thumb - 12 interviews for saturation

Example of one thing you can do with
interviews: build robust personas

Validation & Evaluation
Build Learn
A/B Testing
Contextual Inquiry
Diary Studies
Ethnography
Field Studies
Focus Groups
Hallway Testing
Interviews
Lab Studies
Log Analysis
Remote Testing
Think-Aloud
Surveys
User Observation

Types of Usability Testing
Formative Summative
A/B Testing ✔
Field Studies ✔
Hallway Testing ✔ ✔
Heuristic Evaluation ✔ ✔
Interviews ✔ ✔
Log Analysis ✔
Remote Testing ✔
Surveys ✔ ✔
Think-Aloud ✔ ✔
Wizard of Oz ✔

Lab Usability Studies
Formative Summative
A/B Testing ✔
Field Studies ✔
Hypothesis Testing ✔
Interviews ✔ ✔
Log Analysis ✔
Remote Testing ✔
Surveys ✔ ✔
Think-Aloud ✔ ✔
Wizard of Oz ✔

Lab Usability Testing
http://trydevkit.com/blog-post/a-beginner-s-guide-to-usability-testing/81da0af5-fb17-fd8e-016b-536948e32ced

http://trydevkit.com/blog-post/a-beginner-s-guide-to-usability-testing/81da0af5-fb17-fd8e-016b-536948e32ced

http://usabilitygeek.com/an-introduction-to-website-usability-testing/

are good for….
learning how easy or
difficult it is for users to
learn and use your interface
if language and
iconography are intuitive
how users encounter and
recover from errors
Usability Studies

Lab Usability Studies
Beneﬁts
Cheap (in $)
Observe user behavior as
they encounter a design for
the first time
See the consequences of
design decisions first-hand
Limitations
Usually takes a moderate
amount of time and set-up
Can sometimes feel staged,
or unauthentic

Running a Usability Study
Planning: create test plan, recruit participants
Pilot: practice with internal users, resolve any technical or
logistical issues
Test session: run test plan, be present (formative) or simply
observe (summative)
Debrief: short Q&A with participants, discuss observations
with other study observers
Analysis

Contextual Usability Studies
Formative Summative
A/B Testing ✔
Field Studies ✔
Interviews ✔ ✔
Log Analysis ✔
Remote Testing ✔
Surveys ✔ ✔
Think-Aloud ✔ ✔
Wizard of Oz ✔

Field Studies
http://www.oracle.com/webfolder/ux/applications/getInvolved/customerFeedback.html

are good for….
learning how customers
actually use your product in
day-to-day life
Field Studies

Field Studies
Beneﬁts
Allows you to observe
authentic, contextual, user
behavior
Can observe the day-to-day
experience users have with
your product, across a
longer period of time
Limitations
Significant cost ($)
Takes more time to run

Field Study Example
One laptop +
projector
Computer Lab
Laptop Carts

Field Study Example
[00:24:19.08] Boy says to the girl
on his right: "you cheating”
[00:24:21.19] Girl to the left:
"what? Its fun. ::mumble:: the
simulation. Look.”
[00:24:25.21] The boy looks to
the girl on his left, then back to
the girl on the right, then down to
his workbook in front of him. He
puts his head on the table.

A/B Testing
http://www.smashingmagazine.com/2010/06/the-ultimate-guide-to-a-b-testing/

is good for….
asking “how much”, “how
many”, “which one is better”
sampling from actual users
testing live apps/services
A/B Testing

A/B Testing
Beneﬁts
Low maintenance: release and
wait for data
Can measure very specific
questions
Live testing, measures actual
vs. self-reported user behavior
Limitations
Missing context of why users
take an action
May have no results, not
guaranteed to be conclusive
Only measures certain user
interactions

Running an A/B Test
Make sure to:
test conditions simultaneously (fewer confounding factors)
be consistent, keep track of which users see which version
deploy tests cautiously; user research can help inform
Things to watch out for:
don’t jump to conclusions, wait for statistical significance
waiting too long could cost you potential conversions
you might be interfering with user habit

https://medium.com/@adlon/threats-of-a-b-tests-and-ux-research-adoption-time-and-incrementalism-991c0c3c61b6

A/B Testing Tools
Optimizely
Visual Website Optimizer
Unbounce*

is good for….
seeing page views, entry/
exit, platforms, engagement
sampling from actual users
testing live apps/services
Log Analysis

Log Analysis
Beneﬁts
Low maintenance: release and
wait for data
Flexibility, can measure a wide
range of data
Live testing, measures actual
vs. self-reported user behavior
Limitations
Missing context of why users
take an action
Often requires initial
development overhead

Example: Google Analytics
Question: how many mobile users does my app have?
http://www.smallbox.com/blog/quick-and-dirty-ux-3-things-google-analytics-can-tell-you-about-your-users

Question: what paths do users take on my site/app?

Question: how long are users spending on my site?

Custom logging: track any event you want (links, performance, etc.)
http://www.sitepoint.com/5-ways-use-google-analytics-ux-research/

Usability Studies
Formative Summative
A/B Testing ✔
Field Studies ✔
Interviews ✔ ✔
Log Analysis ✔
Remote Testing ✔
Surveys ✔ ✔
Think-Aloud ✔ ✔
Wizard of Oz ✔

How to Choose a Usability Study?
Triangulation (n.) - using two or more methods to
discover and validate a finding.

Blind Men and the Elephant Parable

Consider Tradeoffs and Select
Methods that Meet Your Needs

Usability Studies
Formative Summative Quantitative Qualitative
A/B Testing ✔ ✔
Field Studies ✔ ✔
Hallway Testing ✔ ✔ ✔
Heuristic Evaluation ✔ ✔ ✔
Hypothesis Testing ✔ ✔
Interviews ✔ ✔ ✔
Log Analysis ✔ ✔
Remote Testing ✔ ✔ ✔
Surveys ✔ ✔ ✔
Think-Aloud ✔ ✔ ✔
Wizard of Oz ✔ ✔
what are you testing? what kind of results do you want?

Potential Pitfalls of Quantitative Research
Easy to make mistakes:
phantom correlations
finding may not generalize (participant selection)
requires sound experimental design
But it’s a great supplement to qualitative research

Activity - Part 2
Reflect on the one interview you conducted.
How has your knowledge grown? What do you still
need to learn about?
What do you need to get there? (What do you need
to do complete a full persona?)
How would you build a research plan for your topic?

Validation & Evaluation
A/B Testing
Contextual Inquiry
Diary Studies
Ethnography
Field Studies
Focus Groups
Hallway Testing
Interviews
Lab Studies
Log Analysis
Remote Testing
Think-Aloud
Surveys
User Observation
Learn

http://anotheruxguy.com/2015/07/06/analysis-is-cool/
Understanding Your Data

So Why Doesn’t My UI Work?
Seven Stages of Action
Mental Models
Psychological Biases
Dark UX Patterns

Seven Stages of Action
USER
SYSTEM
Establish
goal
Form
intention
Specify
action sequence
Execute action
Perceive
system
state
Interpret
system
state
Evaluate system
state
EXECUTION
EVALUATION
Gulf of Execution
How do I do it?
Gulf of Evaluation
What does it mean?

The Gulfs
Gulf of Execution
Does your app have good mappings? Can they easily
figure out how to execute on their desired goal?
Gulf of Evaluation
Does your app provide good feedback and visual
cues? Can users easily interpret what the data the
app is conveying to them?

Experimental Biases
Selection Bias
Confirmation Bias
Diagnosis Bias
Regression Towards the Mean

conﬁrmation bias (n) - the tendency to search for or
interpret information in a way that confirms one’s
preconceptions
possible pitfalls:
you focus your questioning on behaviors that you
expected to see, that confirm or validate your design
you discount negative comments about your design

diagnosis bias (n) - the tendency to label things
based on initial impressions, and the difficulty or
inability to change minds after the initial impression
possible pitfalls:
discounting a participant’s responses based on their
initial responses to selected questions

regression towards the mean (n) - when a non-
random sample is selected, the average of that
sample tends to regress towards the mean
possible pitfalls:
thought your intervention was the reason for an
improvement, but it was simply due to sampling

Cognitive Biases
Anchoring
Framing
Change Blindness
Illusion of Control
Loss Aversion

Anchoring
Anchor (n) - something that serves as a reference point

Anchoring
Frame (n) - the way we present a decision may
highlight different attributes
A pound of meat that is 90% lean
or
A pound of meat that is 10% fat

Framing
Frame (n) - the way we present a decision may
highlight different attributes
This treatment has a 90% chance of saving your life
or
This treatment has a 10% chance of failure, resulting in death

change blindness (n) - the tendency to overlook
alterations, especially when they appear immediately
after a visual interruption

Loss Aversion
the tendency that loss is more acutely felt than gain

Dark UX Patterns
Privacy: Should it be opt-in or opt out?

UX is a holistic approach,
driven by process & iterations

Case-Study: Anemia in Cambodia
Iron deficiency is a global problem
In the US: affects 3.5 million Americans each year
In Cambodia: affects 68% of children, 50% of adults

Image borrowed from: http://www.bustle.com/articles/84173-the-lucky-iron-fish-helps-fix-iron-
deficiencies-just-by-boiling-it-with-food-and-it

Refine Build Learn
A/B Testing
Contextual Inquiry
Diary Studies
Ethnography
Field Studies
Focus Groups
Hallway Testing
Interviews
Lab Studies
Log Analysis
Remote Testing
Think-Aloud
Surveys
User Observation

Feedback & QA
Questions? Comments?
Are there topics you wished we spent more time on?
How do you see some of these topics applying to
your current work?

Thank you!
Meg Kurdziolek
meg.kurdziolek@gmail.com
www.megkurdziolek.com
Karen Tang
karen@kptang.com
www.kptang.com

Resources
Surveys:
http://uxmastery.com/better-user-research-through-surveys/
Interviews:
http://theuxreview.co.uk/user-interviews-the-beginners-guide/
http://www.nngroup.com/articles/interviewing-users/
https://whitneyhess.com/blog/2010/07/07/my-best-advice-for-conducting-user-
interviews/
Usability Studies:
http://www.usability.gov/how-to-and-tools/methods/usability-testing.html

Resources
Lucky iron fish TED Talk
https://www.youtube.com/watch?v=0Lf6glgKt3Q

Great UX Research Books
Just Enough Research by Erika Hall
Usability Testing Essentials by Carol M. Barnum
Observing the User Experience by Elizabeth
Goodman

UI/UX Foundations - Research

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie UI/UX Foundations - Research

Ähnlich wie UI/UX Foundations - Research (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

UI/UX Foundations - Research