Collaborative
Research
with Erika Hall

Hello!

I have a
question...

Do you enjoy
being right?

You are correct!

Yessss!

?

Flickr/Chris Voll

Ego!

Assuming
!=
Determining

Research!

Agenda

9–10:30
Introduction. Collaboration and
Research. Forming Your Questions.
10:30–11
Morning Tea
Agenda
11–12:30
12:30–1:30
Activities. Interviewing.
Lunch
1:30–3
Analysis & Models
3–3:30
Afternoon Tea
3:30–5
Analysis & Models Contd. Reporting
& Sharing.Wrap-up

Flickr/Jerome Collins

Design-Led
Design-led
with
participatory
mindset
Design-led
with
expert mindset
Expert
Mindset
Users seen as
subjects
Participatory
Mindset
Users seen as
partners
Research-led
with expert
mindset
Dubberly Design Office
Research-led
with participatory
mindset
Research-Led

Goal Driven
Increase chance
of success
Reduce risk
Skeptical Mindset
Willing to question the
value of any approach

Collaboration

Dogma!

How?

Chris Noessel

Why?

Goals

LeanUX Principles:
Design thinking
Agile methods
Lean startup method

Extreme
Uncertainty

Design
Thinking

“Design thinking is a humancentered approach to
innovation that draws from the
designer's toolkit to integrate
the needs of people, the
possibilities of technology, and
the requirements for business
success.”
– Tim Brown, president and CEO IDEO

Dogma

Context

Real World
Your
Organization
Your
Users
Context

UX

Business
Business
UX
UX
Business
UX
Business
UX
UX
UX

Locate your risk

Flickr/Kris Krug

Where are you
coming from?

Commit to working
collaboratively

Establish a process.

Overcome
objections

We don’t have
the time.

We don’t have
the money.

We don’t have
the expertise.

We’re already
A/B testing

Everyone wants
better products.

No one wants to
read a report.

Design is active.
Reading is passive.
Research is active.

Embrace conflict

One Simple
Process

Form
Questions
Gather
Data
Analyze
Data

Form
Questions
Think
Critically
Analyze
Data

Form
Questions
Observe
Analyze
Data

Form
Questions
Interview
Analyze
Data

Form
Questions
Read
Analyze
Data

Think
Critically
Observe
Form
Questions
Interview
Experiment
Read
Analyze
Data

Interpretation
Interpretation
Interpretation
Interpretation

Shared
Reality

Collaborating
with
Strangers

A design project
is a series of
decisions.

Research
is a craft.

Bias

Confirmation Bias:
You selectively weight
the information that
confirms what you
already believe.

Sampling Bias:
Your sample of
research subjects isn’t
sufficiently
representative.

Interviewer Bias:
You insert your
opinion into
interviews.

Social Desirability Bias
People don’t say the
true things that they
worry will make them
look bad.

Dunning-Kruger
Effect
Unskilled people feel
overly confident.
Competent people are
less so.

Forming
Questions

Good Questions
Specific
Actionable
Practical

A Bad Question
“What do
people think
about pets?”

A Better Question
“How do
single urban
adults choose
and acquire a
pet?”

A Bad Question
“What do
people do
around here all
day?”

A Better Question
“How do we
coordinate
communication
priorities across
departments?”

Critical
Thinking

Critical Thinking
Disciplined
Self-correcting
Clear
Logical

Uncritical Thinking
“I hate yellow,
so all yellow
websites are
total failures.”

Critical Thinking
“I hate yellow,
but based on
the evidence, it
might work for
our audience.”

Research and Collaboration
Working together across disciplines and making decisions
based on evidence shouldn’t be hard, but they can be.
Done right, research and working collaboratively reinforce each
other through a shared understanding of reality.
Start with your goal in mind, not with any process or buzzword.
Asking questions and cutting across traditional roles can both
be threatening to the established order.
Commit to clear communication and critical thinking.
Research questions follow from goals, assumptions, and risk.
Always have a framework and a plan.

Break!

Activities!

Form
Questions
Gather
Data
Analyze
Data

Gather Data

Interviews
Interviews
Users
Contextual
Inquiry
Literature
Review
Usability
Testing
Descriptive
Descriptive
A/B
Testing
Evaluative
Generative
Questions
About
Org
Analytic
Product
Evaluative
Evaluative
SWOT
Analysis
Usability
Testing
Analytic
Competitive
Analysis
Brand
Audit
Competition
Heuristic
Analysis

Research
Activities
Topic
Purpose
Time
Money

Purpose (Decision Type):
What needs doing?
What are people doing?
How is this thing working?

Purpose (Decision Type):
Find new product idea.
Better meet an identified need.
Iterate on existing product.

Purpose (Decision Type):
Generative.
Descriptive
Evaluative

Why Not Just
Prototype?

If we only test bottle
openers, we may
never realize
customers prefer
screw-top bottles.
– Victor Lombardi, Why We Fail

Organizational
Research

Organizational
research is good for:
Requirements
Politics
Workflow
Capabilities
Goodwill

Requirements
What are the
top business
priorities for
this project/
product?

Politics
What does
success mean
to the
individual
stakeholders?

Workflow
Do we have to
change how
people are
working
together to be
successful?

Workflow
How do we
have to change
how people are
working
together to be
successful?

Workflow
How can we
possibly change
how people are
working
together?

Capabilities
What are the
strengths and
weaknesses of
our team?

Capabilities
Where is the
internal
expertise?

Goodwill
How can this
project make
your job easier
(or harder)?

Basic Stakeholder Questions
What is your title? How long have you been in this role?
What are your essential duties and responsibilities?
What does a typical day look like?
Who are the people you work most closely with? How is that
going?
What does success mean from your perspective, what will have
changed for the better once this project is complete?
Do you have any concerns about this project?
What do you think the greatest challenges to success are?
Internal and external?

For each stakeholder, note the following:
What’s their general attitude toward this project?
What’s the goal as they describe it?
To what extent are this person’s incentives aligned with the
project’s success?
How much and what type of influence do they have?
Who else do they communicate with on a regular basis?
To what extent does this stakeholder need to participate
throughout the project, and in which role?
Is what you heard in harmony or in conflict with what you’ve
heard from others throughout the organization?

Stakeholder power moves
“Why are you asking me this?”
“I don’t understand that question. It doesn’t make any sense.”
“I don’t feel comfortable talking to you about that.”
“No one pays attention to anything I have to say, so I don’t
know why I should bother talking to you.”
“How much more time is this going to take?”

10 minutes practice.
What is your title? How long have you been in this role?
What are your essential duties and responsibilities?
What does a typical day look like?
Who are the people you work most closely with? How is that
going?
What do you think the greatest challenges to success are?
Internal and external?

User
Research

Ethnography
Photo: Flickr/theloushe

The Four Ds of
Design Ethnography

Deep Dive
Daily Life
Data Analysis
Drama

“...true ethnography
reveals not just what
people say they do,
but what they actually
do.”
–PARC

Photo: Flickr/lintmachine

The Art of The
Interview

How to do bad user
research:
Ask what people want

Everybody Lies

Interviewing is not
talking.

Interviewing is
listening.

You
The
Comfort
Zone
Subject

You
The
Comfort
Zone
Subject

You
The
Comfort
Zone
Subject

You
The
Comfort
Zone
Subject

You
The
Comfort
Zone
Subject

Good Interviewers:
Know Your Question
Warm Up
Shut Up

Introduction
Body
Conclusion

Introduction:
Smile
Express gratitude
Describe the process
Ask to record
Warm up questions

Body:
Ask open-ended
questions
Probe for more
Allow silence
Use questions as
checklist

Conclusion:
Transition to wrap-up
Ask if there is
anything else
Thank for time

You are the host
You are the student

Interview Checklist
Create a welcoming atmosphere to make participants feel at
ease.
Always listen more than you speak.
Take responsibility to accurately convey the thoughts and
behaviors of the people you are studying.
Start each interview with a general description of the goal,
but be careful of focusing responses too narrowly.
Avoid leading questions and closed yes/no questions. Ask
follow-up questions.
Prepare an outline of your interview questions in advance, but
don’t be afraid to stray from it.
Also note the exact phrases and vocabulary that participants
use.

Look for
Goals
Priorities
Tasks
Motivators
Barriers
Habits
Relationships
Tools
Environment

Lunch!

Interview Scenario
You work for an e-Commerce site that wants to develop a new
service to help people give gifts. The goal of the research is to
identify unmet needs people might have with regard to giving
gifts.

Interview Practice
Break into groups of 3-4 people
1 interviewee, interviewer , 1 notetaker, 1 observer (optional),
Switch in 15 minutes
3 rounds

Listen for:
Goals
Priorities
Tasks
Motivators
Barriers
Habits
Relationships
Tools
Environment

How did that go?

How about a
focus group?

“Even when the subjects are
well selected, focus groups
are supposed to be merely
the source of ideas that need
to be researched.”
–Robert K. Merton, Sociologist,
the guy who invented focus groups

Competitive
Research

How else might
your target
customer solve the
same problem?

Competitive Review
How do they explicitly position themselves? What do they say
they offer?
Who do they appear to be targeting? How does this overlap or
differ from your target audience or users?
What are the key differentiators? The factors that make them
uniquely valuable to their target market, if any?
How do the user needs or wants they’re serving overlap or
differ from those that you’re serving or desire to serve?
What do you notice that they’re doing particularly well or
badly?
Based on this assessment, where do you see emerging or
established conventions in how they do things, opportunities to
offer something clearly superior, or good practices you’ll need
to adopt or take into consideration to compete with them?

Your target customers
have to love you
more than they
hate change.

(Usability)
Testing

A good research activity:
• Answers a key question
• Addresses identified assumptions
• Informs specific decisions
• Involves your team
• Fits your level of expertise
• Fits your schedule and budget

Collaborative
Recruiting!

How to find people:
• From your existing, high-traffic site
• Social networks
• Friends and family
• Mailing lists
• Flyers

A good research activity:
• Answers a key question
• Addresses identified assumptions
• Informs specific decisions
• Involves your team
• Fits your level of expertise
• Fits your schedule and budget

• Fundamentally research is a simple process
• There are many activities and definitions
• No pressure!
• Select the methods that inform decisions
• Begin by understanding your organization
• Never ask what people like
• People are lazy, forgetful creatures of habit
• Keep each other honest
• Practice and learn

Analysis
and Models

Creating Meaning
From Data

1. Compile data
2. Analyze
3. Identify Insights
4. Create Model

Analysis

Basic Analysis
Closely review the notes.
Look for interesting behaviors, emotions, actions, and verbatim
quotes.
Write what you observed on a sticky note (coded to the
source, the actual user, so you can trace it back).
Group the notes.
Watch the patterns emerge.
Rearrange the notes as you continue to assess the patterns.

Observation
Observation
Observation
Observation
Observation
Observation

Observation
Observation
Observation
Observation
Observation
Observation
Observation

Collaborates
on purchases
Observation
Observation
Observation
Observation
Observation
Observation
Observation

Collaborates
on purchases
Observation
Observation
Uses several
devices
Observation
Observation
Observation
Observation
Observation

Collaborates
on purchases
Observation
Observation
Uses several
devices
Observation
Observation
Observation
Needs
affirmation
Observation
Observation

Ground rules
Acknowledge that the goal of this exercise is to better
understand the context and needs of the user. Focus solely on
that goal.
Respect the structure of the session. Refrain from identifying
larger patterns before you’ve gone through the data.
Clearly differentiate observations from interpretations (what
happened versus what it means).
No specific solutions until after you’ve gone through insights
and principles. Solutions come next.

Look for
Goals
Priorities
Tasks
Motivators
Barriers
Habits
Relationships
Tools
Environment

25 minutes analysis.
Break into groups of 6-8 people
Each group work together to fill out one diagram with the
strongest patterns.
Negotiate and advocate for your perspective.

Models &
Pictures

Extract information

Extract information

Get thoughts
out of your
head

Personas

I’ve never seen a
persona called
“Married woman,
no kids, with
pristine
hardwood.”
God, how I aspire
to see that
persona.
-Steve Portigal

Make a persona based on your interviews
Back into the analysis groups
One person will describe the personas to
everyone and we’ll decide whether they
can be collapsed.

Other
Models

ThoughtWorks

A concept map is a
picture of our
understanding of
something.
–Dubberly Design Office

Generate lists of words related to the main
concept.
The list can come from research, reading, experts,
brainstorming, or any other source.
The second step is to edit the list. Some terms may be related
to the subject, but not in a way that meets the project goals.
The third step is to define the terms on the edited list. This is
particularly important with unfamiliar or technical terms. But it
also helps with familiar terms, too.
Create a matrix listing all the terms down one side and
repeating the list across the top. Note the relationship in the
boxes where a row and column intersect. The resulting matrix of
relationships provides a checklist for building the concept map.

• List terms
• Edit the list
• Define the remaining terms
• Create a matrix showing the relations of
terms
• Rank the terms
• Decide on main branches or write framing
sentences
• Fill in the rest of the structure
• Revise
• Apply typography to reinforce structure
• Revise

Analysis and Models
Everyone on the team should be involved in turning data into
insights. A productive session requires rules.
Once you and your team have extracted insights from data,
document those insights in models.
A model distills and documents thinking so everyone on the
team can see it and make decsions based upon it.
Remember than models are still an interim document. They are
tools. Think “useful” not precious. Update as needed.
The affinity diagram comes straight out of analysis sessions.
Personas are one of the most intelligible research outputs for
people throughout the organization.

Break!

Reporting
and Sharing

How to make
research meaningful
to your organization

Flickr/Kris Krug

Flickr/Jerome Collins

You are
collaborating with
your future selves.

Design synthesis is the most
critical part of the design
process. Yet in our popular
discussions of design and
innovation, we've largely
ignored this fundamental role.
–John Kolko

Building a Culture
of Research

How to make
research meaningful
to your organization

It is your job to make
it easy for everyone
else.

Study Title
Date Completed
Research Goal
Related Decisions
Activities
Key Insights
Supporting Observations
Recommended Actions
Questions for Further Study
Research Report

Clear goals
Shared values
Access to information
Clear decision-making

The goal determines
the form

How to apply research

Reasons to Share

Flickr/loozrboy

The report is not
the research.

Why report at all?

Informing?
Inspiring?
Focusing?
Remembering?
Recording?
Deciding?

Wrap Up

In summary
Research creates a shared understanding of reality.
Asking questions is uncomfortable. Embrace that feeling.
A truly collaborative approach and environment is necessary for
research to be effective, and it also makes it more fun.
Clear goals and good questions are required.
Choose only the research activities that answer real questions
and inform your top priority design and development decisions.
Practice! Observe and listen every day.
Document! Report! Share! It’s easy to lose what you learn.

Any questions?

Brief books for people who make websites
Erika Hall
JUST ENOUGH
RESEARCH
9
No.
You might
enjoy the
book.
www.abookapart.com