Every product decision is an experiment. If we can reframe our thinking from launching campaigns and building features to challenging assumptions and running experiments, then we can increase our productivity and get better results with less effort. Grace will share a structured process for designing effective experiments, and talk about some of the common pitfalls and challenges teams face when running experiments in their companies. Using examples from her work helping product teams within large companies, Grace will reveal a few ways to overcome these challenges.
7. Who is your customer?
Must be specific to be easily testable!
@uxceo
8. What is his/her problem?
Must be specific to the customer
@uxceo grace@QuickMVP.com
@uxceo
9. Problem
INCORRECT:
“People are not buying enough cameras” is not a
problem someone has
CORRECT:
“The outdoor adventurer hates the bulkiness”
“The amateur photographer doesn’t think his skills are
good enough to invest in a DSLR ”
Different customers, different problems
@uxceo @uxceo grace@QuickMVP.com
10. Problem Hypothesis
I believe [my customer] has a
problem [achieving this goal]
@uxceo @uxceo grace@QuickMVP.com
11. Solution
Do not define until the problem is
validated
Why? Because every problem
has multiple solutions
@uxceo @uxceo grace@QuickMVP.com
13. What is the solution?
What will solve your customer’s problem?
@uxceo
14. Solution Hypothesis
I believe [my solution] will
result in [my customer]
taking [specific action]
@uxceo grace@QuickMVP.com
@uxceo
15. Step 2
Plan your experiment
@uxceo @uxceo grace@QuickMVP.com
16. What must we LEARN?
What are the Core Assumptions of
our business?
Assumption:
A behavior, mentality, or action that needs to
happen in order for hypothesis to be true.
@uxceo @uxceo grace@QuickMVP.com
17. What must we TEST?
Prioritize: Which Assumption is
the Riskiest?
The one that is most unknown,
and crucial to the business
@uxceo @uxceo grace@QuickMVP.com
18. Minimum Viable Product
Use the least amount of effort to get
the most amount of customer feedback
@uxceo @uxceo grace@QuickMVP.com
19. What What do must we BUILD we BUILD?
to test?
1. Exploration: test validity of problem
2. Pitch: test value of your solution
3. Concierge: test customer
satisfaction or behavior
@uxceo @uxceo grace@QuickMVP.com
20. How to MEASURE success?
Minimum Success Criteria :
What’s the weakest outcome you
need to see in order to move
forward?
@uxceo @uxceo grace@QuickMVP.com
21. How to MEASURE success?
1. Cost
• Project Cost
• Opportunity Cost
2. Existing metrics
• Business KPI
• Specific user behavior
@uxceo @uxceo grace@QuickMVP.com
22. Step 3
Run your experiment
@uxceo @uxceo grace@QuickMVP.com
30. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
32. Effective Interviews
1. I’m doing some research around...
2. Have you ever [experienced problem]?
3. Tell me about that time, what happened?
4. What did you do to resolve it?
5. How satisfied were you with the outcome?
6. If unsatisfied, how else have you tried to solve it?
@uxceo @uxceo grace@QuickMVP.com
33. Collect Data
• Level of pain/frustration
• Frequency of encounter with pain
• Existing behavior to solve problem
• Time/money spent searching for a
solution
@uxceo @uxceo grace@QuickMVP.com
34. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
9/12
Persevere
Friends.
Classes.
Paid
UX designers
hourly for help
@uxceo
35. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
@uxceo
36. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
@uxceo
39. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Pay for
virtual UX
feedback
Pitch:
8/10
10/10
Test
2nd side
of market
Don’t mind
virtual at all.
Price is cheap
9/12
Persevere
Friends.
Classes.
Paid
UX designers
for help
@uxceo
40. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
@uxceo
41. Boot-strapped
startup
founders
Getting
UX
feedback
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
No
reliable
source
Interview:
6/10
Interview:
6/10
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
@uxceo
43. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Pay for
virtual UX
feedback
Pitch:
8/10
10/10
Test
2nd side
of market
Don’t mind
virtual at all.
Price is cheap
Will
consult
virtually
Pitch:
8/10
0/10
Pivot!
9/12
Persevere
Friends.
Classes.
Paid
UX designers
for help
@uxceo
44. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Pay for
virtual UX
feedback
Pitch:
8/10
10/10
Test
2nd side
of market
Don’t mind
virtual at all.
Price is cheap
Will
consult
virtually
Pitch:
8/10
0/10
Pivot!
No!
5 yrs exp.
Established
9/12
Persevere
Friends.
Classes.
Paid
UX designers
for help
@uxceo
45. Boot-strapped
startup
founders
Getting
UX
feedback
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
No
reliable
source
Interview:
6/10
Interview:
6/10
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
Recent
grad UX
designers
experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
Recent
grad UX
designers
Getting
clients for
quick UX
consulting
@uxceo
46. Boot-strapped
startup
founders
Getting
UX
feedback
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
No
reliable
source
Interview:
6/10
Interview:
6/10
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
Recent
grad UX
designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Give quick
online
feedback
Pitch:
60%
experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
Recent
grad UX
designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Give quick
online
feedback
Pitch:
60%
@uxceo
52. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
8/10
Persevere
Friends.
Classes.
2 paid
UX designers
for help
Pay for
virtual UX
feedback
Pitch:
8/10
10/10
Test
2nd side
of market
Don’t mind
virtual at all.
Price is cheap
Will
consult
virtually
Pitch:
8/10
0/10
Pivot!
No!
5 yrs exp.
Established
Give quick
online
feedback
Pitch:
60%
37/184
emails
Persevere
Started
referring
friends
@uxceo
53. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
Recent
grad UX
designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Give quick
online
feedback
Pitch:
60%
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Online
platform is
sufficient
Concierge:
60%
satisfied
@uxceo
56. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
8/10
Persevere
Friends.
Classes.
2 paid
UX designers
for help
Pay for
virtual UX
feedback
Pitch:
8/10
10/10
Test
2nd side
of market
Don’t mind
virtual at all.
Price is cheap
Will
consult
virtually
Pitch:
8/10
0/10
Pivot!
No!
5 yrs exp.
Established
Give quick
online
feedback
Pitch:
60%
37/184
emails
Persevere
Started
referring
friends
Online
platform is
sufficient
Concierge:
60%
satisfied
20%
Iterate
Wanted
more
function-ality
@uxceo
59. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
GET OUT OF THE BUILDING!
@uxceo
60. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
GET OUT OF THE BUILDING!
@uxceo
61. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
Shoppers
will buy
GET OUT OF THE BUILDING!
@uxceo
CMS
Block
62. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Concierge:
Purchase
up 20%
GET OUT OF THE BUILDING!
@uxceo
Shoppers
will buy
64. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
Shoppers
will buy
from CMS
Concierge:
Purchase
up 20%
GET OUT OF THE BUILDING!
@uxceo
$0 in
Revenue
No CT
65. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
Shoppers
will buy
from CMS
Concierge:
Purchase
up 20%
GET OUT OF THE BUILDING!
@uxceo
$0 in
Revenue
No CT
Shoppers don’t
looks at the
home feed,
only
emails
66. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Shoppers
will buy
from CMS
GET OUT OF THE BUILDING!
@uxceo
Concierge:
Purchase
up 20%
67. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Shoppers
will buy
from CMS
GET OUT OF THE BUILDING!
Opensky
Shoppers
Not seeing
curator
messages
@uxceo
Concierge:
Purchase
up 20%
68. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Shoppers
will buy
from CMS
GET OUT OF THE BUILDING!
Opensky
Shoppers
Not seeing
curator
messages
@uxceo
Concierge:
Purchase
up 20%
Email
messages
69. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Shoppers
will buy
from CMS
GET OUT OF THE BUILDING!
Opensky
Shoppers
Not seeing
curator
messages
@uxceo
Concierge:
Purchase
up 20%
Email
messages
Shoppers
will buy
70. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Shoppers
will buy
from CMS
GET OUT OF THE BUILDING!
Opensky
Shoppers
Not seeing
curator
messages
@uxceo
Concierge:
Purchase
up 20%
Email
messages
Shoppers
will buy
Concierge:
Purchase
up 20%
73. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
Shoppers
will buy
from CMS
Concierge:
Purchase
up 20%
GET OUT OF THE BUILDING!
$0 in
Revenue
No CT
Shoppers don’t
looks at the
home feed,
only
emails
Shoppers
will buy
Concierge:
Purchase
up 20%
300%
on
purchases
Shoppers
like videos,
photos, from
curator in
email
@uxceo
77. 1. Form your hypothesis
2. Identify your riskiest assumption
3. Decide how you’ll test it
4. Define success
5. Analyze results and try again
@uxceo