Innovation & Growth
With Experimentation
March 2023
By Haley Carpenter | Founder

User research &
experimentation
Consulting, execution, & training
mychirpy.com
haley@mychirpy.com
512-200-3510
https://www.linkedin.com/in/haley-carpenter-490/
Haley Carpenter
Founder
Past: CXL/Speero, Optimizely, Hanapin (now Brainlabs)

Every business is looking for
innovation and growth.
That’s obvious.

Innovate
“make changes in something established, especially by
introducing new methods, ideas, or products”
Grow
“become larger or greater over a period of time;
increase”
Google Definitions

How does your team decide on
what changes to make?

We make educated
guesses.
Common answer.

“We think…”
“We feel…”
“We believe…”

“We think…”
“We feel…”
“We believe…”

How does your team determine
increases?

We’re not measuring anything…
we don’t know what to choose.
Common answer.

Our data is broken.
Common answer.

Maybe you’re getting lucky.
That’s not strategic winning.

Experimentation can be a
primary driver of both
innovation and growth.
That’s maybe not so obvious.

Experimentation…
● Eliminates
guesswork
● Validates decisions
● Minimizes bias
● Mitigates risk

Experimentation will lead you
to strategic winning.

Experimentation you say?
What is that?

“Digital experimentation is similar, if not identical,
to the scientific method.”
“Businesses attempt to answer a question by
establishing a hypothesis, testing the hypothesis
through experimentation and analyzing the
results.”
“…experimentation is ‘the reliable process of
delivering winning digital experiences without
guesswork or risk…”
Contentful Link

Image Source
50% of traffic 50% of traffic

How do we start an
experiment?

Precursor:
Make sure you have
enough data volume.

Pre-test calculations…
one of the most important
things you can do.

WATCH THIS

Step 1:
Get a data point →
Create a hypothesis

What’s a Data Point?
Where Do I Get One?

What’s a Data Point?
Where Do I Get One?
RESEARCH

Source

Research Experiments
Source

Source: Speero
Link

Research will lead you to
strategic winning, too .

Good starting point: analytics
(e.g., Google Analytics, Amplitude, Heap, Adobe Analytics, Mixpanel, etc.)

Example data point:
The dropout rate from the
cart to the checkout is 92%.

Ideal: data points from 2+
methodologies

Example data points:
The dropout rate from the cart to the
checkout is 92%.
Users don’t know about the free return
policy or money-back guarantee
policy.

Step 2:
Get a data point →
Create a hypothesis

“We know X. If we do A,
then B will happen because of C.”
Hypothesis Format

“We know X. If we do A,
then B will happen because of C.”
Research & data Change something
Increase result

We know the dropout rate from the cart to the
checkout is 92% and that users don’t know about the
free return policy or the money-back guarantee policy.
If we do A, then B will happen because of C.
Hypothesis Format

We know the dropout rate from the cart to the
checkout is 92% and that users don’t know about the
free return policy or the money-back guarantee
policy. If we add content about the free return policy
and money-back guarantee policy on the cart page,
then B will happen because of C.
Hypothesis Format

We know the dropout rate from the cart to the
checkout is 92% and that users don’t know about the
free return policy or the money-back guarantee
policy. If we add content about the free return policy
and money-back guarantee policy on the cart page,
then the dropout rate from the cart to the checkout
will decrease and transactions will increase because
of C.
Hypothesis Format

We know the dropout rate from the cart to the
checkout is 92% and that users don’t know about the
free return policy or the money-back guarantee
policy. If we add content about the free return policy
and money-back guarantee policy on the cart page,
then the dropout rate from the cart to the checkout
will decrease and transactions will increase because
friction will decrease due to users feeling there is less
risk to place an order.
Hypothesis Format

We know the dropout rate from the cart to the
checkout is 92% and that users don’t know about the
free return policy or the money-back guarantee
policy. If we add content about the free return policy
and money-back guarantee policy on the cart page,
then the dropout rate from the cart to the checkout
will decrease and transactions will increase because
friction will decrease due to users feeling there is less
risk to place an order.
Hypothesis

“We know X. If we do A,
then B will happen because of C.”
Guessing with little to no
accurate data involved
Change something
Unclear metrics and
broken data are common
????????
?

Step 3:
Get a data point →
Create a hypothesis →
Create a variation design

Design
Image Source

Dev > QA
Launch > Monitor
Conclude > Analyze
Step 4+:

Result: +61%
Action: Implement the variation
Insight for what’s next: Should we
make the policies more visible
elsewhere, too?
RUN ANOTHER TEST!

Source

“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
Test 1
Test 2
Test 3
Test 4
Test 5
Test 6
Test 7
Test 8
Test 9
Test 10
Test 11
Test 12
Test 13
Test 14
Test 15
Test 16
Test 17
Test 18
Test 19
Test 20
Test 21
Test 22
Test 23
Test 24
Test 25
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight

Wins
+
Learnings

Exp
Leaders
Product
Marketing
Other teams

A few final topics

Prioritization
Article Link

Roadmapping

Common Misconceptions
● We’ll hack it together and be okay
● You just need a testing tool and an
idea
● You don’t need developers
● Every business can test
● It’s too hard
● It’s easy

Recap
● You need to experiment if you have enough data volume.
(Check by running pre-test calculations.)
● Starting an experiment
Step 1: Get a data point to inform your test idea.
Step 2: Write a hypothesis. (e.g., If…then…because)
Step 3+: Roadmap & prioritize your initiatives.
● The more teams involved in experimentation the better.
● If you’re not experimenting and doing user research,
your competitors are ahead of you.