A detrimental mistake many startups do is neglecting product experimentation. They focus mainly on experimentation in marketing - optimising ads, copy, creatives, landing pages - because it's relatively easy to do. But this kind of negligence is extremely damaging for your success! But why don't more startups do this then? EARLY-STAGE - Specific challenges: -No budget: “proper” testing tools quite expensive -Little data to play with -Not sure what to experiment on / what moves the metrics massively? -No time; need to focus on the business GROWTH STAGE - Specific challenges: -More budget now, but limited by free tools -More data about what's happening but not really sure why it's happening -So many ideas and opportunities, how to prioritise? SCALING STAGE - Specific challenges: -All tools in place, but now constrained by tech time: experiments need to be “bigger” to have impact + are competing with other projects -Harder to experiment with the core product, the company is branching out on other products -Experimentation becomes scarier: more users, more to lose, users get used to things being a certain way (“Why did they change the UI again?”) Milena Court, Product Manager in the Growth Squad at Tails.com, joined us at The Family to share her incredible expertise and explain how to overcome the above challenges!

1. How to scale your product
through experimentation
Milena Court
Product Manager (Growth Squad)
@milenacourt

2. When did you last experiment to learn about your users?
Marketing
Experimentation
Product
Experimentation
Facebook Ads Split Testing, Google Ads
testing, Landing page test etc.
��

3. My journey experimenting across the startup stages
Scale-Up
Ridesharing platform
Growth Stage
Tailored pet food
Early Stage
Office food delivery

4. Tails.com
We believe every dog deserves to live
it’s happiest, healthiest life.
That’s why we create a unique recipe
for every dog, in the taste they love
with the nutrition they need.

5. Tails.com

6. What is Product Experimentation?
📈 Measuring the impact of products we’re building
on the metrics we care about
-> Comparing the existing version of your website (or app) with
another version with changes (also known as A/B tests)
-> A mindset, evaluating ideas and making decisions based on data
and customer behaviour

7. Why is it important?
Define feature
Build
Forget about it
Shorter
learning
cycles

8. Top 3 excuses not to experiment with your product
I don’t have the right tools to experiment
I don’t know which experiments to run
I am time-constrained and experimenting is risky
This will happen at every stage of your startup,
not only to early-stage startups!
🤔
👩‍💻
😱

9. Experimenting with your product at every stage
This will happen at every stage of your startup,
not only to early-stage startups!
Early-stage
Growth-stage
Scale-up
● Challenges at each step
● How to overcome these
challenges
● Which types of
experiments can you run
at each stage?

10. 1. Early-stage startups

11. Challenges for early-stage startups
This will happen at every stage of your startup,
not only to early-stage startups!
No budget Not much data to
play with
Winning experi-
mentation pressure
Testing tools can be
expensive and you
have a very limited
budget
You have Google
Analytics in place but
not much traffic yet
The clock is ticking
before your next
fundraise, what if
experiments don’t work?
😫
💪
Start using a free tool
Google Optimise,
Sixpack etc.
Find experiments in
customer feedback
-> Go through your flow
with 10 customers
-> Email people who did
not convert and ask why
Make big bets experiments,
leave the small experiments
for later (e.g. CTA colors)
-> Test your riskiest
assumptions to move the
metrics

12. Types of experiments you can run
This will happen at every stage of your startup,
not only to early-stage startups!
Focus mainly on experiments around acquisition (get users to your
first main action!)
1) Test different way to describe your product and your USP
-> How do we explain how our product work?
2) Test where your main action should be in your funnel
-> Should we capture email at the beginning (ex: landing page)
or at the end of the funnel (ex: after adding items to the basket)?
3) Test your pricing model
-> Should we have a minimum spend and offer free delivery or
have no minimum spend but make you pay for delivery?

13. 2. Growth-stage startups

14. Challenges for growth-stage startups
This will happen at every stage of your startup,
not only to early-stage startups!
Increasing testing
speed
Finding more
insights
Prioritisation
Now you have a bit more
budget but you’re being
limited by free tools
More data about what
is happening but not
really sure why it is
happening
So many ideas and
opportunities for
experiments
😫
💪
Set up a more powerful
testing tool like Optimizely
or Monetate, focus on
quality of data + quantity
of the experiments
Do user research on
high exit points in flows
for qualitative insights
and invest in Product
Analytics for
quantitative insights
Have a good product
portfolio strategy: balance
safe bets, small rewards
with some higher-risk,
higher-reward bets (use
RICE scoring)

15. Types of experiments you can run
This will happen at every stage of your startup,
not only to early-stage startups!
Focus mainly on experiments around acquisition but also some
experiments around other key metrics like retention
1) Test the order of information in journeys
-> When is the right moment to talk about subscription?
2) Test nudges to help users go through a flow
-> How can we improve security messaging on checkout pages?
3) Test different designs for a key action
-> How can we improve design to show your price more clearly?

16. 3. Scale-ups

17. Challenges for scale-ups
This will happen at every stage of your startup,
not only to early-stage startups!
Hard to get results vs.
tech time invested
Experimenting with
new product/market
Experimentation
becomes scarier
Hard to do changes in a
mature product that moves
the metrics massively
Not all your customers
are the same across
products / markets
More users, more to lose,
being scared of customer
response
😫
💪
Running small tests to
justify time to develop a
bigger test, we’d rather
get the bad news and go
back to the drawing board,
as early as possible
Accept different levels
of confidence for tests:
you build your testing
instinct overtime
Add capabilities to test
with only a subset of users
(e.g. 10%) and scan
customer messages for
helpful feedback on tests

18. Types of experiments you can run
This will happen at every stage of your startup,
not only to early-stage startups!
Experiments across the customer journey to move your main metrics
1) Test launching new product line
-> How can we test if launching buses is valuable?
2) Test tailored prices vs. same price for everybody
-> Should we have dynamic prices based on supply and demand?
3) Test personalised content for specific segments
-> How can we help users in remote areas to be more successful?

19. Invest in product experimentation tools at every stage
of your growth from free tools to more advanced tools
Use experiments to understand your customers better
(create a bank of insights overtime widely shared)
Don’t aim for 100% win with experiments, keep
iterating and balance your bets
✔
✔
✔
If you only remember 3 things :)

20. “The only strategy that is guaranteed to fail is not
taking risks.” — Mark Zuckerberg, Facebook