(video summary at: https://conversionhotel.com/session/keynote-2019-using-experimentation-to-drive-product/) Conversion optimization and experimentation is focussing a lot on the digital marketing side of things. To turn more online visitors into buyers. But to get those users to spend and keep on spending money you will need a product that’s so good that marketing is hardly needed. We need better products. And to make better products we need to make better choices. Isn’t Conversion Optimization the business of adding effectiveness to a company? Hopefully to add effectiveness to efficiency? Effectiveness is the outcome of making better choices and boy, do we humans fail at making better choices… this is why we run experiments. But mostly focussing on marketing and sales. Shouldn’t we use experimentation as a product development framework? Stephen Pavlovich is here to take a deep dive into product experimentation. I’ve known Stephen quite a while now and have a deep respect on what he has build as agency founder in the UK. What we have in common, as many agency owners, is that we have a strong believe in what experimentation can bring to companies. Thank you for accepting this last minute keynote talk Stephen, replacing our common friend and also agency owner André Morys (who broke his knee…) Tell them about product experimentation! Good luck, Ton Wesseling Founder & host of The Conference formerly known as Conversion Hotel

1. Stephen Pavlovich | Experimentation Island | Conversion

2. “We are our choices.”

3. “It’s not our abilities that make us
who we are, Harry. It’s our choices.”

4. To make better products,
we need to make better choices.

5. But...
we suck at making good choices.

6. We value position, authority, experience
and opinion more than evidence.

7. Decisions by committee
will always be shitty.

8. The most successful companies
use experimentation as a
product development framework.

9. “Our success at Amazon is a function of
how many experiments we do per year,
per month, per week, per day.”

10. Four principles of
product experimentation.

11. 1
Experiment to solve your biggest problems.

14. “Cookie-gate”

15. Everyone says
they’re data driven.

17. 2
Be bold.

18. Most decisions are reversible.

19. “Amazon will be experimenting at the right scale
for a company of our size if we occasionally
have multibillion-dollar failures.”

20. If we aren’t prepared to fail,
we can’t innovate.

21. “What would you do if you were stuck in one place and
every day was exactly the same and
nothing that you did mattered?”

22. 3
Test early and often.

23. Build a new feature
Run an experiment to see
what impact it has
Wins Loses
Implement it
Implement it
anyway

24. Beware the
puppy dog close.

25. With dendritic branching, we constantly
test, validate and iterate to explore new ideas.

26. 4
Start small and scale.

28. https://blog.crisp.se/2016/01/25/henrikkniberg/making-sense-of-mvp

30. Source: https://medium.com/@AlbertoSavoia/the-pretotyping-effect-32712167cbd

31. Source: https://medium.com/@AlbertoSavoia/the-pretotyping-effect-32712167cbd

32. You can test more ideas
and increase your chances of success.

33. You can see what happens
before you make your choice.

35. STEP 3
Taste tests
Product reviewed and scores
assigned
STEP 4
Launch the product
First time actual customers
feedback on new product
STEP 1
Market research
Identify market trends,
run ideation sessions
STEP 2
Focus groups & research panels
Shortlist products

38. Your teams can experiment with autonomy
because it’s not your opinion they need.

39. Why is this better?

42. Stephen Pavlovich | Experimentation Island | Conversion