Are you already running experiments at scale and think there is nothing left to learn? Well, first of all congratulations. And second, think again. Because when it comes to the number of experiments you run, more is always more. Join this webinar to find out why there are no limits to experimentation – and how you can keep on making tangible progress. This webinar will show you: - How to embed ownership of experimentation into each and every team - How to keep the experimentation engine running with fresh and creative testing ideas - How to embed a winning experimentation culture across the entire business

1. 1
Accelerating
Customer Speed in
Europe: Where Next?
Hazjier Pourkhalkhali
Kenneth Kutyn

2. 2
Hazjier Pourkhalkhali
Global Director of Strategy & Value
Housekeeping: Ask questions in the Q&A widget
We will share the recording and slides
Kenneth Kutyn
Lead Solutions Engineer

3. 3
Customers move fast, you need to move faster
Customers today have sky-high expectations.
Digital transformation is ongoing and organizations need to repeatedly optimize
their customer digital experience using experimentation.
Established brands are under threat from digital dominators:
Spotify, Netflix, Amazon, Booking.com, Revolut.

4. 4
Getting experimentation started
More experiments?
Time to take action and keep progressing
your experimentation strategy.

5. 5
Your new ‘business as usual’:
The two steps to get started
Revitalize your hypotheses and the digital
experimentation ideas engine.
Embed digital experimentation across
the entire business.

6. 6
Your new ‘business as usual’:
The two steps to get started
Revitalize your hypotheses and the digital
experimentation ideas engine.
Embed digital experimentation across
the entire business.

7. 7
Revitalize your hypotheses and the digital
experimentation ideas engine

8. 8
Step 1
Use the experimentation cycle.
Revitalize your hypotheses and the digital
experimentation ideas engine

9. 9
Step 1: Gather data about customer behaviour and use insights
to ask questions.
Step 2: Formulate a hypothesis based on these insights.
Step 3: Design and implement an experiment based on your
hypothesis. Ideate solutions and decide which variations
of the user experience you will experiment with.
Step 4: Analyze your results to determine if your hypothesis is
confirmed or rejected.
Step 5: Formulate and document conclusions.
Step 6: Use conclusions to create new hypothesis. Repeat.
Step 1
Revitalize your hypotheses and the digital
experimentation ideas engine

10. 10
Step 1
Use the experimentation cycle.
Maintain pipeline with innovative, disruptive, creative ideas.
Revitalize your hypotheses and the digital
experimentation ideas engine

11. 11
Step 1
You’ve mastered the first two stages of digital experimentation.
But, there is still more to learn.
Reinvent the wheel.
Revitalize your hypotheses and the digital
experimentation ideas engine

12. 12
First there was Uber.
Step 1
Revitalize your hypotheses and the digital
experimentation ideas engine

13. 13
First there was Uber.
Then there was Uber Eats.
Step 1
Revitalize your hypotheses and the digital
experimentation ideas engine

14. 14
Step 1
Revitalize your hypotheses and the digital
experimentation ideas engine

15. 15
Step 1
Revitalize your hypotheses and the digital
experimentation ideas engine

16. 16
Constant disruption is happening all around you.
Step 1
Become a disruptor yourself.
Everything starts with creative, risky ideas.
Revitalize your hypotheses and the digital
experimentation ideas engine

17. 17
Everything begins with the information you already have.
Where does hypothesis creation and ideation begin?
So then how do you build the analytics engine to fuel the
‘right’ kind of experimentation?
Let’s look at the tools you can and should draw from, quantitative
and qualitative.
Revitalize your hypotheses and the digital
experimentation ideas engine

18. 18
Most companies rely on “Reporting Analytics”
Quantitative
Revitalize your hypotheses and the digital
experimentation ideas engine
Advanced insights rely on exporting large data-sets into platforms like
RedShift / BigQuery and relying on data scientists to manually analyze and
extract insights

19. 19
Quantitative
Modern analytics platforms allow faster access to prioritized
insights: Amplitude, Contentsquare, FullStory, Quantum Metrics
Revitalize your hypotheses and the digital
experimentation ideas engine

20. 20
Quantitative
Heatmapping solutions allow for better paint point understanding:
Amplitude, Contentsquare, FullStory, Quantum Metrics
Revitalize your hypotheses and the digital
experimentation ideas engine

21. 21
Quantitative
Customer Data Platforms reduce integration hurdles, increase data
consistency, and reduce browser calls:
Revitalize your hypotheses and the digital
experimentation ideas engine

22. 22
Don’t forget about qualitative methods.
Qualitative
Use your pre-existing relationships with customers. Talk
to and interview them.
Set up a customer advisory board for detailed feedback.
Revitalize your hypotheses and the digital
experimentation ideas engine

23. 23
Is this risky?
Taking risks
Isn’t risk-taking behind me?
Would you rather disrupt yourself or have a
competitor disrupt you?
Revitalize your hypotheses and the digital
experimentation ideas engine

24. 24
To summarize:
• Take your analytics to the next level.
• Use richer, more granular data to reinvent the
creative process.
• Fill your pipeline with creativity, innovation…
• ...and some calculated risks!
Taking risks
Revitalize your hypotheses and the digital
experimentation ideas engine

25. 27
Embed digital experimentation
across the entire business

26. 28
Scope.
Fully embedding digital experimentation
Function.
Embed digital experimentation
across the entire business

27. 29
Scope.
Fully embedding digital experimentation
Function.
Embed digital experimentation
across the entire business

28. 30
Making the business case for digital experimentation
Build a solid business case with hard numbers and evidence.
Create urgency and desire.
Demonstrate your strategy for digital experimentation.
Set clear milestones on a straightforward timeline.
Create a complex, data-driven and decision-making culture.
Embed digital experimentation
across the entire business

29. 31
Pop quiz
What do HelloFresh and Amazon have in common?
Embed digital experimentation
across the entire business

30. 32
What does Jeff Bezos do?
There’s no mystery or secret sauce.
Jeff spelt it out in 1997.
Embed digital experimentation
across the entire business

31. 33
What does Jeff Bezos do?
Jeff spelt it out again in 2012.
Embed digital experimentation
across the entire business

32. 34
What does Jeff Bezos do?
Every year!
There’s no shortage
of inspiration.
Embed digital experimentation
across the entire business

33. 36
Scope.
Fully embedding digital experimentation
Function.
Embed digital experimentation
across the entire business

34. 37
Making the business case for digital experimentation
Digital experimentation is needed in every person in every team in
every business function.
It is too easy for digital experimentation specialists to appear in
business teams.
How do you take digital experimentation from a niche activity to
standard procedure?
Embed digital experimentation
across the entire business

35. 38
The philosophy of product development
Alter the process to include experimentation:
• Workflow of production development.
• Strategy around product roadmaps.
Build the coaching and knowledge sharing abilities inside
each team.
Embed digital experimentation
across the entire business

36. 39
Changing roles in a digital experimentation landscape
Jobs need to change and grow.
Old job titles might need upgrading.
Sharing successes and education on experimentation will
prepare people for the change.
Embed digital experimentation
across the entire business

37. 40
Changing roles in a digital experimentation landscape
Planning for the implementation of experimentation
needs to be dynamic.
Maturity can vary dramatically between businesses.
IBM ran only 97 experiments in 2015
IBM increased experimentation x30 in 3 years.
Embed digital experimentation
across the entire business

38. 41
Growing pains
When you grow in digital experimentation,
new problems can emerge.
Online fashion retailer Farfetch has experienced this exact issue.
3,200 employees all learning experimentation.
Embed digital experimentation
across the entire business

39. 42
The Farfetch example
“Experimentation has always been a part of our DNA, but different
parts of the business were using different tools and adopting
different approaches. As we grew, this started to present
challenges – such as duplication of effort. What we needed was a
single framework that would enable us to consistently apply
experimentation across the entire customer journey.”- Luís Trindade, Principal Product Manager, Farfetch
Embed digital experimentation
across the entire business

40. 43
The Farfetch example
“Our test and learn center of excellence includes a regular
experimentation clinic. This is where we identify new areas for
experimentation, hold regular sharing and training sessions, and evolve
our on-boarding experimentation kit for new joiners. The goal is to make
sure we encourage people to experiment - not just because they need
to, but because they really want to.”- Luís Trindade, Principal Product Manager, Farfetch
Embed digital experimentation
across the entire business

41. 44
One more thing...
Remember experiment champions?
Farfetch uses experiment ambassadors to share best practices
across the business.
Embed digital experimentation
across the entire business

42. 45
Your new ‘business as usual’:
The two steps to get started
Revitalize your hypotheses and the digital
experimentation ideas engine.
Embed digital experimentation across
the entire business.

43. 46
Download the European
research report.
Visit optimizely.com for more
case studies and inspiration.

44. 47
You’ve got questions,
we’ve got answers

45. 48
Thank you!

46. 49