Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup Guide

70 Aufrufe

Veröffentlicht am

How do you know where to start with experimentation? What if you don’t have enough information, or simply too much to decide where to begin and where to invest your time/effort/money?

In this breakout session we will cover how to cut the BS by treating experimentation as an “internal services startup”, where the customers are the teams in your business: commercial, trading, marketing, product, SEO etc.

You wouldn’t start a startup by hiring a bunch of people without a tool or an idea to work on, or buy an office or expensive work management solution for a startup of 3 people without developing a product and taking it to market first. So why treat experimentation that way?

Veröffentlicht in: Internet
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup Guide

  1. 1. Ellie Hughes DON’T PANIC The no-confusion experimentation startup guide Twitter: @EllieExperiment medium.com/@EllieExperiment
  2. 2. PROBLEM: You’re a business who want to start experimenting, but don’t know where to start: - Where should you invest your time, money or resources first? • In these cases, panic often sets in - this leads to a scatter gun approach to trying to get experimentation going: - Let’s buy a tool - Let’s pay an agency - Let’s hire a specialist - Let’s add more tools or get rid of an existing tool - Let’s get everyone in the business to come up with ideas PANIC!!
  3. 3. SOLUTION: This talk! Grab your towel and listen to a guide who has done the hitch-hiking so you don’t have to: • Advice on approach • A strategy to get started - Mindset - Enablers • Finally, how to assess how things are going DON’T PANIC!!
  4. 4. 1. Think of optimisation like a mini-business or services startup within your business: whose customers are the product and tech teams, commercial, marketing, the leadership team…… anyone who could be more successful if they can run experiments. 2. Develop your optimisation startup like a real business: - Hire a great CEO: get a CRO owner with vision and expertise - They develop & rollout a strategy: give them ownership of the programme and strategy - Invest: get the tools, people and skills they need to make the strategy successful - Run and assess: Let the programme run and see what is working - Double-down or pivot: keep doing what makes your customers happy, and fix the things that don’t work Advice
  5. 5. • Build a Mindset for experimentation: getting everyone to think about experiments constructively - The How, Why and What of Experimentation • Utilise a Model to enable experimentation: - Across the experiment lifecycle try different approaches and ways to enable experiments - Measure the results • Quick aside: The structure of your team doesn’t matter… What is my strategy for my experimentation startup?
  6. 6. Mindset Experiments consist of 3 key variables or decisions: ‣ What you are testing: Changes you think will impact your KPI ‣ Why you are testing: The impact you want to make on your KPI/metric or goal ‣ How you will test that effectively and rigorously: - Process/Planning - Ideation - Execution - Analytics - Value - … many more The How, Why and What of Experimentation
  7. 7. Mindset In many teams, the decision about what to test often comes first They are employing the following strategy: The How, Why and What of Experimentation: As-is
  8. 8. Mindset In many teams, the decision about what to test often comes first They are employing the following strategy: The How, Why and What of Experimentation: As-is • What phase: The team are doing ‘feature-led’ testing, where they decide what to test and then justify why they are testing it. • Why phase: the team look around for KPIs to support their pre-selected idea: And so increase conversion… • How: Finally, they work out how to test their feature, or whether it is even possible/feasible to run the experiment. This happens well after choosing the experiment.
  9. 9. Mindset How: Do we find out customer behaviour and problems from our metrics? Do we ideate experiments? Do we plan experiments? How do we do experimental design? Do we build experiments? Do we test statistically and analytically? Do we design the feature? Do we know an experiment has won? In actual fact, this hierarchy needs to be inverted: - Enable the team by teaching the principles of How to Test, which leads naturally into the Why and What to Test. The How, Why and What of Experimentation: To-be
  10. 10. Mindset • After the How, it becomes clear which KPIs can be influenced and whether these are actually important: The Why. • This leads to a growing number of much better ideas: The What. • The correct training and enablement free the team to be more autonomous and agile with their experiments. In actual fact, this hierarchy needs to be inverted: - Enable the team by teaching the principles of How to Test, which leads naturally into the Why and What to Test. The How, Why and What of Experimentation: To-be
  11. 11. Enablement This is designed to show how we can inject the How Why and What into our ways of working, and help us diagnose where this is going wrong. The enablement model Having got the right mindset, we then need to do the second half of our strategy: Utilise an enablement model across the experiment lifecycle
  12. 12. @ellieexperiment Business is Successful
  13. 13. Team Goal Impacted Team is successful and Healthy @ellieexperiment Business is Successful
  14. 14. Team Goal Impacted Team is successful and Healthy EXPERIMENT DESIGN SKILLS/ TRAINING ANALYSIS TECHNOLOGYEXECUTION IDEATION @ellieexperiment Business is Successful
  15. 15. @ellieexperiment
  16. 16. @ellieexperiment
  17. 17. Enablement • Lack of Impact on goal over time or over many experiments • Poor team health • Using experiment Retros How do you diagnose a failure of How, Why, What?
  18. 18. Enablement • An experiment retrospective looks back at what happened during the experiment, and what the team did and how well they did it. • Make it quantitative! What is an experiment Retro? • Ask a lot of questions: - How did the experiment go? - Was it a winner? - Did it impact their goal/KPI? If not did it impact there metrics instead? - Where did they feel it went wrong? - What knowledge and skill gaps were revealed? - What processes need to change for next time? - ….
  19. 19. Enablement • Using a simple spider scorecard, I work with our teams to reflect where they need additional help • For all the areas where they score 1 out of 5, we focus to bump that up to a 2 or 3. Diagnose a failure of How, Why, What
  20. 20. Enablement Diagnose a failure of How, Why, What • Low Maturity: The team biases towards their comfort zones: technical skills and execution/delivery. @ellieexperiment
  21. 21. Enablement Diagnose a failure of How, Why, What • Low Maturity: The team biases towards their comfort zones: technical skills and execution/delivery. • Medium Maturity: The team starts to focus more on analytics learnings and quality of ideation, leading to more impact on their goal. @ellieexperiment
  22. 22. Enablement Diagnose a failure of How, Why, What • Low Maturity: The team biases towards their comfort zones: technical skills and execution/delivery. • Medium Maturity: The team starts to focus more on analytics learnings and quality of ideation, leading to more impact on their goal. • High maturity: The team invests heavily in training (whole team), analysis & product insights, and experiments with planning/process models and ideation techniques @ellieexperiment
  23. 23. Enablement Diagnose a failure of How, Why, What So what? The retro shows: - Specifically how the team are struggling - Where the “How, Why, What” has failed - How they feel about it - The impact on their goal Now what? - Use the enablement model, and invest in the lagging areas.
  24. 24. Hopefully this has made it easier not to panic! In Summary: - Treat optimisation like a start-up inside your business - Get the experimentation mindset right - Enable experimentation by enabling the teams running experiments - Measure and fix! Need more ideas and tools you can use to improve experimentation, or want to try them out? - come talk to me! - checkout my blog post on medium Twitter: @EllieExperiment Medium.com/@EllieExperiment DON’T PANIC!!
  25. 25. Q&A
  26. 26. Thank you!
  27. 27. Session Exit Slide (Do Not Remove) Up next… ▪ Announcement 1 ▪ Announcement 2 ▪ Announcement 3

×