Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Optimizely & FedEx - Setting North-Star Metrics to Drive ROI from Experimentation

83 Aufrufe

Veröffentlicht am

Managing a large-scale testing program comes with its own unique set of challenges. As a leader, you are further removed from the actual experiments and more focused on the high-level approach. Your teams have different motivations and objectives that take effort to keep aligned. And it is hard to have the helicopter view on the program. Which teams are over and underperforming? What is the value that the entire experimentation program generated?

In this session, we will describe how leaders can create an organization that is aligned around a common North-Star metric that drives experimentation ROI and cascades down to individual team objectives. We will also discuss how to set up the experimentation workflow and metrics to create robust program-level and ROI reporting for other stakeholders.

Veröffentlicht in: Internet
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Optimizely & FedEx - Setting North-Star Metrics to Drive ROI from Experimentation

  1. 1. Hazjier Pourkhalkhali Global Director, Strategy & Value Optimizely Christiaan van der Waal VP of Digital FedEx Setting North Star Metrics To Drive ROI From Experimentation
  2. 2. 2 Using Your North Star to Drive Performance Hazjier Pourkhalkhali Global Director, Strategy & Value Optimizely Christiaan van der Waal VP of Digital FedEx
  3. 3. 3 Three quarters of companies surveyed reported that their digital revenues increase by over 5% because of experimentation n = 808 companies, >500 employees, self-reported (March 2019) 12% 14% 32% 26% 10% 5% No change or unsure Increased revenues by 1- 4% Increased revenues by 5- 9% Increased revenues by 10- 14% Increased revenues by 15- 19% Increased revenues by 20%+ SOURCE: “How to Succeed in the Digital Experience Economy” (March 2019)
  4. 4. 4 In August, professors at Harvard Business School and Duke Fuqua published the largest analysis of the value of Experimentation Experimentation and Startup Performance: Evidence from A/B Testing Rembrand Koning Sharique Hasan Aaron Chatterji Working Paper 20-018 Working Paper 20-018 Copyright © 2019 by Rembrand Koning, Sharique Hasan, and Aaron Chatterji Experimentation and Startup Performance: Evidence from A/B Testing Rembrand Koning Harvard Business School Sharique Hasan Fuqua School of Business, Duke University Aaron Chatterji Fuqua School of Business, Duke University and NBER Experimentation and Startup Performance: Evidence from A/B Testing Rembrand Koning Sharique Hasan Aaron Chatterji Working Paper 20-018
  5. 5. 5 They analysed over 35,000 startups over a period of 3 years Experimentation and Startup Performance: Evidence from A/B Testing Rembrand Koning Sharique Hasan Aaron Chatterji Working Paper 20-018 Working Paper 20-018 Copyright © 2019 by Rembrand Koning, Sharique Hasan, and Aaron Chatterji Experimentation and Startup Performance: Evidence from A/B Testing Rembrand Koning Harvard Business School Sharique Hasan Fuqua School of Business, Duke University Aaron Chatterji Fuqua School of Business, Duke University and NBER Experimentation and Startup Performance: Evidence from A/B Testing Rembrand Koning Sharique Hasan Aaron Chatterji Working Paper 20-018  Websites with and without experimentation snippets  Date snippets are added / removed  Pageviews, time on site  Venture Capital funding  Weeks with new products or features announced on key marketing websites n = 35,913 startups, 2015 – 2018
  6. 6. 6 Benefits of one year of experimentation for startups n = 35,913 startups, 2015 – 2018 PAGEVIEWS TIME ON SITE PRODUCTS LAUNCHED VC FUNDS RAISED +12% +4% +9-18% +10% >99.9% >99% >99% >95% Significance SOURCE: “Experimentation and Startup Performance” (Koning, Hassan, Chatterji 2019)
  7. 7. 7 Without clear business cases, even high performing programs face constant risks Organizational inertia halts growth or collaboration Executive inattention creates perpetual risk of backsliding Lack of resources and risk of losing resources to other projects Employees leave due to lack of recognition or career growth
  8. 8. 8 Without a clear business case Organizational inertia halts growth or collaboration You can generate urgency and momentum Executive inattention creates perpetual risk of backsliding You ensure executive focus Lack of resources and risk of losing resources to other projects You can better advocate for and protect resources Employees leave due to lack of recognition or career growth You can better recognize and reward performance With a clear business case…
  9. 9. It’s not my job to say what product decisions are right or wrong. My role is to be an organizational system architect of sorts. I need to ensure all the essential elements are in place to create the environment where innovation and experimentation can thrive. Mark Okerstrom CEO, Expedia Group “ ”
  10. 10. 10 Estimating returns from future experiments Average Test Impact Annual Experiments Win Rate Conservative Factor Average Uplift How many revenue driving experiments will you run over a year? What is the improvement to your financial metrics per experiment? Example: If 10% of experiments win on revenue, and the average winning uplift is 3%, then the test impact is 10% x 3% = 0.30% How much will we discount the total result in order to be conservative in our projections and give margin for error? What percentage of your digital revenue is affected by the average experiment? Revenue Scope Digital Revenue What is the digital revenue this property generates per year?
  11. 11. 11 Value Projection Example $125M ARR business, mobile and web experimentation, 42 experiments Annual Revenue Revenue Scope # of Experiments Average Test Impact Diminishing Return Conservative Factor Incremental Annualized Revenue Web Checkout £ 75 million 100% 20 tests per year 0.30% per test 50% +4.0% revenue £ 2.25 M per year Search £ 25 million 75% 10 tests per year 0.30% per test 50% +1.5% revenue £ 0.28 M per year Mobile App £ 50 million 50% 12 tests per year 0.30% per test 50% +1.2% revenue £ 0.45 M per year Total + £ 2.98 M per year
  12. 12. 12 Airbnb ran an experiment to attribute cumulative value to six winning experiments Bottoms-up 7.2% 5.3% Adjusted estimate 4.0% Split holdout
  13. 13. 13 2.1X Development resources are crucial to long-term success 8% 10% 11% 13% 15% 1 – 5 6 – 10 11 – 20 21 – 50 51 – 100 17%>100 Lines of Code / Variant Significant Uplift on Primary Metric
  14. 14. 14 You need to ask yourself two big questions: How willing are you to be confronted every day by how wrong you are? And how much autonomy are you willing to give to the people who work for you? And if the answer is that you don’t like to be proven wrong and don’t want employees decide the future of your products, it’s not going to work. – David Vismans Chief Product Officer, Booking.com “ ”
  15. 15. 15
  16. 16. 16
  17. 17. 2 Variations 3 Variations 4 Variations >5 Variations 77% 14% 5% 3% Experimentsrunbynumberofvariations
  18. 18. 2 Variations 3 Variations 4 Variations >5 Variations Significant uplift Significant reduction Inconclusive 77% 14% 5% 3%
  19. 19. +75% +48% +32% 25% 33% 37% 44% +75% 2 Variations 3 Variations 4 Variations >5 Variations Significant uplift Significant reduction Inconclusive Teamswiththefreedomtotestmorevariationsarefarmoresuccessful
  20. 20. Christiaan van der Waal Vice President of Digital FedEx
  21. 21. Transforming a 450,000-people company
  22. 22. 4 years of countless mistakes
  23. 23. The kind of mistakes that could get you fired (and still got to be promoted to VP)
  24. 24. We connect >99% of the world’s production,
  25. 25. handle 250 million tracking requests daily, (fun fact: we invented track & trace)
  26. 26. ship 15 million packages daily,
  27. 27. via 680 airplanes and 180.000 trucks.
  28. 28. The most expensive thing we ship is air
  29. 29. 32 Discussion and Q&A Hazjier Pourkhalkhali Global Director, Strategy & Value Optimizely Christiaan van der Waal VP of Digital FedEx
  30. 30. Thank you!

×