Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Myths, Lies and Illusions of AB and Split Testing

5.660 Aufrufe

Veröffentlicht am

What are the common assumptions about AB (split) testing that are wrong? What are the lies told by vendors, consultants and the stuff you have convinced yourself about. What is illusory - what can you trust - what's it really all about. 20 top myths debunked after asking fellow CRO professionals what is on THEIR top list.

Veröffentlicht in: Internet
  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Myths, Lies and Illusions of AB and Split Testing

  1. 1. Myths, Lies and Illusions of Split Testing @OptimiseOrDie
  2. 2. @OptimiseOrDie • UX, Analytics, Split Testing and Growth Rate Optimisation • Started doing testing & CRO 2004 • Split tested over 45M visitors in 19 languages • 67+ mistakes I MADE with AB testing • Like riding a bike… • Optimise your optimisation? Get in touch!
  3. 3. Myths, Lies and Illusions of Optimisation 1. Optimisation and Testing are the same 2. Optimisation and Testing are easy 3. Statistical significance tells you when to stop 4. High traffic always makes tests shorter 5. It’s all about Conversion Rates 6. You don’t need to test the test 7. Your tests will give you the quoted lift 8. High traffic pages give the best tests 9. It’s about getting all your tests to win 10. Optimisation is only about websites @OptimiseOrDie 11. Segmentation will tell you what happened 12. You can’t run multiple simultaneous tests 13. Testing is great for settling arguments 14. You can spot trends early on in a test 15. More test volumes = better results 16. Your tests tell you truths that last forever 17. You can test even on low traffic sites 18. Other people’s tests are ‘Best Practice’ 19. Doesn’t involve changing the way you work 20. Testing makes you a Data Scientist
  4. 4. #fail @OptimiseOrDie
  5. 5. @OptimiseOrDie 26.6M
  6. 6. @OptimiseOrDie28.4M
  7. 7. Oppan Gangnam Style! @OptimiseOrDie6.9M
  8. 8. You naughty boy…
  9. 9. @OptimiseOrDie1. Testing & Optimisation are the same thing Testin g Optimise
  10. 10. What is CRO/Optimisation? • “Using Analytics data and Customer feedback to improve the performance of your website.” • “Finding out why visitors aren’t converting and then resolving these issues.” • “Running loads of crappy split tests randomly until the heat death of the universe” @OptimiseOrDie
  11. 11. What is my definition? “A structured, systematic and continuous application of techniques that are used to discover, quantify and prioritise issues. These can be turned into hypotheses to drive experiments and opportunity in the following business outcomes:” • Increased revenue or profitability • Increasing LTV, loyalty, NPS/Sat scores, Kristoffer Potential • Removing cost from the business or contact center • Higher productivity or labour flexibility • Delighting customers • Reduced development effort @OptimiseOrDie
  12. 12. What Optimisation is NOT! • A way to change things you (or others) hate • A methodology for running split tests • A guarantee of increased conversion • A methodology for looking at analytics data • A rescue for silo-bound or non-agile design or development processes • A way to trick people into buying • A bolt-on – it IS the process @OptimiseOrDie
  13. 13. Optimisation is a: • Way of joining the worlds of Customer insight, UX, Analytics, Split testing and Business Strategy • Overarching Design and development process which prioritises work around opportunity • Strategic, not tactical, response to wasted development effort or product change • Way to create a deep and meaningful connection between the team, customers, business and the outcomes of making product changes • The killer app to remove ego, opinion, assumptions, cherished notions or ‘we just do it that way’ from decision making. • More powerful method than UX research or analytics alone, in guiding the directionality of product change @OptimiseOrDie
  14. 14. Optimisation includes these: • Qualitative research • Analytics, Quant analysis and insight • UX inspection and discovery • Competitive Intelligence • Priority based opportunity • VOC, Surveys and Customer Satisfaction (NPS) • Call tracking & Call centre optimisation • AB testing • Multivariate testing • Photography optimisation • EEG / ECG / Galvanic response @OptimiseOrDie • Web performance tuning • Forms analytics • Eye tracking • Market research • Big & Unstructured data analysis • PPC optimisation • Session replay analysis • Customer journey mapping • Ethnographic (diary) study research • Cross device, platform and channel insight • Email optimisation
  15. 15. 2. Optimisation & Testing are EASY
  16. 16. Surely you just add Javascript? • This work is harder than anything I’ve ever done! • You all have expensive & limited resources for testing - like Airport take-off slots • You MUST make use of these efficiently • You HAVE to balance resource cost with opportunity by prioritising carefully @OptimiseOrDie • You’re doing the equivalent of drug trials! • For large OR small companies, the instrumentation, analytics, tools setup, plan design, test methodology and analysis cannot be ‘done later on’. It’s not complex but it is vital. • The best companies have a structural shell (process, methodology, management) around their activities • If you’re not optimising the process continuously (Kaizen), you won’t increase your velocity of iterations. • Optimise the Optimisation
  17. 17. 3. Statistical significance tells you when to stop @OptimiseOrDie
  18. 18. The 95% Stopping Problem @OptimiseOrDie • Many people use 95, 99% ‘confidence’ to stop • This value is unreliable and moves around • Nearly all my tests reach significance before they are actually ready • You can hit 95% early in a test (18 minutes!) • If you stop, it could be a false result • Read this Nature article : bit.ly/1dwk0if • Optimizely have changed their stats engine • This 95% thingy – must be LAST on your stop list • Let me explain
  19. 19. The 95% Stopping Problem @OptimiseOrDie Scenario 1 Scenario 2 Scenario 3 Scenario 4 After 200 observations Insignificant Insignificant Significant! Significant! After 500 observations Insignificant Significant! Insignificant Significant! End of experiment Insignificant Significant! Insignificant Significant! “You should know that stopping a test once it’s significant is deadly sin number 1 in A/B testing land. 77% of A/A tests (testing the same thing as A and B) will reach significance at a certain point.” Ton Wesseling, Online Dialogue
  20. 20. The 95% Stopping Problem @OptimiseOrDie “Statistical Significance does not equal Validity” http://bit.ly/1wMfmY2 “Why every Internet Marketer should be a Statistician” http://bit.ly/1wMfs1G “Understanding the Cycles in your site” http://mklnd.com/1pGSOUP
  21. 21. 4. High Traffic always means short tests @OptimiseOrDie
  22. 22. Business & Purchase Cycles @OptimiseOrDie • Customers change • Your traffic mix changes • Markets, competitors • Be aware of all the waves • Always test whole cycles • Don’t exclude slower buyers • When you stop, let test subjects still complete! Start Test Finish Avg Cycle
  23. 23. • TWO BUSINESS CYCLES minimum (week/mo) • 1 PURCHASE CYCLE minimum • 250 CONVERSIONS minimum per creative (e.g. checkouts) • 350 & MORE! if response is very similar • FULL WEEKS/CYCLES never part of one • KNOW what marketing, competitors and cycles are doing • RUN a test length calculator - bit.ly/XqCxuu • SET your test run time , RUN IT, STOP IT, ANALYSE IT • ONLY RUN LONGER if you need more data • DON’T RUN LONGER just because the test isn’t giving the result you want! @OptimiseOrDie How Long? Simple Rules to follow
  24. 24. 5. It’s ALL about Conversion Rate @OptimiseOrDie
  25. 25. It’s all about the business @OptimiseOrDie • You’re optimising a business here, not a page or site • Tricking, pushing or persuading people at a superficial level to take an action is not a viable strategy • Your optimisation strategy is a series of steps, not a tool • Testing is about learning, not converting. • Tests that fail to tell you anything (regardless of outcome) are a failure themselves • If you don’t shift the business goals, your optimisation and testing budget will be threatened
  26. 26. 6. You don’t need to test the test – just go @OptimiseOrDie Browser testing www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.saucelabs.com www.multibrowserviewer.com Mobile devices www.appthwack.com www.deviceanywhere.com www.opendevicelab.com Read this article bit.ly/1wBccsJ
  27. 27. 7. The test result gives the promised lift @OptimiseOrDie
  28. 28. The result is a range @OptimiseOrDie • Version A is 3% conversion • Version B is 4% conversion • Yay! That’s a 25% lift • Let’s tell everyone • When it goes live, you get 5.7% • That’s because it was A RANGE • 3% +/- 0.5 • 4% +/- 0.4 • Actual result was 3.5% for A • Actual result was 3.7% for B
  29. 29. AB Testing Visualisation Tool @OptimiseOrDie abtestguide.com/calc/
  30. 30. 8. Testing is best on high traffic pages @OptimiseOrDie Think like the CEO of a department store! If you can’t refurbish the entire store, which floors or departments will you invest in optimising? Wherever there is: • Footfall • Low return • Opportunity
  31. 31. Ecommerce Bounce Search or Category Product Page Add to basket View basket Checkout Complete @OptimiseOrDie 8. Testing is best on high traffic pages
  32. 32. @OptimiseOrDie 8. It’s all about prioritisation!
  33. 33. 9. It’s all about WINNING test results @OptimiseOrDie
  34. 34. Failing is good @OptimiseOrDie • Tests that are ‘about the same’ are a failure • They’re also very hard to call • That means you have to be BOLD not conservative • A test that comes out negative is NOT a failure • If a ‘negative’ test teaches you something, it’s a success! • If you hit 40/50/60% failed tests, that’s fine • If you aren’t failing regularly, you’re not BOLD enough • Success is about the number of tests you finish each month, and what you learn from them
  35. 35. We believe that doing [A] for People [B] will make outcome [C] happen. We’ll know this when we observe data [D] and obtain feedback [E]. (reverse) @OptimiseOrDie
  36. 36. 10. Optimisation is only for Websites @OptimiseOrDie
  37. 37. Optimisation is just for websites @OptimiseOrDie • Service Design (Airbnb) • Onboarding flows with emails • Email templates • Apps – testing, debugging, tracking • Phone tracking and call centre optimisation • Social, Display, TV, Video and other advertising • Print adverts, Direct Mail • In-store promotions • Product manuals, guides, interfaces • EVERYTHING has elasticity – just find it • Even Multi-variate call centre scripts
  38. 38. 11. Segmentation will explain things @OptimiseOrDie
  39. 39. Segmentation explains stuff @OptimiseOrDie • Beware of small sample sizes • A = 350 conversions • B = 300 conversions • A Conversions for Safari = 20 • B Conversions for Safari = 25 • Only needs 2 people to change that from 25% lift to 14%
  40. 40. 12. You can’t run concurrent split tests @OptimiseOrDie
  41. 41. Oh yes you can, with GA! @OptimiseOrDie • If you push events or variables into GA, you can report on behaviour for A or B (or any variations). • If you do it that way, you can easily run multiple tests on different page targets simultaneously. • You grab AAA, AAB, ABA, ABB and analyse. • Test subjects get a recipe of tests, so one caveat • If you pick things that clash or jar the experience • If you have $5 delivery but then $2 in another test • Apart from that, it’s fine to run these • They tell you about the right experience recipe across several pages being optimised in concert
  42. 42. 13. Testing is great for settling arguments @OptimiseOrDie
  43. 43. 14. You can spot trends early in a test @OptimiseOrDie
  44. 44. You can spot trends early @OptimiseOrDie • Tests are volatile in the early stages • Watch but shut your mouth and wait • Keep an eye on any odd behaviour • Be patient!
  45. 45. 15. More tests = Better results @OptimiseOrDie
  46. 46. More tests = Better results @OptimiseOrDie • Increasing volume without optimising velocity • Not optimising the process means it scales badly • If you put garbage in, you’ll get garbage out • Ramping up before you have it tuned is crazy • Get the team, process, project management, methodology, toolkits, selling and PR nailed first • THEN optimise and scale • I know that one UK retailer is doing 140 per month! • That’s LOW compared to some companies I work with • Big companies doing 2 a month? Meh.
  47. 47. 16. Each test is a truth FOREVER @OptimiseOrDie
  48. 48. Tests are a Truth Forever @OptimiseOrDie • Traffic changes • Prices change • Product mix changes • Advertising evolves and changes • Markets are different • Customers have changed • Competitors or regulatory landscape moves • Things happen outside of your control • You need to revisit tests or ITERATE • Always be trying to beat the new winner, not basking in the glory of a test you ran 9 months ago • The lift may have vanished • Schrodinger’s AB test
  49. 49. 17. Testing is fine on Low Traffic sites @OptimiseOrDie
  50. 50. Testing is fine on low traffic sites @OptimiseOrDie • Yes, if you estimate your minimum testing unit • This is the time to AB test a sample of say 250 conversions, with the rules I set earlier. • A payday loan company? 2 months minimum! • Run the calculations • Check the test length • If it takes like, 8 million years, what can you do? • Read this or download my AB testing decks: bit.ly/1umy5Y6
  51. 51. 18. Other people’s tests are Best Practice @OptimiseOrDie “STOP copying your competitors They may not know what the f*** they are doing either” Peep Laja, ConversionXL
  52. 52. Tests you see online? @OptimiseOrDie • Your customers are not the same • Your site is not the same • Your advertising and traffic is not the same • Your UX is not the same • How the f*** do you think it guarantees a result? • Use them to inform or suggest ideas • They’re like the picture on meal packets • Serving Suggestion Only
  53. 53. 19. Testing makes you a Data Scientist @OptimiseOrDie
  54. 54. 20. Testing and Innovation is a plugin @OptimiseOrDie
  56. 56. Guessaholics Anonymous
  57. 57. 2004 Headspace What I thought I knew in 2004 Reality
  58. 58. 2014 Headspace What I KNOW I know Me, on a good day
  59. 59. Rumsfeldian Space
  60. 60. @OptimiseOrDie #1 Culture & Team #2 Toolkit & Analytics investment #3 UX, CX, Service Design, Insight #4 Persuasive Copywriting #5 Experimentation (testing) tools The 5 Legged Optimisation Barstool
  61. 61. @OptimiseOrDie Marketing ? Variation Heritability of good ideas Selection based on death
  62. 62. Business Future Testing? Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy. Mr D. Vader
  63. 63. Thank You! Mail : sullivac@gmail.com Deck : Linkedin : linkd.in/pvrg14
  64. 64. READ STUFF
  65. 65. READ STUFF
  66. 66. READ STUFF
  67. 67. #5 : FIND STUFF @OptimiseOrDie @danbarker Analytics @fastbloke Analytics @timlb Analytics @jamesgurd Analytics @therustybear Analytics @carmenmardiros Analytics @davechaffey Analytics @priteshpatel9 Analytics @cutroni Analytics @avinash Analytics @Aschottmuller Analytics, CRO @cartmetrix Analytics, CRO @Kissmetrics CRO / UX @Unbounce CRO / UX @Morys CRO / Neuro @UXFeeds UX / Neuro @Psyblog Neuro @Gfiorelli1 SEO / Analytics @PeepLaja CRO @TheGrok CRO @UIE UX @LukeW UX / Forms @cjforms UX / Forms @axbom UX @iatv UX @Chudders Photo UX @JeffreyGroks Innovation @StephanieRieger Innovation @BrianSolis Innovation @DrEscotet Neuro @TheBrainLady Neuro @RogerDooley Neuro @Cugelman Neuro @Smashingmag Dev / UX @uxmag UX @Webtrends UX / CRO
  68. 68. #5 : LEARN STUFF @OptimiseOrDie Baymard.com Lukew.com Smashingmagazine.com ConversionXL.com Medium.com Whichtestwon.com Unbounce.com Measuringusability.com RogerDooley.com Kissmetrics.com Uxmatters.com Smartinsights.com Econsultancy.com Cutroni.com www.GetMentalNotes.com
  69. 69. #12 : The Best Companies… • Invest continually in analytics instrumentation, tools, people • Use an Agile, iterative, cross-silo, one team project culture • Prefer collaborative tools to having lots of meetings • Prioritise development based on numbers and insight • Practice real continuous product improvement, not SLEDD* • Are fixing bugs, cruft, bad stuff as well as optimising • Source photos and content that support persuasion and utility • Have cross channel, cross device design, testing and QA • Segment their data for valuable insights, every test or change • Continually reduce cycle (iteration) time in their process • Blend ‘long’ design, continuous improvement AND split tests • Make optimisation the engine of change, not the slave of ego * Single Large Expensive Doomed Developments
  70. 70. THE FUTURE OF TESTING : www.conductrics.com