Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015

2.464 Aufrufe

Veröffentlicht am

My Slides from Reaktor Breakpoint 2015 - This is by far the best deck (and hopefully talk) I've done this year. Masses of info, reading, articles, useful reports and more.

Veröffentlicht in: Internet
  • Isn't it about time you started ➜➜➜ https://t.cn/A6ybK1ra
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/ydNlH ◀ ◀ ◀ ◀
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015

  1. 1. Surviving the Hype Cycle Shortcuts to Split Testing Success @OptimiseOrDie, Yo Helsinki!
  2. 2. @OptimiseOrDi e • Split Testing, Analytics, UX • Lean, Agile, Growth Optimisation • 100M+ visitors tested • 200+ sites, 26 languages • Starting Testing? Bad ROI? • Want to scale volume & quality? • Get in touch! @OptimiseOrDiecolinski.tumblr.com
  3. 3. @OptimiseOrDie The Gartner Hype Cycle ™
  4. 4. 1 Tool Installed 2 Stupid testing 3 4 Peak of Stupidity 5 ROI questioned 6 Statistics debunked 7 Faith crisis 8 The Trough of Testing Scaled up Stupidity 9 Where, How, Why 10 Data science 11 Testing to learn 12 Innovation Testing @OptimiseOrDie
  5. 5. #fai l @OptimiseOrDie
  6. 6. @OptimiseOrDie 26.6M
  7. 7. @OptimiseOrDie28.4M
  8. 8. Oppan Gangnam Style! @OptimiseOrDie6.9M
  9. 9. You been naughty again?
  10. 10. 1. Get Analytics Health Checked 2. Test in the right place 3. Understand Cross Device 4. Do your Research 5. Prioritise your testing @OptimiseOrDie 6. Perform Pre Flight Checks 7. Know how long to test 8. Have a good reason to test 9. Learn from your tests 10. Burn down the silos 10 Shortcuts to Testing Success
  11. 11. @OptimiseOrDie 1. Your Analytics Setup is Broken
  12. 12. • Nearly 100 Sites in 3 years • 95% were broken, often badly • Trust in data was missing • Management made bad calls • Nobody checked the tills • Calibrate from the basics up! @OptimiseOrDie • What sales do we capture? • What categories? • What about refunds, lunch money, gift certificates? • How do we monitor fraud? • Do we check it adds up? • Where does this data go? 1. What about MY clients?
  13. 13. @OptimiseOrDie 1. Bulls***t flows upwards! Cool BS Dashboard BS metrics BS Collection BS metrics BS Collection BS metrics BS Collection BS metrics BS Collection BS reports BS Reports TILLS DEPT STORE DIVISIO N
  14. 14. @OptimiseOrDie • Review takes 1-3 days • Prioritise the issues • Fix directly with developers • Integrate with the Testing Tool1 Get an Analytics Health Check
  15. 15. @OptimiseOrDie 2. You Test in the Wrong Place
  16. 16. 2. Let’s do Random Testing Let’s try the homepage I’ve got targets to hit! I hate this job Let’s test button colours! Has lots of opinions but no data Spends too much time on Twitter Driven by Ego and Competitors Wishes he cared about testing
  17. 17. @OptimiseOrDie “STOP copying your competitors They may not know what the f*** they are doing either” Peep Laja, ConversionXL 1. Let’s do Random Testing
  18. 18. Best Practice Testing? • Your customers are not the same • Your site is not the same • Your advertising and traffic are not the same • Your UX is not the same • Your X-Device Mix is not the same • You have no idea of the data • Use them to inform or suggest approaches • Use them for ideas • Do not use them as a playbook • It will make you very unhappy @OptimiseOrDie
  19. 19. @OptimiseOrDie 2. Modelling - Intent All traffic HearingSight StoreOther Step 1 Step 2 Step 3 Goal Page PageHearing
  20. 20. @OptimiseOrDie 2. Modelling – Multiple Endings All traffic InfluenceIntent Influence Step 1 Step 2 Step 3 Goal Page Page Entry Page 1 2 3 4 1 2 3 4 1 2 3 4
  21. 21. Bounce Search or Category Product Page Add to basket View basket Checkout @OptimiseOrDie 2. Modelling – Ring Model
  22. 22. 2. Modelling – Horizontal Funnels
  23. 23. @OptimiseOrDie • Do some Analytics modelling • Understand the shedding of layers • Narrow your focus and scope • Bank better gains earlier in time 2 Test in the Right Places
  24. 24. @OptimiseOrDie 3. Responsive solves everything, right?
  25. 25. @OptimiseOrDie
  26. 26. @OptimiseOrDie Vs.
  27. 27. @OptimiseOrDie
  28. 28. @OptimiseOrDie
  29. 29. @OptimiseOrDie
  30. 30. @OptimiseOrDie
  31. 31. @OptimiseOrDie 1. Motorola Hardware Menu Button 2. MS Word Bullet Button 3. Android Holo Composition Icon 4. Android Context Action Bar Overflow (top right on Android devices)
  32. 32. @OptimiseOrDie Increase in revenue of > $200,000 per annum! bit.ly/hamburgertest
  33. 33. BURGER SHISH DONER
  34. 34. @OptimiseOrDie • Do you really know your mix? • Most people undercount Android! • What iPhone models visit? • How big is tablet traffic? • What screen sizes do they have? • Find out BEFORE you design tests • Check BEFORE you launch tests • Use Google Analytics to find out • 3 reports to rule them all https://www.google.com/analytics/web/template?uid=lpVf8LveSqyd3mdsHjdfzQ https://www.google.com/analytics/web/template?uid=fmUzp_gzRIy7LnvZJjCDOQ https://www.google.com/analytics/web/template?uid=y7sYIXDhQrmswHAiNo8iLA 3. Our customers use iPhones, right?
  35. 35. @OptimiseOrDie 3. What iPhone Models do we see? Screen Resolution 320 x 480 = iPhone 4/4S 320 x 568 = iPhone 5/5S 375 x 667 = iPhone 6 414 x 736 = iPhone 6+ https://www.google.com/analytics/web/template?uid=lpVf8LveSqyd3mdsHjdfzQ https://www.google.com/analytics/web/template?uid=fmUzp_gzRIy7LnvZJjCDOQ https://www.google.com/analytics/web/template?uid=y7sYIXDhQrmswHAiNo8iLA
  36. 36. @OptimiseOrDie • Desktop Browsers & versions • Tablet Models • Mobile Device Models • Screen Resolutions3 Figure Out the Device Mix for Testing
  37. 37. Is there anything holding you from doing conversion research? 1. Time 2. Client/Company Buy-In 3. Budget 4. Don’t know where to start 4. You don’t do any Research before testing?
  38. 38. 40
  39. 39. @OptimiseOrDie 4. If you have 4 hours PLUS • Snap interviews (Sales, Customer Services, Tech Support) • Run a quick poll or survey (See my tools slides) Less Bullshit!
  40. 40. @OptimiseOrDie 4. 1 Hour Page Analytics Influence Pages Entry Points Landing Pages Device Mix Customer Mix Traffic Mix Flow Intent Marketing -> Site flow Page or Process Next Steps Abandonment Exits Mix of abandonment Flow
  41. 41. @OptimiseOrDie 4. 1 Hour Landing Page Analytics • How old are the visitors? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What are the key metrics like (e.g. bounce rate, conversion)? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What is the goal or ecommerce conversion through this page? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What channel traffic comes to the page? https://www.google.com/analytics/web/template?uid=Kjb9q8M4QN-fsPe8dOGaig • What is the mix of tablet / mobile / desktop to the page? https://www.google.com/analytics/web/template?uid=wLMUWs8eTIa3_mmQHOtPkw • What are the resolutions of devices? https://www.google.com/analytics/web/template?uid=wLMUWs8eTIa3_mmQHOtPkw • How slow are the landing pages? https://www.google.com/analytics/web/template?uid=AavFsgMoRkucYYKnxlB76Q • What are the pages right after the landing page? (Use a landing page report and choose the ‘Entrance Paths’ to show next pages.) • What is the flow like from this page? (Use the Behaviour Flow Report) • What does it look like on the top devices? (Use real devices + Appthwack.com, Crossbrowsertesting.com or Deviceanywhere.com)
  42. 42. Ad SERP Banner Email Campaign Aff Referrer Landing Page Template Goal Reached Interaction Step or Layer PPC Organic Display Email Social Desktop - Mobile - Tablet
  43. 43. @OptimiseOrDie 4. If you have 2 hours • Form Analytics data • Scroll or Click Maps • Session Recording Videos (Hotjar, Decibel Insight, Yandex) • Make a horizontal funnel from the landing page • Check the: – Marketing Creatives / SERP fully – Look at Landing page ZERO!
  44. 44. @OptimiseOrDie 4. If you have 4 hours • Set up a poll or survey (See my tools slides) • Set up an exit (bail) survey • Friends, Family, New Employee user testing • Guerrilla user testing • Snap Interviews – 5-10 minutes: • Customer Services, Sales team (if applicable) then Customers • 5 Second Test • Article is here : bit.ly/conversiondeadline
  45. 45. @OptimiseOrDie • Lean Analytics • UX Research • Interviewing • Surveys and Polls4 No Excuses – Do Your Research
  46. 46. @OptimiseOrDie 5. You don’t Prioritise your Tests Scoring can be cost, time to market, resource, risk, political complexity • Cost 1-10 Higher is cheaper • Time 1-10 Higher is shorter • Opportunity 1-10 Higher is greater Score = Cost * Time * Opportunity • For financial institutions, risk should be a factor • Want to build your own? – ask me!
  47. 47. @OptimiseOrDie 5. Opportunity vs. Cost 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 Cheap (high is better) Opportunity MONEY!
  48. 48. @OptimiseOrDie 5. Make a Money Model Test Description Metric 2% lift 5% lift 10% lift Estimate Product page Simplification Basket adds 200,000 500,000 1,000,000 500,000 Register new Improve onboarding New register funnel ratio 25,000 62,500 125,000 250,000 IE8 bugs in cart Fix key broken stuff IE8 Conversion 80,000 200,000 400,000 200,000 Category list page Get product higher User Category -> Product 500,000 1,250,000 2,500,000 1,250,000 Payment Page New card handling User Payment -> Thank you 60,000 150,000 300,000 300,000
  49. 49. @OptimiseOrDie • Score all Test Targets • Use Cost vs. Opportunity minimum • Check it works! • Make a Money Model 5 Prioritise your Testing Targets
  50. 50. @OptimiseOrDie 6. You Don’t Test Before Launch • Dirty secret of AB testing? • People break their tests all the time! • Most people don’t notice Why? • Because developers can break them very easily • What if your AB test was broken on iPhones? • If you didn’t know, would your results be valid? • About 40% of my tests fail basic QA
  51. 51. Browser Checks www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.saucelabs.com Mobile & Tablet www.appthwack.com www.deviceanywhere.com www.opendevicelab.com Article & Info bit.ly/devicetesting 6. Here is my £80M testing rig!
  52. 52. @OptimiseOrDie 6. Always Use Real Devices
  53. 53. @OptimiseOrDie • Check Every Test Works • Browser and Devices • Check Analytics records correctly 6 Perform Pre Flight Checks
  54. 54. @OptimiseOrDie 7. You Stop When you Hit 95% Confidence
  55. 55. The 95% Stopping Problem @OptimiseOrDie • Many people use 95, 99% ‘confidence’ to stop • This value is unreliable and moves around • Nearly all my tests reach significance before they are actually ready • You can hit 95% early in a test (18 minutes!) • If you stop, it could be a false result • Read this Nature article : bit.ly/1dwk0if • Optimizely and VWO have updated their tools • This 95% thingy – must be LAST on your stop list
  56. 56. The 95% Stopping Problem Scenario 1 Scenario 2 Scenario 3 Scenario 4 After 200 observations Insignificant Insignificant Significant! Significant! After 500 observations Insignificant Significant! Insignificant Significant! End of experiment Insignificant Significant! Insignificant Significant! “You should know that stopping a test once it’s significant is deadly sin number 1 in A/B testing land. 77% of A/A tests (testing the same thing as A and B) will reach significance at a certain point.” Ton Wesseling, Online Dialogue
  57. 57. The 95% Stopping Problem @OptimiseOrDie “Statistical Significance does not equal Validity” http://bit.ly/1wMfmY2 “Why every Internet Marketer should be a Statistician” http://bit.ly/1wMfs1G “Understanding the Cycles in your site” http://mklnd.com/1pGSOUP
  58. 58. • TWO BUSINESS CYCLES minimum (week/month) • 1 PURCHASE CYCLE minimum (or most of one) • 250 CONVERSIONS minimum per creative • 350, 500, more if creative response is similar • FULL WEEKS/CYCLES never part of one • KNOW what marketing, competitors and cycles are doing • RUN a test length calculator - bit.ly/XqCxuu • SET your test run time , RUN IT, STOP IT, ANALYSE IT • ONLY RUN LONGER if you need more data • DON’T RUN LONGER just because the test isn’t giving the result you want! @OptimiseOrDie 7. Know How Long to Test for…
  59. 59. @OptimiseOrDie • Most critical mistake • Use a test calculator • Full business cycles, 2 minimum • Don’t waste time hoping7 Know How Long to Test for
  60. 60. Insight - Inputs #FAIL Competitor copying Guessing Dice rolling An article the CEO read Competitor change Panic Ego Opinion Cherished notions Marketing whims Cosmic rays Not ‘on brand’ enough IT inflexibility Internal company needs Some dumbass consultant Shiny feature blindness Knee jerk reactons @OptimiseOrDie 8. So you think you have a Hypothesis?
  61. 61. Insight - Inputs Insight Segmentation Surveys Sales and Call Centre Session Replay Social analytics Customer contact Eye tracking Usability testing Forms analytics Search analytics Voice of Customer Market research A/B and MVT testing Big & unstructured data Web analytics Competitor evalsCustomer services @OptimiseOrDie 8. So you think you have a Hypothesis?
  62. 62. @OptimiseOrDie 1. Because we saw (data/feedback) 2. We expect that (change) will cause (impact) 3. We’ll measure this using (data metric) bit.ly/hyp_kit 8. Use this to deflect stupid testing!
  63. 63. @OptimiseOrDie 1. Because we saw (an angry email from the CEO) 2. We expect that (changing button colours) will cause (the office to cool down for a day) 3. We’ll measure this using (some metric we pluck out of the air – whatever, man) bit.ly/hyp_kit 8. Let’s try a real one
  64. 64. @OptimiseOrDie • Don’t do Ego driven testing • Use the Hypothesis Kit! 8 Get a Proper Hypothesis Going bit.ly/hyp_kit
  65. 65. @OptimiseOrDie 9. Our Testing teaches us Nothing! • Either your research or hypothesis is weak • Work back from the outcome! What if A won – what would that tell us? What if A failed – what would that tell us? • What is the value to the business in finding out the answer? • Is the finding actionable widely and deeply? • Testing isn’t about lifts – it’s about learning
  66. 66. @OptimiseOrDie 9. Our Testing teaches us Nothing! “You are trying to run a bundle of tests, whose expected additional information will give you the highest return.” Matt Gershoff, CEO, Conductrics.com
  67. 67. @OptimiseOrDie • Do your research • Form a solid hypothesis • Work back from the outcomes • Learning useful stuff = huge lifts9 Design Tests for Maximum Learning
  68. 68. @OptimiseOrDie 10. Burn Down the Silos • Non agile, non iterative design • Silos work on product separately • No ‘One Team’ per product/theme • Large teams, unwieldy coordination • Pass the product around • More PMs and BAs than a conference • Endless sucking signoff • AB testing done the same way!
  69. 69. @OptimiseOrDie 10. FT Example • Small teams (6-15) with direct access to publish • Ability to set and get metrics data directly • Tools, Autonomy, Lack of interference • No Project Managers or Business Analysts • Business defines ‘outcomes’ – teams deliver • No long signoff chain • No pesky meddling fools • 18 Month projects over budget?
  70. 70. @OptimiseOrDie 10. FT Example • 100s of releases a day! • MVP approach • Launch as alpha, beta, pilot, phased rollout • Like getting in a shower • Read more at labs.ft.com
  71. 71. @OptimiseOrDie 10. Positive Attributes • Rapid, Iterative, User Centred & Agile Design. No Silos. • Small empowered autonomous teams • Polymaths and Overlap • Toolkit & Analytics investment • Persuasive copywriting & Psychology • Great Testing & Optimisation Tools
  72. 72. @OptimiseOrDie • Agile, Lean, Iterative x silo teams • Ability to get and set metrics • Autonomy, Control, Velocity • Iterative MVP approach • Work on outcomes, not features 10 Burn Down the Silos!
  73. 73. 76 “If you think of technology as something that’s spreading like a sort of fractal stain, almost every point on the edge represents an interesting problem.” Paul Graham
  74. 74. 77 Time ROI
  75. 75. Rumsfeldian Space• What if we changed our prices? • What if we gave away less for free? • What if we took this away? • What about 3 packages, not 5? • What are these potential futures I can take? • How can I know before I spend money? • McDonalds Hipster Test Store bit.ly/1TiURi7 @OptimiseOrDie
  76. 76. Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy! Innovation Testing @OptimiseOrDie
  77. 77. @OptimiseOrDie
  78. 78. 2004 Headspace What I thought I knew in 2004 Reality
  79. 79. 2015 Headspace What I KNOW I know Me, on a good day
  80. 80. WE’RE ALL WINGING IT
  81. 81. Guessaholics Anonymous
  82. 82. 1 Tool Installed 2 Stupid testing 3 4 Peak of Stupidity 5 ROI questioned 6 Statistics debunked 7 Faith crisis 8 The Trough of Testing Scaled up Stupidity 9 Where, How, Why 10 Data science 11 Testing to learn 12 Innovation Testing @OptimiseOrDie
  83. 83. Thank You! Email me sullivac@gmail.com Slides bit.ly/reaktorOOD Linkedin linkd.in/pvrg14

×