Smart web and ecommerce companies companies are turning to Website Optimization (A/B and MVT Testing) as the next big area for growth. When done right, optimizing your website can be the most highly leveraged function within web marketing. Most companies fail at the first step - organization design - and never get their A/B testing program off the ground. Jared Waxman discusses how Intuit changed up the organization process to achieve one-day A/B testing sprints, and saw multi-million dollar payback.

1. Jared Waxman Jared Waxman presents… Fast & Furious: 10 Principles for Rapid A/B Testing Questions? Email jared@benchmark-analytics.com

2. Experiences and Lessons While Running A/B Testing for Ecommerce SITES at: Makers of Quicken,TurboTax, QuickTaxand Quickbooks Presented at Internet Marketing Conference Sept 16-17 Vancouver, CANADA

3. Intuit Small Business – Web Business How much: On order of ~$100 M annual revenue transacted online What: Desktop software, Services, Software-as-a-Service (subscription model) Where: 45 Websites, 10 Shopping Carts Traffic: QuickBooks.com ~ 1 M monthly unique cookies 3 Jared Waxman

4. QuickBooks.com – Various Timepoints 4 Jared Waxman

5. I’ve tested… I’ve run 1 or MORE tests against each of these hypotheses: How much does a Free product cannibalize Paid products? What’s the optimal way to position a Free product in the lineup to maximize all units while minimizing paid cannibalization? Does one Free offering cannibalize less than another Free offering? What is the most effective “voice” used in describing the product? What category of images sells the product the best? 5 Jared Waxman

6. …and tested… Do videos sell better than static images? Do expanded versus collapsible product specs perform better? Does having a special price for upgraders yield higher overall sales than a flat pricing model? Does it improve customer sentiment? Does exposing retail options significantly shrink direct online sales? Which discount % level of promotional price discounts yields highest revenue? Does adding cart CTA on home and line-up pages increase conversion or change pathing? What is the optimal color, size, placement, and copy on CTAs? 6 Jared Waxman

7. …and tested… Where is the optimal place in the purchase funnel for customer to configure their product? Where is the optimal place in the purchase funnel to ask for credit card info? How many pages in the checkout flow is optimal? What home page weighting of offerings moves us to the desired platform mix while continuing to grow revenue? How effective are certain product chooser tools, demo, or guides in moving conversion rates? How effective is a certain cart abandonment tactic in moving conversion rates? 7 Jared Waxman

8. …and tested… Where is the right step in the purchase funnel to cross-sell other offerings? How many cross-sell offerings are optimal for a given funnel step? Should cross-sells be passive or intrusive? How effective and efficient is proactive chat in boosting over conversion? How many CTAs and/or featured offerings are optimal on the home page? How many offerings is optimal within the line-up? 8 Jared Waxman

9. …and tested… Are bullet points or prose styles more effective on landing pages? Are PPC ad group specific links on landing pages more effective than generic links? What’s the likelihood of successful for a new product offering if added to the online store? How would a New User vs Existing User focused site perform against overall online revenue? Can lift conversion rates by guessing at New vs Existing and then sending them down different paths? How effective would a simple User Contribution System be at engaging and converting prospects? 9 Jared Waxman

10. …and tested. Does simplifying the home page by removing promotions on the homepage to other auxiliary Company sites/product offerings reduce the net traffic that gets sent? Do menu titles perform better if they are job-oriented or product-oriented? Do promos perform better as %s or as $s off? Do promos perform better as %/$ off vs Free Shipping? Does a certain messaging approach consistently outperform another regardless of specifics? 10 Jared Waxman

11. What has been the benefit of testing? 11 Half our growthin the online business comes from testing the pure site experience. The other half from acquisition programs. Testing is highly leveraged growth – a handful of employees. Produces 3-digit ROI. Acquisition programs requires serious cash and overall produces 2-digit ROI. Jared Waxman

12. Today’s Focus: Organization Design 12 These gears must all work together. Each deserves separate treatment. Focus of this presentation Jared Waxman

14. 11 Factors Tested

15. 26 Recipes In All1 Big Winner: 20% increase on the success formula Annual impact ~ $10M Plus… 1 Loser (but killed quickly) Some breakevens Jared Waxman

16. Then Narrow (8 Recipes) 3/4/09 Control Product A: Index 1.60 Formula: Index 1.20 WE/WD 3 Narrow Again (4 Recipes) 4/9/09 Go Broad (8 recipes) 2/20/09 1 2 4 Product A: Index 2.00 And Broad, Again! 3/9/09 Formula: Index 1.29 Weekend, .91 Weekday Rapid Testing Team Formula: Index 0.50 Jared Waxman

17. Top 10 “Go Fast” Testing Principles 10. Testing “Strike Force” with no outside dependencies 9. Exactly one “Approver” 8. One straight-forward formula for success 7. Inform at go-live, then again with results 6. Fully-baked ideas as the starter, then testing team tightens it. After phase 1, iterate with true MVT and refine as appropriate. 5. Test ideas based in theory and customer data. Then go test, not talk. 4. Manage to deadlinewith quality. Reduce scope as needed! 3. Take in ideas from anywhere and everywhere. Don’t take in orders. 2. Open, flexible test calendar basing next tests on data from existing ones. No pre-slotted commitments and finally… If you follow the prior 9 principles you’ll be able to deliver… 1. One-Day Test Sprints! From schedule-it to go-live in 8 hrs. Jared Waxman 15

20. Management not aligned – competing goalsJared Waxman

21. In the org, you’ll encounter… Digging Deeper into Organization Design Jared Waxman

22. HiPOs 18 Jared Waxman

23. HiPOs: Tips Get Management to understand why testing success probability means it’s in their interest to have a healthy distance from individual test designs. It’s in your interest too. Get Management engaged at the right level – setting hierarchy of goals and resourcing for highly leveraged testing. 19 Jared Waxman

24. Targets 20 Jared Waxman

25. Targets: Tips Set by Management, not testing team. Testing Team Lead responsible to push back if needs more constraints. Put in place a hierarchy of success metrics. Should be consistent across seasons, managers, businesses. Revenue per visitor (RPV) typically at top of hierarchy. Can assign each success metric it’s own team. 21 Jared Waxman

26. Design by Committee Jared Waxman

27. Design By Committee: Tips Hire a Test Lead whose business instinct and creative eye you know and trust. Keep him/her in the kitchen on business strategy. Then let him/her be the one and only approver on what goes live. Ideas get thrown “over the wall” to the testing team. Testing Teams set up as neutral, with no built-in biases towards specific businesses, ideas, or managers. Idea submitter and/or related business unit have no official role during the execution phase. 23 Jared Waxman

28. Sacred Cows Jared Waxman

29. Sacred Cows: Tips 25 Assign Executives to Goals Hierarchy. Assign Segment Marketers job of throwing ideas “over the wall”. Assign the Test Lead the job of selecting and executing. Separate Segment Marketers from Testing. Get a clear list of constraints, a consistent success metric, and then challenge everything. Outside viewpoint. Much easier to get people think outside the box when they know some crazy-sounded idea worked somewhere else. Test big. And if you can’t, test small. Remember you can also pull if it’s losing money. Influence through your framing. Testing can be framed not as a challenge, but a way to quantify a “known” lever so it can be expanded or further optimized. Jared Waxman

30. Stampedes Jared Waxman

31. Stampedes: Tips Cordon off your Testing Teams to focus your highly leveragable resources Set clear, long-term goals for the Teams. Empower the Team Leads to decide priorities, timing, workload. Leverage outside agencies to rapidly expand capabilities and expertise as needed. 27 Jared Waxman

32. Jared Waxman Fast & Furious: 10 Principles for Rapid A/B Testing Want to learn more? Email jared@benchmark-analytics.com