Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

#EpicFail! A/B Test Results vs. UX Best Practices

If you design a flow, you’re going to include a progress indicator, right? If you have a button, you’re going to specify where it takes the user, right? Well, don’t tell the thought leaders we told you this, but it really does depend! At Vistaprint, we are constantly running “Mega Split Run” tests (i.e., A|B testing on steroids!). We balance these test results with site metrics and user-centered design principles to inform our design decisions. Some UX best practices have been validated for our customers, while others we flat out ignore -- we’ll share examples of how tried and true usability best practices aren’t always the right answer. Our discussion will highlight some of the takeaways that brought us to this conclusion. We’ll ask the audience to compare notes on collective experiences, hypotheses and expectations around design performance. Finally, we’ll discuss how we balance site performance testing with user centered design activities as a whole, including usability research and testing. This presentation will be a lecture format with some audience participation.

  • Loggen Sie sich ein, um Kommentare anzuzeigen.

#EpicFail! A/B Test Results vs. UX Best Practices

  1. 1. #EpicFail!A/B Test Results vs. UX Best PracticesSusan Rice & Kirk DoggettGlobal UX Center of Excellence 5/22/2012 UPA Boston Conference 1
  2. 2. Overview• Introduction• Examples – Pop Up Windows – Progress Indicators – Homepages – Buttons – Website Purchase Flow – People Don‘t Read• Summary• Questions? 5/22/2012 UPA Boston Conference 2
  3. 3. Who We AreVistaprint provides high-impact personalizedproducts and electronic services that empowermicro businesses and consumers at an affordableprice.Company Facts• 25 localized websites• 3,700 employees• 3 state of art manufacturing facilities• 13 offices worldwide 5/22/2012 UPA Boston Conference 3
  4. 4. ―Test Before You Invest‖We have dozens of multivariate―split run‖ site tests live at any User Populationgiven time during our 3 weekrelease cycles.• Scalable: Random Most site changes are measured Sample before rollout• Low Risk: Price Price Multiple tests can be run Control Test simultaneously with a sample UI of the population Control 5% 5%• Accurate: UI Test sample is representative Test 5% 5% of user base 5/22/2012 UPA Boston Conference 4
  5. 5. UX & A/B Testing• UX designs the UI and helps inform the test plan.• Test runs and is analyzed.• Layer the quantitative with the qualitative to create a holistic picture to inform future design decisions.But what happens when A/B testing results challenge triedand true usability principles and best practices? 5/22/2012 UPA Boston Conference 5
  6. 6. POP UP WINDOWS 5/22/2012 UPA Boston Conference 6
  7. 7. What the experts say…• Avoid popup windows: [1] – Extra windows keep users from getting to your site content, and even if such windows contain valuable information, users are likely to dismiss them immediately as ads. – The other drawback to popup windows is that they go away — once the user gets rid of them, theyre gone, so users often cant find the information again even if they want to.• Do not have unsolicited windows or graphics ―pop up‖ to users.[2] – Users have commented that unsolicited windows or graphics that ‗pop up‘ are annoying and distracting when they are focusing on completing their original activity.• Selective attention is very powerful, and Web users have learned to stop paying attention to any ads that get in the way of their goal-driven navigation. [3] – “pop-up purges mean that users close pop-up windoids before they have even fully rendered; sometimes with great viciousness‖ [1] Jakob Nielsen, excerpt from 113 Design Guidelines for Homepage Usability, 2001 [2] US Government, Research-Based Web Design & Usability Guidelines, 2006 [3] Jakob Nielsen, Top 10 Mistakes in Web Design, updated 2011 5/22/2012 UPA Boston Conference 7
  8. 8. What the experts say…• You’re going to lose my business if your web site experience is annoying.• According to preliminary information from the survey… major peeves about commercial Web sites were: [1] – Pop-up advertising (34.9 percent) – Registration log-on pages (16.7 percent) – Software installation (15.7 percent) – Slow-loading pages (9.1 percent) [1] John P. Mello Jr. http://www.ecommercetimes.com/story/44966.html, July, 2005 5/22/2012 UPA Boston Conference 8
  9. 9. In 2007 we added a pop-up…Control Test 5/22/2012 UPA Boston Conference 9
  10. 10. Results of adding the pop-up• Significant increase in conversion rate across most countries and across most products, from 5% to 11%.• More business cards were created Sum of Documents Created 25000 20000 15000 Category Name Music & Entertainment Flowers & Candles Construction & Trade 10000 Beauty/Massage/Spa Music & Entertainment Flowers & Candles Construction & Trade 5000 Beauty/Massage/Spa Most Popular Most Popular 0 Control Experiment 3700 3701 Feature Value Id Tested in August, 2007 5/22/2012 UPA Boston Conference 10
  11. 11. Today: In 2008, Nielsen declares the lightbox as the interaction design technique of the year [1] [1] Jakob Nielsen, 10 Best Application UIs, Alertbox, 8/12/2008 5/22/2012 UPA Boston Conference 11
  12. 12. PROGRESS INDICATORS 5/22/2012 UPA Boston Conference 12
  13. 13. What the experts say…• Show the steps of the process [1] – Recommended for checkout• Guideline: Provide feedback to let users know where they are in the Web site. [2] – Feedback provides users with the information they need to understand where they are within the Web site, and for proceeding to the next activity.• In this example, note the right-pointing arrows. Nielsen discusses his changeover from colons to arrows. The rationale for the arrows instead of colons or other indicators was because ―the arrows seem to be slightly easier to understand as an indication of moving deeper and deeper into the site.‖ [3] [1] Nielsen Norman Group, E-Commerce User Experience, Vol. 4: Shopping Cart, Checkout and Registration, 2011 [2] US Government, Research-Based Web Design & Usability Guidelines, 2006 [3] Beth A. Martin, Breadcrumb Navigation, November 2006 5/22/2012 UPA Boston Conference 13
  14. 14. We tried removing the progress indicator• Removing progress bar from Options improves conversionControl Results: • We observed a significant increase of +0.47% of the conversion rate overall. • The testTest significantly increased the number of sessions that make it to the Cart. 5/22/2012 UPA Boston Conference 14
  15. 15. We tried adding a progress indicator…• Adding a progress bar to the design pages hurts conversion: Test condition sessions CR CR Change Control 50175 7.33% Display Nav 49362 7.06% -3.69% 5/22/2012 UPA Boston Conference 15
  16. 16. Progress Bar visual design…ControlTest Visual redesign hurts conversion. Results: • CR is down 0.9% for New Customers 5/22/2012 UPA Boston Conference 16
  17. 17. HOMEPAGES 5/22/2012 UPA Boston Conference 17
  18. 18. What the experts say…• “Clearly designate one page per site as the official homepage.” [1]• For a decade, one of the primary homepage usability guidelines has been to designate a single page as the one and only official homepage for any given website. Users are confused when several pages are referred to as ―home.‖ [2]• [3] [1] Jakob Nielsen, 113 Design Guidelines for Homepage Usability, 2001 [2] Jakob Nielsen, Overloaded vs. Generic Commands, 12/19/2011 [3] US Government, Research-Based Web Design & Usability Guidelines, 2006 5/22/2012 UPA Boston Conference 18
  19. 19. Two Home pages?• It worked for HomeRuns in 2001:First Time Home Returning Home 5/22/2012 UPA Boston Conference 19
  20. 20. A decade later…First Time Home Returning Home• For returning customers, conversion rate is significantly higher for the Returning Home page than from the First Time Visitor home page. 5/22/2012 UPA Boston Conference 20
  21. 21. BUTTONS 5/22/2012 UPA Boston Conference 21
  22. 22. What the experts say…• ―Ensure that a pushbutton‘s label clearly indicates its action.‖ [1]• ―The text on the button should begin with a verb. [2]• ―Trigger words are the words and phrases that trigger a user into clicking. They contain the essential elements to provide the motivation to continue with the site.‖ [3]• Fitts’s Law: ―The time required to move to a target is a function of target size and distance to the target.‖ [4] [1] US Government, Research-Based Web Design & Usability Guidelines, 2006 [2] UX Booth, Good Call-to-Action Buttons, 2009 [3] Jared Spool, The Right Trigger Words, 2004 [4] Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better Design Decisions, and Teach through Design, 2010 5/22/2012 UPA Boston Conference 22
  23. 23. E-Commerce ExamplesStaplesAmazon 5/22/2012 UPA Boston Conference 23
  24. 24. Cart Buttons • We tested larger target areas with improved visual hierarchy and verbs with security info like Staples. ControlTest 1 Test 2 Significant increase of CTR to Shipping Significant decrease in CTR to Shipping Increased cognitive friction? 5/22/2012 UPA Boston Conference 24
  25. 25. E-Commerce ExamplesAmazon 5/22/2012 UPA Boston Conference 25
  26. 26. Checkout Buttons • We tested distinctive trigger words on our checkout buttons and similar language to Amazon. Control Test 1Could simpler buttons actually decrease CR decreased significantly in FRcognitive load and prevent overthinking? CTR decreased significantly from shipping to confirmation in top 5 locales Shut off after 9 days in testing 5/22/2012 UPA Boston Conference 26
  27. 27. Product Page Buttons• We tested rollover states, more prominent call-to-actions and larger targets.Control Test 1 Test 2 Significant increase in CR of 1.5% was seen. Test 2: No impact. Law of diminishing returns? 5/22/2012 UPA Boston Conference 27
  28. 28. WEBSITE PURCHASE FLOW 5/22/2012 UPA Boston Conference 28
  29. 29. What the experts say…• Offer ―sufficient information about your products to answer users‘ pre- purchase questions.‖ [1]• ―Helping customers see what they want, when they want it, and in the way they want to see it, is substantially preferable to forcing them through a hub-and-spoke, back-and-forth path.‖ [2]• ―The number of clicks isnt what is important to users, but whether or not theyre successful at finding what theyre seeking.‖ [3]And internally we‘ve learned…• When a customer becomes invested in one of our designs, the more likely they‘ll convert.• Multiple screens/clicks are OK if customers know they‘re making progress. [1] NN/g, E-Commerce User Experience [2] Bryan Eisenberg et. al., Call to Action: Secret Formulas to Improve Online Results, 2005 [3] Jared Spool, Testing the Three-Click Rule, 2003 5/22/2012 UPA Boston Conference 29
  30. 30. Design / Flow Get started Choose Design Template 5/22/2012 UPA Boston Conference 30
  31. 31. Design / FlowPreviewTemplate Personalize! Checkout Process 5/22/2012 UPA Boston Conference 31
  32. 32. That seems to work well. But what if…?Significant increases to Website CR fornew customers.• DTI CR increased 9.58%• Acquisition campaigns increased 5.18% Less friction? 5/22/2012 UPA Boston Conference 32
  33. 33. PEOPLE DON’T READ 5/22/2012 UPA Boston Conference 33
  34. 34. Good ol‘ Gary Larson… 5/22/2012 UPA Boston Conference 34
  35. 35. What the experts say…• ―Weve known since our first studies of how users read on the Web that they typically dont read very much.‖ [1]• ―Satisficing describes the situation where people settle with a solution to a problem that is "good enough".‖ [2]• “Users don‘t make optimal choices…Neither do they scan web-page in a linear fashion, going sequentially from one site section to another one.‖ [3] [1] Jakob Nielsen, How Little Do Users Read?, 2008 [2] Interaction-Design.org, Satisficing, 2004 [3] Smashing Magazine, 10 Principles of Effective Web Design, 2008 5/22/2012 UPA Boston Conference 35
  36. 36. Product PageControl Relevant copy is helpful for decision making. Significant increase inTest 2 paper stock upgrades 5/22/2012 UPA Boston Conference 36
  37. 37. More Product Page TestingControl Test Marginal increase in GM/orderSo copy is making a difference… Significant shift in take rate of 100 lb. premium and recycled paper and a decrease in Matte and Glossy. 5/22/2012 UPA Boston Conference 37
  38. 38. Sign In ButtonControl Significantly hurt overall CR by 1.25% Sign in page exit rate increased significantly. Shut off after 7 days of testing Even changing 2 letters can have an impact. Test 5/22/2012 UPA Boston Conference 38
  39. 39. Summary• Best practices are a great starting point but they‘re general.• A/B testing can help you zero-in on specific changes that can help you make changes that benefit your site and your customers.• Context and relevancy matters.• Even small changes can make a significant difference in site performance.• Nothing is straightforward and simple. 5/22/2012 UPA Boston Conference 39
  40. 40. Summary• There‘s a healthy tension between Standardization and Innovation.• This isn‘t the end state. We‘ll continue to optimize and learn from our customers.• It‘s all about balance… and learning more about your customers through various avenues…• What the experts say: It depends… [1] [1] Jared Spool, User Interface Engineering 5/22/2012 UPA Boston Conference 40
  41. 41. Summary• Consistency is one of the most powerful usability principles…• A foolish consistency is the hobgoblin of little minds… - Ralph Waldo Emerson• A/B testing helps us learn when context is more important than consistency. 5/22/2012 UPA Boston Conference 41
  42. 42. QUESTIONS? 5/22/2012 UPA Boston Conference 42
  43. 43. Thanks!Special thanks to Dave Yuknat for his fantastic editingexpertise… and for gathering supporting examples for thepresentation. Susan Rice leads global UX design and Kirk Doggett leads global UX research at Vistaprint. And just a plug… we’re hiring! 5/22/2012 UPA Boston Conference 43