1.
#EpicFail!
A/B Test Results vs. UX Best Practices
Susan Rice & Kirk Doggett
Global UX Center of Excellence
5/22/2012 UPA Boston Conference 1

2.
Overview
• Introduction
• Examples
– Pop Up Windows
– Progress Indicators
– Homepages
– Buttons
– Website Purchase Flow
– People Don‘t Read
• Summary
• Questions?
5/22/2012 UPA Boston Conference 2

3.
Who We Are
Vistaprint provides high-impact personalized
products and electronic services that empower
micro businesses and consumers at an affordable
price.
Company Facts
• 25 localized websites
• 3,700 employees
• 3 state of art manufacturing facilities
• 13 offices worldwide
5/22/2012 UPA Boston Conference 3

4.
―Test Before You Invest‖
We have dozens of multivariate
―split run‖ site tests live at any User
Population
given time during our 3 week
release cycles.
• Scalable:
Random
Most site changes are measured Sample
before rollout
• Low Risk:
Price Price
Multiple tests can be run Control Test
simultaneously with a sample
UI
of the population Control 5% 5%
• Accurate: UI
Test sample is representative Test 5% 5%
of user base
5/22/2012 UPA Boston Conference 4

5.
UX & A/B Testing
• UX designs the UI and
helps inform the test plan.
• Test runs and is analyzed.
• Layer the quantitative with
the qualitative to create a
holistic picture to inform
future design decisions.
But what happens when A/B testing results challenge tried
and true usability principles and best practices?
5/22/2012 UPA Boston Conference 5

6.
POP UP WINDOWS
5/22/2012 UPA Boston Conference 6

7.
What the experts say…
• Avoid popup windows: [1]
– Extra windows keep users from getting to your site content, and even if such
windows contain valuable information, users are likely to dismiss them
immediately as ads.
– The other drawback to popup windows is that they go away — once the user
gets rid of them, they're gone, so users often can't find the information again
even if they want to.
• Do not have unsolicited windows or graphics ―pop up‖ to users.[2]
– Users have commented that unsolicited windows or graphics that ‗pop up‘ are
annoying and distracting when they are focusing on completing their original
activity.
• Selective attention is very powerful, and Web users have learned to stop
paying attention to any ads that get in the way of their goal-driven
navigation. [3]
– “pop-up purges mean that users close pop-up windoids before they have even
fully rendered; sometimes with great viciousness‖
[1] Jakob Nielsen, excerpt from 113 Design Guidelines for Homepage Usability, 2001
[2] US Government, Research-Based Web Design & Usability Guidelines, 2006
[3] Jakob Nielsen, Top 10 Mistakes in Web Design, updated 2011
5/22/2012 UPA Boston Conference 7

8.
What the experts say…
• You’re going to lose my business if your web site
experience is annoying.
• According to preliminary information from the survey…
major peeves about commercial Web sites were: [1]
– Pop-up advertising (34.9 percent)
– Registration log-on pages (16.7 percent)
– Software installation (15.7 percent)
– Slow-loading pages (9.1 percent)
[1] John P. Mello Jr. http://www.ecommercetimes.com/story/44966.html, July, 2005
5/22/2012 UPA Boston Conference 8

9.
In 2007 we added a pop-up…
Control Test
5/22/2012 UPA Boston Conference 9

10.
Results of adding the pop-up
• Significant increase in conversion rate across
most countries and across most products, from
5% to 11%.
• More business cards were created
Sum of Documents Created
25000
20000
15000
Category Name
Music & Entertainment
Flowers & Candles
Construction & Trade
10000 Beauty/Massage/Spa
Music & Entertainment
Flowers & Candles
Construction & Trade
5000 Beauty/Massage/Spa
Most Popular
Most Popular
0
Control Experiment
3700 3701
Feature Value Id
Tested in August, 2007
5/22/2012 UPA Boston Conference 10

11.
Today:
In 2008, Nielsen declares the lightbox as the
interaction design technique of the year [1]
[1] Jakob Nielsen, 10 Best Application UIs, Alertbox, 8/12/2008
5/22/2012 UPA Boston Conference 11

12.
PROGRESS INDICATORS
5/22/2012 UPA Boston Conference 12

13.
What the experts say…
• Show the steps of the process [1]
– Recommended for checkout
• Guideline: Provide feedback to let users know where they are in the
Web site. [2]
– Feedback provides users with the information they need to understand
where they are within the Web site, and for proceeding to the next activity.
• In this example, note the right-pointing arrows. Nielsen discusses
his changeover from colons to arrows. The rationale for the arrows
instead of colons or other indicators was because ―the arrows seem
to be slightly easier to understand as an indication of moving deeper
and deeper into the site.‖ [3]
[1] Nielsen Norman Group, E-Commerce User Experience, Vol. 4: Shopping Cart, Checkout and Registration, 2011
[2] US Government, Research-Based Web Design & Usability Guidelines, 2006
[3] Beth A. Martin, Breadcrumb Navigation, November 2006
5/22/2012 UPA Boston Conference 13

14.
We tried removing the progress indicator
• Removing progress bar from Options improves
conversion
Control
Results:
• We observed a
significant increase
of +0.47% of the
conversion rate
overall.
• The test
Test significantly
increased the
number of sessions
that make it to the
Cart.
5/22/2012 UPA Boston Conference 14

15.
We tried adding a progress indicator…
• Adding a progress bar to the design pages hurts
conversion:
Test condition sessions CR CR Change
Control 50175 7.33%
Display Nav 49362 7.06% -3.69%
5/22/2012 UPA Boston Conference 15

16.
Progress Bar visual design…
Control
Test
Visual redesign hurts
conversion.
Results:
• CR is down 0.9% for
New Customers
5/22/2012 UPA Boston Conference 16

17.
HOMEPAGES
5/22/2012 UPA Boston Conference 17

18.
What the experts say…
• “Clearly designate one page per site as the official
homepage.” [1]
• For a decade, one of the primary homepage usability
guidelines has been to designate a single page as the
one and only official homepage for any given website.
Users are confused when several pages are referred to
as ―home.‖ [2]
• [3]
[1] Jakob Nielsen, 113 Design Guidelines for Homepage Usability, 2001
[2] Jakob Nielsen, Overloaded vs. Generic Commands, 12/19/2011
[3] US Government, Research-Based Web Design & Usability Guidelines, 2006
5/22/2012 UPA Boston Conference 18

19.
Two Home pages?
• It worked for HomeRuns in 2001:
First Time Home Returning Home
5/22/2012 UPA Boston Conference 19

20.
A decade later…
First Time Home Returning Home
• For returning customers, conversion rate is significantly higher for the
Returning Home page than from the First Time Visitor home page.
5/22/2012 UPA Boston Conference 20

21.
BUTTONS
5/22/2012 UPA Boston Conference 21

22.
What the experts say…
• ―Ensure that a pushbutton‘s label clearly indicates its
action.‖ [1]
• ―The text on the button should begin with a verb. [2]
• ―Trigger words are the words and phrases that trigger a
user into clicking. They contain the essential elements
to provide the motivation to continue with the site.‖ [3]
• Fitts’s Law: ―The time required to move to a target is a
function of target size and distance to the target.‖ [4]
[1] US Government, Research-Based Web Design & Usability Guidelines, 2006
[2] UX Booth, Good Call-to-Action Buttons, 2009
[3] Jared Spool, The Right Trigger Words, 2004
[4] Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence
Perception, Increase Appeal, Make Better Design Decisions, and Teach through Design, 2010
5/22/2012 UPA Boston Conference 22

23.
E-Commerce Examples
Staples
Amazon
5/22/2012 UPA Boston Conference 23

24.
Cart Buttons
• We tested larger target areas with improved visual
hierarchy and verbs with security info like Staples.
Control
Test 1 Test 2
Significant increase of CTR to Shipping Significant decrease in CTR to Shipping
Increased cognitive friction?
5/22/2012 UPA Boston Conference 24

25.
E-Commerce Examples
Amazon
5/22/2012 UPA Boston Conference 25

26.
Checkout Buttons
• We tested distinctive trigger words on our checkout
buttons and similar language to Amazon.
Control Test 1
Could simpler buttons actually decrease CR decreased significantly in FR
cognitive load and prevent overthinking? CTR decreased significantly from
shipping to confirmation in top 5 locales
Shut off after 9 days in testing
5/22/2012 UPA Boston Conference 26

27.
Product Page Buttons
• We tested rollover states, more prominent call-to-actions
and larger targets.
Control Test 1
Test 2 Significant increase in
CR of 1.5% was seen.
Test 2: No impact.
Law of diminishing
returns?
5/22/2012 UPA Boston Conference 27

28.
WEBSITE PURCHASE FLOW
5/22/2012 UPA Boston Conference 28

29.
What the experts say…
• Offer ―sufficient information about your products to answer users‘ pre-
purchase questions.‖ [1]
• ―Helping customers see what they want, when they want it, and in the
way they want to see it, is substantially preferable to forcing them
through a hub-and-spoke, back-and-forth path.‖ [2]
• ―The number of clicks isn't what is important to users, but whether or not
they're successful at finding what they're seeking.‖ [3]
And internally we‘ve learned…
• When a customer becomes invested in one of our designs, the more
likely they‘ll convert.
• Multiple screens/clicks are OK if customers know they‘re making
progress.
[1] NN/g, E-Commerce User Experience
[2] Bryan Eisenberg et. al., Call to Action: Secret Formulas to Improve Online Results, 2005
[3] Jared Spool, Testing the Three-Click Rule, 2003
5/22/2012 UPA Boston Conference 29

30.
Design / Flow
Get started Choose Design Template
5/22/2012 UPA Boston Conference 30

31.
Design / Flow
Preview
Template
Personalize! Checkout Process
5/22/2012 UPA Boston Conference 31

32.
That seems to work well. But what if…?
Significant increases to Website CR for
new customers.
• DTI CR increased 9.58%
• Acquisition campaigns increased 5.18% Less friction?
5/22/2012 UPA Boston Conference 32

33.
PEOPLE DON’T READ
5/22/2012 UPA Boston Conference 33

34.
Good ol‘ Gary Larson…
5/22/2012 UPA Boston Conference 34

35.
What the experts say…
• ―We've known since our first studies of how users read on the
Web that they typically don't read very much.‖ [1]
• ―Satisficing describes the situation where people settle with a
solution to a problem that is "good enough".‖ [2]
• “Users don‘t make optimal choices…Neither do they scan
web-page in a linear fashion, going sequentially from one site
section to another one.‖ [3]
[1] Jakob Nielsen, How Little Do Users Read?, 2008
[2] Interaction-Design.org, Satisficing, 2004
[3] Smashing Magazine, 10 Principles of Effective Web Design, 2008
5/22/2012 UPA Boston Conference 35

36.
Product Page
Control
Relevant copy is
helpful for decision
making.
Significant increase in
Test 2
paper stock upgrades
5/22/2012 UPA Boston Conference 36

37.
More Product Page Testing
Control Test
Marginal increase in GM/order
So copy is making a difference… Significant shift in take rate of
100 lb. premium and recycled paper
and a decrease in Matte and Glossy.
5/22/2012 UPA Boston Conference 37

38.
Sign In Button
Control
Significantly hurt
overall CR by 1.25%
Sign in page exit rate
increased significantly.
Shut off after 7 days of
testing
Even changing 2
letters can have an
impact.
Test
5/22/2012 UPA Boston Conference 38

39.
Summary
• Best practices are a great starting point but they‘re
general.
• A/B testing can help you zero-in on specific changes
that can help you make changes that benefit your
site and your customers.
• Context and relevancy matters.
• Even small changes can make a significant
difference in site performance.
• Nothing is straightforward and simple.
5/22/2012 UPA Boston Conference 39

40.
Summary
• There‘s a healthy tension between Standardization
and Innovation.
• This isn‘t the end state. We‘ll continue to optimize
and learn from our customers.
• It‘s all about balance… and learning more about
your customers through various avenues…
• What the experts say: It depends… [1]
[1] Jared Spool, User Interface Engineering
5/22/2012 UPA Boston Conference 40

41.
Summary
• Consistency is one of the most powerful usability
principles…
• A foolish consistency is the hobgoblin of little
minds…
- Ralph Waldo Emerson
• A/B testing helps us learn when context is more
important than consistency.
5/22/2012 UPA Boston Conference 41

42.
QUESTIONS?
5/22/2012 UPA Boston Conference 42

43.
Thanks!
Special thanks to Dave Yuknat for his fantastic editing
expertise… and for gathering supporting examples for the
presentation.
Susan Rice leads global UX design and
Kirk Doggett leads global UX research at Vistaprint.
And just a plug… we’re hiring!
5/22/2012 UPA Boston Conference 43