2. FFC Landing Page Optimization Project
Document Information
Title: FFC Landing Page Optimization project
Author: Robert Sherron
Co-Author: removed
Status: Draft In Review Final
Target Start Date: July 15, 2008 approximately
Target End Date: September 16, 2008 approximately
Change History
Version Distribution Date Description of Changes
1.0 5/15/2008 Initial draft document.
2.0 5/30/2008 Updates made as per 5/28 conference call.
Project Team Members
Name Group
removed Research & Audience Analytics
removed Research
removed Business Analytics
removed Technology, CBSSports.com
removed Marketing
Robert Sherron User Experience/Usability
removed Sales Strategy & Audience Analytics
TBD Quality Assurance
Page 2
3. FFC Landing Page Optimization Project
Table of Contents
1. Project Overview .................................................................................................................................. 4
2. Challenge/Opportunity ......................................................................................................................... 4
3. Project Goals and Objectives ................................................................................................................ 5
3.1 FFC Objectives ............................................................................................................................... 5
3.2 IT/Infrastructure Objectives .......................................................................................................... 5
4. Methodology ........................................................................................................................................ 6
4.1 Basic A/B Split Test........................................................................................................................ 6
4.2 Modified A/B Split Test for FFC ..................................................................................................... 6
4.2.1 Testing Benefits ................................................................................................................. 7
4.2.2 Testing Limitations ............................................................................................................ 7
4.3 FFC Landing Page Alternatives ...................................................................................................... 7
4.4 FFC Landing Page Exposure........................................................................................................... 8
5. Primary Metric for Declaring a Winner................................................................................................. 9
5.1 Additional Metrics for Analysis ..................................................................................................... 9
6. Measurement System ........................................................................................................................... 9
7. Sample Size and Traffic Volume............................................................................................................ 9
8. Test Duration ...................................................................................................................................... 10
9. Analysis and Reporting ....................................................................................................................... 10
10. Iterative A/B Split Testing ................................................................................................................... 11
11. Complimentary Testing Strategies...................................................................................................... 11
12. Open Issues and Considerations ......................................................................................................... 11
13. Useful Information and References .................................................................................................... 11
Appendix I: Fantasy Acquisition Landing Page ........................................................................................... 13
Appendix II: FFC Landing Page - Unregistered User or Registered but Not Signed In ............................... 14
Appendix III: FFC Landing Page – Registered User, Signed In .................................................................... 15
Page 3
4. FFC Landing Page Optimization Project
1. Project Overview
The Fantasy Sports industry has become increasingly competitive – the landscape includes
organizations of all sizes – from large media operations (ESPN, YAHOO!, Fox Sports) and the
league itself (NFL) with millions of unique users down to small operations with user counts in
the thousands.
The flagship of CBSSports.com is Fantasy Football Commissioner (FFC) - a league manager
product that allows individuals to organize their league using robust customization capabilities
and features. FFC is the most profitable of all fantasy football product offerings and is the
primary focus of marketing efforts.
FFC is subscription based and must be re-purchased each year. Annual league (customer)
retention surpasses 90%; nonetheless, what can be done to promote new league creation (new
customers)?
The project here-in, will explore landing page optimization and testing strategies for FFC to
better understand and promote customer conversion in 2008.
2. Challenge/Opportunity
As customer acquisition becomes more difficult each year due to outside competition in the
Fantasy arena, growth opportunities have become increasingly important for CBSSports.com.
Despite 90% FFC league retention annually for existing customers, new growth must also be
fostered to sustain a leading industry position. To this end, CBSSports.com recognizes the
importance of landing page design and its impact on attracting new customers and converting
them to subscribers.
In 2003, CBSSports.com (then CBSSportsline) collaborated with Optimost (now owned by
Interwoven) to examine landing page optimization via Mulitvariable testing. As a result of this
effort, modifications were made to landing pages based on feedback and data collected. Since
that time, this type of testing has not been conducted on Fantasy product pages.
After some open dialog between the Fantasy FFC, Research, and Usability groups, we believe
there is an opportunity to modify and improve the current FFC landing page and grow the
customer base. Given the effort and expense to work with an outside vendor at this time, an in-
house (CBS Interactive) testing solution is being formulated to conduct landing page
optimization testing. The challenge will be to understand the conversion elements for the
current FFC landing page and create alternative designs to discover which design elements are
the keys to customer conversion: What page elements (e.g., copy text, graphics, buttons, etc.),
and their placement on a landing page promote this process?
Page 4
5. FFC Landing Page Optimization Project
It should also be noted that this in-house testing solution is in its infancy and will therefore be
an effort of trial and error, discovery, and ultimately success!
3. Project Goals and Objectives
The goal of this project will be to formulate and implement a testing strategy to discover design
elements that attract new customers and promote conversion - signup for the FFC product.
Ultimately we want to provide an optimized landing page.
It will also serve as a proving ground for an in-house landing page optimization process which
currently is not in practice within CBS Interactive.
3.1 FFC Objectives
The goals specific to the FFC product are as follows.
a. Determine the landing page variables to manipulate and then create alternative FFC
landing pages (colors, graphics, copy, element placement, etc.).
b. Determine metrics to capture on each landing page (e.g., time spent on a landing page
and the number of new league creations for each landing page alternative).
c. Determine how to aggregate and report metrics.
d. Test, analyze, and determine which landing page designs (and the specific elements on
the page) promote increased conversion rates.
e. Quantify landing page optimization results (e.g., [n%] conversion rate)
3.2 IT/Infrastructure Objectives
The goals specific to IT/Infrastructure are as follows.
a. Implement A/B Split Testing infrastructure and methodology
b. Determine how to measure each metric
Page 5
6. FFC Landing Page Optimization Project
4. Methodology
The basic methodology for the FFC Landing Page Optimization project is to identify design
elements that we believe can be modified to promote increased conversion (new league
creation). These design elements are then incorporated into alternative landing pages. The
current landing page (baseline) and the alternative landing pages are then shown to web site
visitors (live) such that each landing page is exposed fairly and uniformly for a specified
duration. The conversion rates for each landing page are calculated and examined to determine
the winner – that is…the landing page design that promotes the best conversion rate (FFC
league creations).
Figure 4-1: Generic Conversion Illustration
4.1 Basic A/B Split Test
The simplest A/B Split test involves creating two designs (A and B) of a landing page. Version A
is the control – usually the existing landing page. Version B incorporates a single change so that
the effect of that change can be tracked and translated into a conversion rate. Landing page
traffic is diverted (split) to each version equally over the same time period for a specified testing
duration. At the end of the test duration, the conversion rates for A and B are compared to one
another and evaluated to determine which version performed best.
4.2 Modified A/B Split Test for FFC
The specific method planned for the FFC Landing Page Optimization project will be a modified
A/B Split Test. This method incorporates more than one design alternative (A/B/C/D), “A” being
the control and compares conversion rates between the landing pages.
Page 6
7. FFC Landing Page Optimization Project
As currently conceived, a few design changes will be introduced on the landing pages and their
impact on conversion will be examined. Customer exposure to designs that are deemed risky
can be diverted to a small percentage of site visitors, monitored, and removed if performing
poorly in order to limit negative brand impact.
Note: Some of the literature on A/B testing offers conflicting advice. Some experts
vigorously suggest that only one small design change be introduced and compared
to a baseline page while others recommend radical and multiple design departures
from the baseline page. The trade off is the precision to which you can infer the
benefits of a specific design change – with one change you can definitively measure
its impact on conversion whereas with multiple radical changes you are less certain
of which design element(s) accounts for the conversion impact. On the other hand,
radical changes could expose a design alternative that converts quite well but would
otherwise not be tested in a conservative testing scheme.
4.2.1 Testing Benefits
a. Measures actual behavior
b. Measures small performance differences with a high confidence interval given large
sample sizes
c. Prescriptive tool to resolve competing design choices
4.2.2 Testing Limitations
a. Reasons for why one design performed better than another may not be discernable
b. When users decided not to convert (signup) we don’t know why – this knowledge can
be just as meaningful for design improvements
4.3 FFC Landing Page Alternatives
Further discussion is warranted to brainstorm on all of the design elements that could be
manipulated on the FFC landing page. The following is a preliminary list of potential design
elements that could be manipulated:
a. Headline color and copy
b. Call to action buttons , links, and graphics
c. Graphics
d. Color (images, background, etc.)
e. Configuration/Placement of page elements (e.g., login field, button, promotion, etc.)
f. Header vs No Header
g. Variations in product description, page copy, etc.
Page 7
8. FFC Landing Page Optimization Project
4.4 FFC Landing Page Exposure
In theory, there is no set rule for setting the traffic ratio diverted to landing pages in order to
test conversion. For our purposes, 10% of the traffic will be sent to the Baseline group which
will see the current FFC landing page design. The remaining 90% of the traffic is then equally
and randomly diverted to a Test group consisting of three alternative landing pages and the
current FFC landing page. Each page in the Test group receives approximately 22.5% of the
traffic from this slice. The current landing page is repeated in both groups to confirm the
baseline conversion rate. The following illustration outlines the proposed landing page traffic
diversion for landing page designs.
Divert 10% of Current FCC 10%
Baseline Group traffic to Landing Page A
Test Group Current FCC ≈22.5
Landing Page A
%
Alternative ≈22.5
Landing Page B
Divert 90% of %
traffic equally and
randomly to
Alternative
≈22.5
Landing Page C
%
Alternative ≈22.5
Landing Page D
%
Figure 4-2: Traffic Ratio for FFC Landing Pages
A conservative approach would be to divert traffic in a 90/10 ratio (90% Baseline/10% Test).
Depending on how radical or risky the landing page changes are, you can control exposure to
limit negative brand impact. Given that the project team is confident that the design
alternatives will not have a negative brand impact, the traffic ratio has been set to 10/90. This
ratio will expose each design to a high traffic volume and help to ensure a proper statistical
sample size. Higher sample sizes will promote confidence in the results.
Once a visitor is diverted to a specific landing page, that visitor should see the same landing
page on all subsequent visits. Common practice is to use a cookie to track the computer used to
access the page.
Page 8
9. FFC Landing Page Optimization Project
5. Primary Metric for Declaring a Winner
The primary metric proposed to declare a winner will be the landing page design with highest
conversion rate (%) for new FFC league creations.
# of New League Creations - Paid
Conversion Rate % = x 100
# of Landing Page Views (A/B/C/D)
5.1 Additional Metrics for Analysis
The following additional metrics should be collected and analyzed to gain insight into landing
page design elements and their influence on the conversion process:
a. Unique visitors – records conversion rate
b. Time of day conversions – examine peak conversion time or pattern
c. Day of week conversions – examine peak conversion time or pattern
d. Length of time spent (seconds) on each landing page – is there something about the
page that is confusing or not optimally written or is the page engaging?
6. Measurement System
Omniture will be used to collect the page view and unique visitor data. Business Objects will be
used to collect the data for paid leagues created for each design.
7. Sample Size and Traffic Volume
The sample size needed for significance for this project will be determined by the statistic used
in the analysis. Given the high volume of traffic expected during the test duration, we should
have a large enough sample size to validate the results and determine if there are significant
differences.
The following website may be helpful: Sample Size Calculator
For thought, here is a quote from Bryan Eisenberg of FutureNow:
“… you need to decide your sample size and set up the criteria for success.
To decide your ultimate sample size, run a "null" test with your A/B test.
The null test is really just an A/A test, where you are running the control
against itself to determine where the convergence of results matches up
(typically within 0.05 percent of each other, but that's up to you). When
the tests converge, you'll have an idea of the volume of traffic you need to
test. You decide your criteria for success based on a certain number of
conversions or sales, or you can measure results over course of time (two
Page 9
10. FFC Landing Page Optimization Project
weeks). To be sure your test is showing a statistically meaningful impact
on the variables, you have to know if you've demonstrated enough of a
difference (delta) between the tests to declare a clear winner. As a rule of
thumb, you should have at least a three times larger result (e.g., if A is 5, B
should be 15).”
Here is another website that may be helpful: Split Test Calculator and Decision Tool
The following figures show traffic volume for the 2007 season:
Figure 7-1: 2007 Page Views for FFC
Month Average Daily Page Views
June 2,163
2007
July 2,679
August 6,850
Figure 7-2: 2007 Page Views for FFC
8. Test Duration
As planned, the testing effort will start on or about July 15th, 2008 and end on September 12th,
2008.
9. Analysis and Reporting
TBD.
Page 10
11. FFC Landing Page Optimization Project
10. Iterative A/B Split Testing
Given the methodology outlined here, an iterative testing strategy could yield a better
understanding of landing page optimization than one single test. Nonetheless, for this project a
single test is planned. If a true winner is declared at the end of testing, discussion will be given
to tweaking that design with iterative, small changes to refine it in the future.
11. Complimentary Testing Strategies
As a compliment to the knowledge gained in this A/B Split test, other user testing methods can
help quantify and qualify user behavior on the FFC landing page. The following additional user
testing methods are being considered:
a. Eye Tracking: The Interactive lab in Las Vegas, NV has a Tobii Eye Tracker that could be
used to collect data on what people see on each landing page and where they are
looking on the page (and where they are not).
b. Usability Test: Real-world users can be recruited and asked to find, navigate to, and
interact with a landing page for the FFC product. Throughout the test the participant is
asked to provide qualitative feedback.
12. Open Issues and Considerations
1. Request Q/A person(s).
2. Determine which variables will be manipulated on alternative landing pages and when
that Creative will be available (add to document when ready).
3. Confirm that all required data can be collected and transformed into a data file that can
be imported into SPSS (Statistical Package for the Social Sciences).
4. Determine the sample size and confidence intervals for this test.
13. Useful Information and References
Alt, B., Usborne, N., & McGlaughlin, F. (2005 , August 15). A/B Split Testing. Retrieved from Marketing
Experiments Journal: http://www.marketingexperiments.com/improving-website-conversion/ab-split-
testing.html
Buerba, S., & Chiu, L. (2008, March 6). Let Your Customers Decide: 5 Principles of Using A/B and
Multivariate Testing. Retrieved from omniture.com:
http://www.omniture.com/offer/164?fc=true&v_id=337856
Page 11
12. FFC Landing Page Optimization Project
Campbell, J. (2008, January 21). With A/B Testing - You Don't Think, You Know. Retrieved from
FindResolution: http://www.findresolution.com/2008/01/with-ab-testing-you-dont-think-you-
know.html
Eisenberg, B. (2004, May 7). A/B Testing for the Mathematically Disinclined. Retrieved from The ClickZ
Network: http://www.clickz.com/showPage.html?page=3349901
Eisenberg, B. (2005, April 29). How to Improve A/B Testing. Retrieved from The ClickZ Network:
http://www.clickz.com/showPage.html?page=3500811
Eisenberg, B. (2007, February 16). We Tried That Already. Retrieved from The ClickZ Network:
http://www.clickz.com/showPage.html?page=3624994
Fortin, M. (Unknown). Splitting Hairs Over Split-Testing? Retrieved from michelfortin.com:
http://www.michelfortin.com/splitting-hairs-over-split-testing/
Fortin, M. (Unkown). Taguchi Explained… For Dummies. Retrieved from michelfortin.com:
http://www.michelfortin.com/taguchi-explained-for-dummies/
Meyers, P. J. (2007). Measuring Usability: The Basics. Retrieved from usereffect:
http://www.usereffect.com/measuring-usability-basics.pdf
Nielsen, J. (2005, August 15). Putting A/B Testing in Its Place. Retrieved from useit.com:
http://www.useit.com/alertbox/20050815.html
Quarto-vonTivada, J. (2006). AB Testing Too Little, Too Soon? Retrieved from Future Now:
http://www.futurenowinc.com/abtesting.pdf
Unknown. (2008, April 15). Improving Conversions by Split Testing. Retrieved from Internet Marketing
University: http://www.internetmarketinguniversity.com/blog/?p=168
Wilson, R. F. (2008). How to Develop a Landing Page. Retrieved from
http://www.wilsonweb.com/ebooks/landing.htm
Page 12
13. FFC Landing Page Optimization Project
Appendix I: Fantasy Acquisition Landing Page
Fantasy Acquisition Landing Page
Page 13
14. FFC Landing Page Optimization Project
Appendix II: FFC Landing Page - Unregistered User or Registered but Not Signed In
Page 14
15. FFC Landing Page Optimization Project
Appendix III: FFC Landing Page – Registered User, Signed In
Page 15
