Increasing conversions multivariate testing to maximize efficiency of lead generation campaigns Boost Website Performance with Statistical Design of Experiment Web page elements Header Headline Sub-headline Body text Bullet list, charts, graphs, videos, pictures, callouts, or similar Promotion Registration Form, call-to-action Navigation Site search Links on the page Footer User reviews We identified elements and the level of attention We still don’t know which ones have bigger impact on conversion What happens if we change one or more of them? How do we optimize these for boosting performance? Solution: Statistical Controlled Experiment where variations are present Determine the best performing combination of variations Run a validation test on the recommendation to confirm Controlled Experiment in 1747 in England Testing cure for scurvy by James Lind, a surgeon Fisher started with application to agriculture – 1918-1940 Factorial Design and ANOVA Industrial Era: 1950-1970 Application in Chemical and process industries Second Industrial Era: 1970-1990 Wider application in Quality Control to most industries How do we apply this to boost website performance? Select elements that are important or attract more attention Elements should be as much independent as possible Variations are created for each element 2/3 apart from the existing (CONTROL) Develop web pages containing combination of variations ALL possible variations or a SMALLER subset These web pages are run along with Control Traffic volume for creating statistically significant result is directed to these pages Performance of each web page is measured Statistical analysis yields which combination of variation is expected to yield the highest results Develop a final page with recommended combination Run this in parallel with CONTROL to validate results Factors in choosing elements, variations and design Independence, traffic volume, traffic sources, time, cost Experiment Design: Full Factorial (all possible combination) or Partial (selected) Example: A Full Factorial Design with 3 elements with 3 levels of variations each will need 3^3 = 27 web pages to be tested Traditional A/B Testing will also require 27 pages Partial Factorial Design (say in Taguchi Design) will need a L9 array with 9 web pages only Run duration should be enough to Generate statistically significant results Cover cyclical and important variations in environment

1. WELCOME
Boost Website Performance
with
Statistical Design of Experiment
1

2. Website Optimization
 Develop designs based on experience
 Dated? Slow!
 Inputs from focus groups
 Limited and expensive
 Optimization based on web analytics
 Big step forward
 Not granular enough-still making assumptions?
 What next?
 What are elements that go on my web page
 How do they interact/affect user behavior
 Do these elements have the same impact on users
If not which ones lead the pack
 What happens if we change one or more of these?
Confidential | The next-generation online customer acquisition engine

3. Example of elements
 Web page elements
 Header
 Headline
 Sub-headline
 Body text
 Bullet list, charts, graphs, videos, pictures, callouts, or similar
 Promotion
 Registration Form, call-to-action
 Navigation
 Site search
 Links on the page
 Footer
 User reviews
Confidential | The next-generation online customer acquisition engine

4. Web pages are collection of elements: that work together!
Location of
search bar
Left side Model selector
navigation
Display
pricing
Add to cart
Display shipping
and price
Image controls & guarantee
video
© 2010 Confiden1tial | think innovation

5. Use Heat Maps
 To identify elements that are getting visitors attention on web pages

6. In Page Analytics: More quantitative
 Identifies elements AND level of attention from visitors

7. Web page elements: What Next?
 We identified elements and the level of attention
 We still don’t know which ones have bigger impact on conversion
 What happens if we change one or more of them?
 How do we optimize these for boosting performance?
Solution:
 Statistical Controlled Experiment where variations are present
 Determine the best performing combination of variations
 Run a validation test on the recommendation to confirm
Confidential | The next-generation online customer acquisition engine

8. Statistical Controlled Experiment: Reaching back in time
 Controlled Experiment in 1747 in England
 Testing cure for scurvy by James Lind, a surgeon
 Fisher started with application to agriculture – 1918-1940
 Factorial Design and ANOVA
 Industrial Era: 1950-1970
 Application in Chemical and process industries
 Second Industrial Era: 1970-1990
 Wider application in Quality Control to most industries
 How do we apply this to boost website performance?
Confidential | The next-generation online customer acquisition engine

9. High Level Design of Experiment (DOE) Process
 Select elements that are important or attract more attention
 Elements should be as much independent as possible
 Variations are created for each element
 2/3 apart from the existing (CONTROL)
 Develop web pages containing combination of variations
 ALL possible variations or a SMALLER subset
 These web pages are run along with Control
 Traffic volume for creating statistically significant result is directed to these
pages
 Performance of each web page is measured
 Statistical analysis yields which combination of variation is expected to yield
the highest results
 Develop a final page with recommended combination
 Run this in parallel with CONTROL to validate results
Confidential | The next-generation online customer acquisition engine

10. Selecting the right Design
 Factors in choosing elements, variations and design
 Independence, traffic volume, traffic sources, time, cost
 Experiment Design: Full Factorial (all possible combination) or Partial
(selected)
Example:
A Full Factorial Design with 3 elements with 3 levels of variations each will need 3^3
= 27 web pages to be tested
Traditional A/B Testing will also require 27 pages
Partial Factorial Design (say in Taguchi Design) will need a L9 array with 9 web
pages only
 Run duration should be enough to
 Generate statistically significant results
 Cover cyclical and important variations in environment
Confidential | The next-generation online customer acquisition engine

11. Case Study of an E-Commerce portal
 Objectives
 To increase the Shopping Cart creation Rate
 Google Analytics data revealed that the Product Details page
were dropping visitors
 Planned DOE on the Product Details Page
 Selected 7 elements with 2 variations each
 Partial Factorial Design lead to L8 array (8 pages to test)
 Combinations were run sequentially
 Each combination ran for 6 days taking 10%-15% of traffic
 The page ran in parallel with CONTROL

12. Possible List of variables identified
1. Add to Cart placement
2. Displaying price information
3. Members star ratings
4. Image and video controls
5. Related Products
6. Left hand navigation
7. Member Actions – Email to friends, Add to wish list, Price Alerts
8. Product search bar
We finally selected the first 7 variables with one variation each
Partial Factorial design based on Taguchi required 8 pages
© 2010 Confiden1tial | think innovation

13. Example of Page Element Layout
Link to Slide 6
TV1
Link to Slide 10
TV4
TV6
Link to Slide 6 Link to Slide 9
TV2
13

14. Experiments Run
Experiment
Run ID Variable Combinations
Name
Experiment-1 Test Run-2 C-TV1,C-TV2,C-TV3,C+1-TV4,C+1-TV5,C+1-TV6,C+1-TV7
Experiment-2 Test Run-3 C-TV1,C+1-TV2,C+1-TV3,C-TV4,C-TV5,C+1-TV6,C+1-TV7
Experiment-3 Test Run-4 C-TV1,C+1-TV2,C+1-TV3,C+1-TV4,C+1-TV5,C-TV6,C-TV7
Experiment-4 Test Run-5 C+1-TV1,C-TV2,C+1-TV3,C-TV4,C+1-TV5,C-TV6,C+1-TV7
Experiment-5 Test Run-6 C+1-TV1,C-TV2,C+1-TV3,C+1-TV4,C-TV5,C+1-TV6,C-TV7
Experiment-6 Test Run-7 C+1-TV1,C+1-TV2,C-TV3,C-TV4,C+1-TV5,C+1-TV6,C-TV7
Experiment-7 Test Run-8 C+1-TV1,C+1-TV2,C-TV3,C+1-TV4,C-TV5,C-TV6,C+1-TV7
Taguchi
New Test C+1-TV1,C-TV2,C+1-TV3,C+1-TV4,C+1-TV5,C+1-TV6,C+1-TV7
Recommendation
14

15. Experiment-5: C+1-TV1,C-TV2,C+1-TV3,C+1-TV4,C-TV5,C+1-TV6,C-TV7
Affected Variables
TV3: Members star ratings on
the Product details page
TV1(position change):
- Add to Cart link
- Price information display
- Free shipping and lowest
price guarantee
TV4: View image and video link
TV6: Left side navigation links

16. Experiment-4: C+1-TV1,C-TV2,C+1-TV3,C-TV4,C+1-TV5,C-TV6,C+1-TV7
Affected Variables
TV3: Members star ratings on
the Product details page
TV1(position change):
- Add to Cart link
- Price information display
- Free shipping and lowest
price guarantee
TV5: Related products – display
logic
TV7: Member control Links

17. 1. Significance Test Analysis
The Significant Test shows that Experiment 5 and Experiment 4 are ahead of the others
EXP
Exp Number Mean. EXP St. Dev. Control Control Z- P-value
# of Runs SC Cr. SC Cr. Mean SC Cr. St. Dev. score (accept H0) Rank
1 6 5.06% 0.69% 4.60% 0.80% 1.41 0.0700 4
2 6 4.83% 0.17% 4.68% 0.61% 0.60 0.5600 7
3 6 4.89% 0.42% 4.70% 0.36% 1.29 0.2000 6
4 6 5.27% 0.84% 4.67% 0.24% 6.12 0.0002 2
5 6 5.54% 0.98% 4.67% 0.38% 5.61 0.0002 1
6 6 4.90% 0.22% 4.48% 0.44% 2.34 0.0200 3
7 6 4.79% 0.65% 4.49% 0.45% 1.63 0.1000 5
A higher z-score
means that the data is
farther away from the
population mean
(CONTROL)
17

18. 2. Taguchi Design Analysis
The Taguchi design analyzed the variables, and for each variable it
compared Control(1) vs. its Variant(2) and calculated Signal to Noise
ratio for each experiment
Mean – The mean gives the average SC creation rate
SN Ratio – Signal-to-Noise ratio is a measure of how predictable is the SC creation rate. A high SN ratio
means that the Standard Deviation for that given EXP is proportionally smaller than the mean. A lower
SN ratio means that data for the SC creation rate is relatively unpredictable.
Experiment 5 and Experiment 4 have the highest and the second highest SN ratio,
meaning that their SC creation rate is higher, robust and more predictable.
18

19. 2. Taguchi Design Analysis
Impact of elements & variations: Choice 1 (Control) vs. Choice 2
(Variant) for each of the 7 variables
Note that changes in variable 1 pushed the shopping cart creation rate the most while
changes in the second variable caused the rate to drop
19

20. 2. Recommendations Based on Taguchi Design Analysis
Taguchi recommended variable combination for the most performant
Shopping Cart creation rate as follows
Variable Recommendation
TV_1 2
TV_2 1
TV_3 2
TV_4 2
TV_5 2
TV_6 2
TV_7 2
NOTE: Taguchi however will not predict how much better than CONTROL the above hypothesized
template will perform. We recommend creating a design based on this recommendation and
running it for validation.
20

21. Results
 SC Creation Rate moved from 4.48% to 5.02% after validation
 Number of carts created increased from 500 to 560 per day
 Average Cart value moved from $155 to $169
 Revenue increased by 170K a month

22. Things to keep in mind
 Variables to be kept as independent from each other as possible
 Avoid confounding
 Keep environmental variables as much constant as possible
 Avoid introducing new traffic sources
 Select more variables if you have traffic
 Keep the number of variation of each limited to 2-3
 Traffic should be randomly selected
 Load balancer should not introduce any bias
 Should not mix traffic between different variations and CONTROL

23. Regalix Online Marketing
Why partner with Regalix
Industry Best Practices & Framework
Regalix has developed social media framework and best “The ComplianceOnline case study offers insight into how technology
practices leveraging experience from Fortune 1,000 as well marketers can make a vision for a viable online community — that
as venture backed customers across multiple industry embraces customers, partners, and prospects alike — become real.”
verticals including Hi-tech., Retail, Manufacturing, Education, – Laura Ramos, B2B Analyst, Forrester Research
Clean Tech. and Financial Services.
Full-service Execution
“Regalix has social media
Regalix provides strategy, program management, creative,
marketing experience that has
technology, and operations services. This reduces the need
for engaging and managing multiple service providers. been leveraged on behalf of blue- “Citibank views Regalix as a
chip Fortune 100 firms. For this trusted strategic partner. They
reason, I think that you’ll get far have consistently delivered
Superior Client-service more bang for your buck out of innovative solutions of
Regalix’s client services team consists of industry experts and leveraging them as your turn-key exceptional quality over the last
practitioners that have created significant and successful marketing team than you would couple of years of our
marketing programs. out of a single FTE. engagement.
Their reporting dashboards are Regalix has proven experience in
also first-rate. Super stream lined supporting an enterprise of our
– super time efficient.” size and diversity, while
respecting the stringent quality
- Alyssa Rapp, CEO Bottlenotes standards that we have set.”
23

24. Regalix Online Marketing
About Us
Full-Service: Digital Marketing & Technology Services
Company; Strategy, Creative, Campaigns, Technology,
Communities Collateral &
Thought
Talent: Leadership, Advisory, 175+ Team Leadership
Customers: Fortune 500 and Venture-Backed Content
Focus: B2B, B2C, C2C
Verticals: Retail, Hi-Tech., Finance, Healthcare
Global: HQ in Silicon Valley, 4 Offices Website,
Community
Portals,
Background: Built on 8+ years of research & Social
iPhone, iPad
Media
Industry Recognition: applications
Digital
Strategy
Customer & Lead
Prospect Generation
Nurturing & Sales
24

25. Companies that trust us
Retail & Financial Film & Professional Healthcare, Education Hi-Tech. & Clean-Tech.,
Consumer Services Entertainment Services Bio-Tech. Software Bio-Fuel
Products
UCLA
ICICI
BANK
25

26. Thank You
26

27. Landing Page Optimization Overview
 Goal: Increase Lead Conversion Rate (CR)
 Use Regalix’s ROITM Intelligent Design of Experiment (DOE)
 Create a best page with higher CR for the same online traffic
 Create a continuous optimization program for Citibank
 Activity 1: Plan for DOE
 Break the landing pages into individual elements
 Create variations on each element for testing
 Create DOE test and identify the number of pages/element combinations
necessary for testing
 Create landing pages with the necessary combinations
 Activity 2: Conduct DOE
 Set-up landing pages on ROI
 Run DOE and collect statistically significant data
 Activity 3: Analyze data and build best page
 Activity 4: Continuous improvement
Confidential | The next-generation online customer acquisition engine

28. Activity 3: Analyze data and build best page
 Feed data into statistical system and use Regalix’s proprietary
algorithm to calculate the following:
 The tested page elements that influence the conversion rate
 The specific variations of the elements that worked best
 The combination of elements that could provide a significant higher conversion
rate
 Final landing page is built using the best element and their
variation
 New Page is deployed
 Data is collected to establish higher CR
 Optionally to isolate any market dynamics that is likely to influence the CR:
The New Page will be run simultaneously with the Older Page to get a direct
measure of improvement
 New page is ready for wider rollout
Confidential | The next-generation online customer acquisition engine