Site Optimization – Testing Tools & Techniques Digital Marketing Auditing Borislav Kiprin "There are three types of testing strategies: • When and how to use each of the three main testing strategies (A/B, A/B/n, Multivariate)? A/B – This particular test results into the comparison of two versions, where the first one is called the test ..."

1. EI MDMK10
Borislav Kiprin
Site Optimization – Testing Tools
& Techniques
Digital Marketing Auditing – Session 8
Individual Report
Plovdiv, 4024
Phone: +359 879993700
E-Mail: bk@borislavkiprin.com
Web: www.borislavkiprin.com

2. 2 Borislav Kiprin
When and how to use each of the three main testing strategies (A/B, A/B/n, Multivariate)?
There are three types of testing strategies:
• A/B – This particular test results into the comparison of two versions, where the first one is called the test
version and the other the control one. This test practice is particularly good when comparing changes in
a single element of the website and can deliver high statistical accuracy. However, there is a
shortcoming with this technique and this is the time it consumes to be implemented successfully.
• A/B/n – This technique differs from the previous one only in the number of variants to be tested (3,4…,10).
Here there are the same advantages in delivering statistical certainty and the same time consuming
effort.
• Multivariate – This approach is related to testing multiple page/site elements in the same time. It reduces
considerably the time needed to perform the test by employing testing tools. However, since the variants
are growing exponentially in this approach, it is very difficult to reach statistical certainty and one cannot
be really sure which one exactly is the right one, since the sample size can be just so big.
A/B and A/B/n strategies shall be used when statistical accuracy is of an essence and naturally when there is a
single element in multiple variations. As to Multivariate, it shall be used when there are multiple elements on a
page in different variations to be changed; a time restraint and statistics are not of such importance.
What has to be taken into account when implementing a testing program?
The implementation of a testing program requires to be approached in a step-by-step model where division
happens as following:
1. Prioritization – (1) Analyze the website current performance, conduct surveys, focus groups, competitive
research and management opinion on key business drivers on the website. (2) Identifying the fields and
elements that need to be change, their impact on the business goals, monetization model and
potential upsides, required technical implementation, easiness to measure the results of the change
and how time consuming it will be for your business.
2. Creating the test – (1) Identify opportunities (Where you need to improve?); (2) Develop hypothesis
(How do you see the change?); (3) Determine the test methodology (A/B, A/B/n or multivariate testing
will you use?); (4) Define success metrics (How will you measure performance?); (5) Design your options
(What are the variants that are viable); and finally (6) Launching.
3. Post-testing analysis – Evaluate how the change has contributed to website performance.
Also to ensure successful implementation one shall plan budget appropriately, employ the needed skill base
(internally or externally) and gain the support from both the departments involved and the executive.
Optimization is only a small part of the daily life of a company and for that reason it might be disregarded or
wave off as an option for number of false resource related reasons.
Summarize the key aspects of each type of testing (price, promotional, message, page layout, new
launches and functionalities, navigation and taxonomy).
• Price – A relatively easy approach that requires A/B or A/B/n testing strategy to identify which price will
attract more revenue and overall profit for the company.
• Promotional – A relatively easy approach that requires A/B or A/B/n testing strategy to identify which call
for action delivers more conversions.
• Message – A critical and fairly easy testing aiming to identify which copy delivers the click-trough rate
and overall conversion.
• Layout – Identifying the critical spots that deliver the highest value and how to locate them in order to
achieve the success you aim at. This testing can get complicated and multivariate is a way to go.
• New Launches & Functionalities – An A/B testing approach that shall be conducted before introducing
the change.
• Navigation & Taxonomy – The most complex testing that compiles a lot of research in the website
performance data, as well as keeping an eye on consumer behavior metrics, business impact etc. It
requires experience and cautious actions trying not to hurt the website’s performance.