Optimizely building your_data_dna_e_booktthh

How to identify and use the right metrics to delight your customers,
engage your audience, and optimize your business.
Building your Company’s
DataDNA

Building Your Company’s Data DNA 2
Table of Contents:
Section 1: Introducing Data DNA
Section 2: Finding your Company Metrics
Understanding and Measuring Goals
Metrics by Business Type
Section 3: How to Take Action on your Metric
Quantitative Data (with contributions from KISSmetrics)
Qualitative Data (with contributions from Qualaroo)
Section 4: Building Data into the Optimization Flywheel
Six Steps of Optimization
Craft your Experiment Hypothesis
Assembling your Data DNA

There’s no question: data is here to stay. The promise
of data, analytics, and web-based tools make the task
of building a business online accessible and afford-
able. Access has created a new challenge: volume.
Leveraging the entirety of a business’ data on an
ongoing basis is tremendously valuable, but requires
equally tremendous resources, beyond the reach or
scope of many businesses’ technology investments.
Given these challenges, how can you make intelligent
decisions, informed by data, to grow faster?
You can incorporate data to inform decisions you
make about growing your presence online. At each
juncture, you have choices to make—how should I
describe my product to my customers? How can I col-
lect more email addresses to grow my audience? How
should I roll out a new feature and collect feedback?
A clear understanding of metrics and the right type
of data at every level of an organization are essential
for focus. Data is also key to uncovering customer
insights and running a more effective optimization
strategy, but should be properly aligned from the
top-down in order to achieve success.
“The best data-driven companies don’t just passively store and
analyze data, they actively generate actionable data by running
experiments. The secret to getting value from data is testing,
and if you’re looking to grow your online business, implementing
well-executed, consistent A/B testing is a necessity.”
—Wyatt Jenkins, Shutterstock*
Access to Data Increases
Expectations
In a recent survey, 78% of all marketers
report feeling pressure to become more
data-driven.* The challenge is to choose
which measurements of online perfor-
mance are essential to a data-driven
strategy.
*Teradata Data-Driven Marketing Survey 2013
*”A/B Testing and the Benefits of an Experimentation Culture,” Harvard Business Review.

How do I know if this guide is for me?
In this guide, we’ll outline an approach to transform-
ing data into action at every level of your organiza-
tion.
You’ll benefit from this guide if:
• You’re already capturing and reporting on data regularly
internally
• You have a team or company culture that is open to experi-
mentation
• You’re looking for methods to turn your company data into
action
• You’re looking to get more meaningful ‘wins’ from your
optimization strategy
Here’s what you’ll learn:
• Why becoming data-driven as an organization should start
with your optimization programs
• How to align your organization around a failproof guiding
light metric
• How to prioritize your optimization strategy against your
shared company goals
• How to move past random, ad-hoc A/B tests to an optimi-
zation process that produces wins (and if not winning tests,
valuable insights derived from a hypothesis)
Being Data-Driven is
Tied to Performance
Companies that rate themselves sub-
stantially ahead of their peers in their
use of data are three times more likely to
rate themselves as substantially ahead
in financial performance, according to
findings from the Economist Intelligence
Unit.*
*Tableau Software & Economist Intelligence Report:
“Fostering a Data-Driven Culture.”
“Low hanging fruit tastes great, but
the rest is worth the work, too.”
—Patrick McKenzie, Kalzemeus Software

In this guide, we’ll cover many different types of
data. The important point to remember is that
the ideal of being ‘Data-Driven’ is actually to be
‘Data-Informed.’ Data points are an important
set of inputs, but they are not a replacement for
human intuition and judgement. This guide will
provide a list of inputs that are strong positive
indicators for how to improve your company goals
and optimization program. What data points
cannot provide, however, is input on what your
business metrics should be, or what initiatives you
want to prioritize.
What is A/B Testing?
A/B testing is a simple way to test
changes to your page against the cur-
rent design and determine which ones
produce positive results. It is a method to
validate that any new design or change to
an element on your web pages or mo-
bile apps is improving your conversion
rate. It is the assumption that you have
never reached the ‘best’ version of your
website, mobile app, or product, and that
the best method of learning about your
customers is through iterative testing and
experimentation.
What is Optimization?
Optimization begins with broad, sweep-
ing changes and tests that encompass all
of your website traffic, but can become
more granular, by running A/B tests to
understand how changes affect different
types of visitors, and then adapting site
content to eventually deliver a more per-
sonalized experience for each individual.

Section 2:
Finding your
Company Metrics

You’ve probably wondered at some point: “What’s
the fastest way for me to create business impact?”
We could frame this question with two others:
1. “What’s the fastest way to make a measurable business
impact?”
2. “How do I define success?”
Toreachthegoal of becoming a data-drivenorganization
or a data-informed individual, we recommendstriving
foronemetric that defines the goals of your business.
When you choose your ‘guiding light’ metric, ask
these questions to determine whether it will stand the
test of time, and generate strong optimization results:
• Is it quantifiable? Your metric should be measurable and
understandable at-a-glance.
• Is it visible and clearly communicated? Your metric
should serve as a reference point in conversations.
• Is it understandable? It should require very little
explanation.
• Is it comparative? You should be able to compare the
metric to benchmarks, like your competitors’ performance
or other points in time.
• Is it a rate or ratio? Showing a rate of change is more
illustrative than an absolute number.
• Does it change your behavior? If the metric you choose
doesn’t enact change, you need a different one.*
• Is it attributed to revenue? For many people in organiza-
tions, this measurement is what communicates above all
else. If you have difficulty attributing your efforts to value
for the business, you may struggle proving ROI.
Many analytics experts have discussed
the value of identifying a single metric,
which should be carefully chosen and
then rigorously tested. We recommend
Lean Analytics by Ben Yoskovitz and
Alistair Croll.
Use these questions
to test and strengthen
your metric statement.
Before you take any actions to improve
your metric, ensure that you have tested
it internally to ensure that it will with-
stand scrutiny and testing over time.
*Questions 3-6 reference Ben Yoskovitz and Alistair Croll, Author of Lean Analytics

Having a singular goal is helpful not only for orga-
nizational focus, but it will be invaluable for more
specific tasks, like your optimization efforts. In order
to run more intentional, ROI-positive A/B tests and
experiments, you need a guiding metric that you can
brainstorm around and prioritize against. We’ll cover
the data inputs that can inform these types of tests in
Section 3.
“What gets measured, gets managed.”
—Peter Drucker, Business Management Thought Leader

Understanding and
Measuring Your Goals
What do you do on a day-to-
day basis to grow your business?
Use this framework as a starting point for determining what your priority should be for
your business, and by association, your online marketing and other customer touch-
points. What parts of your website directly serve the metric you are working to improve?
Think of these areas as prime candidates for optimization.
What does this activity help you
accomplish?
What is the priority for each of
these activities?
What metrics do you currently
report on on a regular basis?
If you were to pick one activity
and corresponding goal to work
on for the next 1-3 months,
what would it be?
Which of the metrics from Ques-
tion #4 help you to understand
the progress you are making on
the answer to Question #5?
Which of the metrics from Ques-
tion #4 could you live without?
Essential
Important
Nice-to-have
1.
2.
3.
4.
5.
6.
7.

If you: You might use your website to: And could be measured as:
Sell products or
services
(E-Commerce, Platforms)
Collect orders for your
product
Distribute and engage
your content
Capture, engage,
and retain users
Time on site
Articles viewed
Votes or comments generated
Social traffic
Signed up users
Paying customers
Up-sold customers
Churned customers
Number of orders over time
Average order value
Repeat customers
Conversion Rate
Publish content
(Media, user-generated content
like Reddit and Wikipedia)
Develop a web-based
tool
(SaaS or Mobile app company)
Metrics by Business Type
Good candidates for top metric
statements:
“I measure conversion rates from free trials
to customers because my most important
metric is acquiring new customers.”
“I measure average donation value, because
my most important metric is dollars raised
during our fundraising campaigns.”
“I measure conversion rate on my homep-
age, because that is where the majority of
my new leads are acquired.”
Not as strong metric statements:
“I measure pageviews to product pages on my
website.”
“I measure social shares.”
“I measure the amount of traffic to
my pricing page.”
These statements reflect strong metric statements sup-
ported by a clear rationale. Strive to consolidate and
strengthen your overarching metrics so that everyone can
understand and tailor their priorities to match them.
These metrics are not supported by a strong rationale, and
are more likely to be vanity metrics. They are not enough (on
their own) to justify prioritizing for optimization.

Section 3:
How to take
Action to
Improve your
Metric

You may be wondering: how does eliminating metrics
help me use data? The answer lies in the touchpoints
you have with your customers. Once you have identi-
fied your overarching goal, you will be in a position to
identify touchpoints that can be improved in service
of the corresponding metric.
In order to take action that will improve the outcome
of your metric, it is important to effectively identify
high-value areas of your website, product, or other
customer touchpoint. We will explore the best ways
to gather information about these high-value areas
later in this section.
Each business metric can be broken
down to apply to different levels
and functions within your orga-
nization. A team should plan
to adopt a secondary metric
that supports your top-level
company goal. These levels
may resemble the diagram
to the right:
At the most granular level,
your website and product
goals can be supported by
running experiments in an
attempt to improve the con-
versions that increase perfor-
mance, which in turn affect your
highest-level goals.
Hypothesi ze Tests
to Increase AOV

How do you measure the success of these facets of
your online business? Your efforts to present your
business online and capture interest, engagement, or
revenue require conversions.
Take stock of the conversions that support your
top-priority metric. These areas are prime candidates
for optimization, where you’ll run experiments and
A/B tests in an effort to improve conversions.
Getting started with optimization is not the most
difficult step to success; the challenge will present
itself in the struggle to keep continuously A/B test-
ing to find improvements, uncovering winning tests
consistently.
In the following section, we will explore how to use
a combined qualitative and quantitative approach to
build your Data DNA into your website and app per-
formance. Then, we’ll discuss how to funnel these
data points into experiments that uncover improve-
ments and build your data-driven approach to opti-
mization in Section 4.
Using Quantitative Data to Determine
which Areas of your Site to Test
Your website analytics are an indispensable tool for
determining the strengths and weaknesses of your
website. Leverage your website data in order to effec-
tively prioritize and brainstorm the best experiments
to run on your website.
What is a conversion? A desired action
taking place on your at a customer
touchpoint—a click, or registration, or
product action, for instance.
Your conversion rate is the current pace
at which website visitors take action at a
given touchpoint.
Conversion rate optimization and A/B
testing describe the practice of experi-
menting with elements to try to improve
a conversion rate.

Mapping Your Company’s Data DNA 15
KISSmetrics Contributed Section:
Before you can find ways to accelerate your acquisition and
growth, you’ll need to start tracking your core funnel.
What’s your Core Funnel?
It’s slightly different for each business but it measures how
many people move through the major steps of your acquisition.
Don’t worry about tracking every little click, page, or action
that people take to become a customer. Focus on the core
steps.
Let’s look at a few examples.
SaaS (subscription as a service) businesses drive traffic, convert
that traffic into free trials, get those free trials to use their prod-
uct, and convince people to purchase an ongoing subscription
to the software. So a SaaS funnel is built from these steps:
Using Funnels to Find Big
Wins and Accelerate Growth
SaaS Funnel
1. Visited site
2. Signed up for free trial
3. Used product
4. Purchased subscription

An ecommerce funnel looks pretty similar:
For whatever business model you have, break your acquisition
down into 4-5 core steps that everyone goes through.
Tracking your Funnel
Ideally, you’ll have a customer analytics tool that can track
people as they move through the different steps of your funnel.
When you get to the point where you’re doing a lot of optimiza-
tions on your funnel, have plenty of traffic, and acquire custom-
ers consistently, start looking at customer analytics tools that
will help you track the entire funnel.
Even if you’re not there yet, you still want to get a general sense
for how your funnel performs. Google Analytics and some inter-
nal tracking will get you pretty far.
E-Commerce Funnel
1. Visited site
2. Visited product detail page
3. Added product to cart
4. Entered checkout
5. Finished checkout

Ecommerce companies have it easy by setting up ecommerce
tracking in Google Analytics. This tells you how many people
purchase and their conversion rates. You’ll also want to config-
ure custom events to see how many times products get added
to a cart or visitors enter your checkout.
For SaaS, set up Google Analytics and trigger a goal when
people start a trial. You’ll also want to build out some internal
tracking that tells you how many new accounts you acquired,
how many of them started using the product, and how many of
them purchased. Add this data to your internal customer data-
base and query it every week or so to see how you’re doing.
Don’t worry about having perfect tracking when you get start-
ed. Your main goal is to have a general idea for how your funnel
performs.
Optimizing your Funnel
As soon as someone’s ready to start optimizing a funnel, they
usually pick a random element on a random step. It might be
a button color on a call to action, a snippet of copy, an image,
or a layout that just feels “off”. Then they’ll launch an A/B test
if they have enough traffic or maybe they’ll just launch the
change and hope for the best.
After a few rounds of this, you’ll find that your conversions are
exactly the same as where they are now. They won’t budge an
inch.
Why?Becauseit’sincrediblydifficulttofindteststhatreallymake
adifference.Mostonlinemarketing“bestpractices”give2%-3%
wins.Notonlyisitincrediblyhardtodetectsmallwinslikethis
(youneedatonofdatatoteston),smallwinsaren’tusefultoa
business.Theyonlymatterifyoucanfindalotoftheminashort
timeperiod.Which,again,requireslargeamounts of data.

So how do we find bigger wins consistently?
Start with qualitative data on the step of your funnel that you
struggle the most with. Qualitative data is feedback from users
or customers that isn’t a metric. Surveys, customer interviews,
feedback forms, usability studies, and heatmaps are all qualita-
tive data.
It’s common to have one step that is a major roadblock to
acquisition. Focus on collecting feedback from your customers
at that step of the process. Ask them why they don’t want to
purchase, why they didn’t use the trial, or why they didn’t finish
the checkout.
Asking for targeted feedback on an underperforming step of
your funnel will give you plenty of ideas on what to test. There’s
never a 100% guarantee of finding a big win but you’ll definitely
find them more frequently than if you test random elements
across your funnels.

By examining your website analytics, you will be able
to pinpoint valuable areas of your site that are in need
of improvement. If you are successful in running A/B
tests and experiments to improve these key areas of
your website, you’ll be one or more steps closer to
moving that ‘guiding light’ metric, and shifting your
company towards a more data-driven approach to
solving problems.
Website analytics have a tremendous number of ben-
efits for structuring your optimization strategy:
• Quantifiable, high-value traffic segments
• Common goals that sync across your website analytics to
your website optimization platform
• Ability to scale and analyze website data across segments,
cohorts, and more
Quantifiable analysis of your website is only one part
of the data equation. To complete the picture of how
your visitors are converting and engaging (and more
importantly, why they aren’t), you’ll need to supple-
ment your quantitative data with qualitative insights
to enhance your data DNA.
Using Qualitative Insights to Determine
which Areas of your Site to Test
Qualitative, anecdotal feedback is an effective way to
incorporate data into your business at any stage. This
data is essential because it comes directly from your
visitors and users.

This information is essential to determining why
certain events occur on your website. When quantita-
tive data provides insight into what happens (or isn’t
happening), these qualitative data points can illus-
trate why. Why did that customer buy? What brought
them to your website in the first place? What do your
visitors expect from your product or offering?
Qualitative insights come in many forms, including,
but not limited to:
• User testing:
For a fee, a person unfamiliar with your website will com-
plete a task on your website while providing their thoughts
and feedback. There is a bias inherent in this qualitative
data, since the person knows they are being tested.
• On-page surveys:
Use these surveys to collect ‘in the moment’ feedback from
visitors to a website. The more questions you ask, however,
the lower your completion rate is likely to be.
• Heat mapping:
These tools will show which areas of a given web page
attract the most ‘heat,’ or attention, from clicks and scroll-
ing on your site.
• Long-form surveys:
Often delivered via email, these surveys can be used to
collect in-depth feedback on your company’s positioning,
perception from customers, and some of their most valued
offerings or products from your business.
“A solid test hypothesis is an informed solution to a real prob-
lem – not an arbitrary guess. The more research and data
you have to base your hypothesis on, the better it will be.”
—Michael Aargaard, Conversion Rate Optimization Expert

• Feedback from cancellations, returns and complaints:
Take your customers’ constructive feedback, and decide
what visible changes to your website could help to alleviate
pain points and frustrations they might be having.
• Live chat feedback:
Uncovering pain points and visitor intent will be key to
generating winning experiment ideas.
• Customer feedback:
Anecdotal feedback from engagements with customers on
calls, at events, and in other formats.
You can leverage these tools (many of which have
affordable and easy-to-use plans for businesses)
to collect feedback directly from the visitors to your
website or customers. You must hone in on motivat-
ing factors, however, in order to collect meaningful
feedback.

To ask questions of your website visitors and collect more
data points about what you should optimize, practice asking
the right questions.
Here are two types of questions you should ask:
1. Intent - What did your website visitors come to
your website looking to find, or do?
2. Frustration - Could you find what you were look-
ing for? What was broken?
Consider the types of questions you could ask to understand
key parts of your website funnel:
On your homepage:
• What did you come to our website today hoping to accom-
plish?
Before a key conversion:
• Do you have any additional questions about this product
or service? Determine what information your visitors are
looking for before they purchase.
• Is there anything stopping you from completing this order?
Perhaps the visitor needs more information to make their
decision. Maybe an unexpected bug with your checkout
flow is preventing them from converting.
After a key conversion:
• What made you purchase/sign up/donate to us today?
On your content site:
• What topics would you like to see us cover more? You could
make this multiple choice or leave as a free response
• Would you like to see more recommended content on this
topic? Determine the best way to design recommended
content modules for your content.
AAA: Ask your
Audience Anything
A few tips to get even
better data from your
qualitative research:
Be polite: You’re human - so are your
website users. Ask the questions in an
approachable, visitor-friendly manner.
Phrase questions with a “What do you
think of your website? Kindly tell us:”
approach.
Ask for honesty: Your website visitors
are also nice people. It’s sometimes dif-
ficult to provide constructive feedback.
“We’d appreciate your brutally honest
feedback as we work on improving our
website experience for you.”
Keep questions open: Try to avoid ask-
ing questions like, “What was frustrating
about your experience today?” or, “What
do you think of our [product or service]?”
These questions predicate a certain type
of response, and will bias your visitors’
responses.
Ask visitors to self-select: You can
also learn more about your audience by
asking them about their interests, their
business type, or products and services
that they like to use. This will help you to
segment your web traffic and experiment
with running more targeted tests.

Now, it’s your turn: What are the steps (or clicks) in
your website funnel, and what questions would you
ask at each step to reduce friction and improve your
visitors’ experience?
?
?
?
?

What do you think is the most common mis-
conception about Conversion Rate Optimi-
zation and A/B testing?
The number one misconception is that A/B testing is simply
about running a test here and there and hoping for improved
results. Without an optimization process that focuses on
continual improvement A/B testing often fails to live up to it’s
promise. Without organizational rigor to make A/B testing a
priority, companies give up failing to see early wins, which
ultimately costs them the long-term gains that come from a
systematic approach to optimization.
Do you think that most companies online
are effectively communicating with their
customers? Why or why not?
Most companies lack a process for regularly collecting and
then taking action on user feedback. Users provide feedback
in numerous ways, from bouncing off of web pages, to taking
surveys, leaving reviews, filling out customer support tickets
and posting on social media. With all of this feedback com-
ing in, you’d expect companies to be constantly processing it
and using it to improve the visitor experience, and ultimately,
their business. But more often, feedback is triaged to manage
customer complaints rather than used for true learning and
Online Success and Optimization:
Why Quantitative Data
isn’t Enough
Q:
Q:
A:
A:
Interview with Startup Marketer and
Qualaroo CEO Sean Ellis

business improvement. The most successful companies have a
process for collecting, parsing and using customer feedback to
improve their business.
What is the best way to get actionable,
qualitative data from a website survey?
The single best way to get actionable qualitative data from a
website survey is to use them to try to understand specific user
behavior. For example, using a website survey on pages that
have high bounce rates, or pages within your conversion funnel
have high drop-off rates, can give you insights right from the
customer that help you understand why they’re leaving. When
you understand why a behavior is happening, it’s much easier
to take action on the feedback and try to change the behavior.
Do you have any tips on how to choose
which feedback should be incorporated into
your testing pipeline?
Feedback from qualified people is the most important. The
people that are potential customers who aren’t converting are
the ones you want to focus on. Ignore the people who aren’t
qualified or interested in what you’re offering from the begin-
ning. For example asking people who converted a question
such as “What almost stopped you from signing up?” or “What
made you decide to sign up?” helps you understand the needs
of qualified visitors. This qualified feedback will help you sort
through the data from exit surveys that include both qualified
and unqualified responses.
What would you tell someone who is look-
ing for help creating strong hypotheses for
their tests?
Q:
A:
Q:
A:
Q:

There are two ways to create a hypothesis. In the first case you
can look at the data and then spend hours or days with your
team trying to interpret what the data means and what you
should test next. You can create hypotheses from your interpre-
tation and from your team’s opinions. Or you can ask visitors to
that page what the actual problem that they’re encountering is.
No need to interpret data, just ask visitors and get immediate
feedback. I believe asking visitors what issues they’re encoun-
tering on your site, conducting user research, is the best way to
formulate strong hypotheses that make for valuable tests.
How do you think companies should ap-
proach staying focused when it comes to
optimization? How do you measure the
progress of your CRO program?
Having the organizational rigor to stay focused on conversion
optimization is the hardest part of this process. A/B testing is
notaone-offproject—ratheritisacontinuousprocessofimprove-
mentthatneedstobeinmotionatalltimes.Econsultancy reports
that 87% of companies doing A/B testing run between 1 and
5 tests each month. The best companies run many times that
number.
Companies need to commit to A/B testing as a core part of their
digital marketing program and invest in it accordingly. Stick to
the process and eventually it will become an addictive habit,
with the organization constantly trying to outdo its previous
test. Systematic, ongoing A/B testing is going from a compet-
itive advantage to a competitive necessity. It’s a key require-
ment for online success today.
A:
Q:
A:

At Clearlink, the team asked the question: “How can
we see inside the minds of our customers and be sure
we are offering what they need?” Despite following
testing best practices, the team has frequently en-
countered unusual or contradictory A/B test results.
Utilizing qualitative data gives them insights into
these results and helps them better understand and
explain when met with this challenge.
They hypothesized that using a qualitative insights
tool in conjunction with their optimization software
would enable them to better serve their customer
needs through well-researched and well-executed
tests, resulting in the ability to integrate and better
understand tests’ success and failure, and ultimately
leading the to understand our market better.
While many times quantitative metrics might be interesting, they can-
not tell the entire story when it comes to the customer’s path. In short,
we know the ‘what’, but not the ‘why.’ For these issues we have to turn
to qualitative research and customer insights to make sense of the
customer needs and dive in deeper. This is how we innovate; this is
how we iterate; this is how we make sense of the larger picture.
—Rachel Johnson, Consumer Insights Research Manager, Clearlink
How Clearlink Built Qualitative Insights
into their Data DNA
Content and conversion services provider, Clearlink,
uses survey software* to foster its data-driven ap-
proach to engaging with its audience and collecting
qualitative feedback from its online audience.
*Clearlink implemented Qualaroo survey software for the purpose of these experiments.

By using on-page surveys to offer time-sensitive
promotions, the team was better able to serve more
relevant promotions to their website visitors in a
more time-effective manner. This led to overall
higher conversion rates for their sites:
• Stand-Alone vs. Bundle Pricing: The team learned from
survey results that stand-alone pricing was what Clearlink’s
customer was looking for. After testing stand-alone pricing
messaging against bundled pricing options, stand-alone
pricing won with a conversion rate improvement of +5.4%,
contributing to a testing win of $41,500 in annualized
revenue impact.
• Lead Capture: The team hypothesized that by using the
survey tool, they would be able to send the customer
directly to their conversion funnel (the sales floor) by
engaging them more quickly on the initial landing page.
In addition to the phone number, they added promotional
messaging (similar to display ads) to the messages to
further drive conversion. Using the survey tool bypassed
the normal lead time required for a custom overlay to be
designed, developed, and pushed to live site, significantly
increasing Marketing’s agility in responding to changing
customer tastes, preferences, and demands.
Using quantitative and qualitative research meth-
ods together led to several “wow” moments for the
Clearlink team. By combining the two data streams,
qualitative and quantitative, they created an en-
hanced testing cycle, making the most of testing
wins, successes, and failures. Qualitative findings
sped up the testing cycle, and the team gained im-
pactful insights more often. Clearlink’s prevailing
sentiment is that “The more often you fail, the more
often you’ll win; so speed up the frequency at which
you fail.”
Clearlink’s mantra is: “The more often
you fail, the more often you’ll win; so
speed up the frequency at which you
fail.”

Integrating quantitative and qualitative data into the
optimization cycle creates a more effective testing
program for attaining website goals. Although tak-
ing the time to add consumer research and the cus-
tomer’s point of view can seem like the longer path,
for teams like Clearlink it ultimately improved the
team’s feedback loops and led to insights that previ-
ously might have gone undiscovered.
At this point, you have used data to inform your
strategy from your top-level business goals down to
your website and product-level goals. Now that you
have a sense of what could stand to be improved, let’s
turn to brainstorming and prioritizing your new ex-
periment ideas to complete your Data DNA sequence.

Section 4:
Build Data into
the Optimization
Flywheel

You’re well on your way to running high-impact,
ROI-positive experiments to improve your customer
touchpoints. The next step is to distill the data you’ve
collected and begin to take action. In this process,
you will funnel your quantitative and qualitative data
points into concepts for A/B tests that can be run as
experiments on your website to improve key conver-
sions and support your business goals. We call this
six-step framework the “Optimization Flywheel.”
In order to see success with this approach, you
should:
• ●Build a rigorous practice of consistently collecting quantita-
tive and qualitative data about your website funnels.
• Maintain visibility of your organization’s overarching
metric. If your focus as a company changes, your website
goals and individual experiment objectives should change
accordingly.
• Maintain a sense of experimentation as broadly as possible
across your company. Any visible changes to your website
help to understand what works best for your customers and
visitors. The pursuit of this understanding should support
the goals of the company as well, as discussed in Section 2.

Building a Data-Driven Optimization Strategy 32
What is the Optimization Flywheel?
Optimization is a continuous cycle. In order to ensure that you
are responsive to the intent and frustrations experienced by your
audience, you can leverage optimization to run experiments and
discover improvements to your website.
There are six steps of the Optimization Flywheel:

Step 1: Define Goals
As defined in Section 2, it is important that you begin
your optimization process clearly aligns to your busi-
ness goals. A clear understanding of the metrics you
are optimizing for will help with prioritization and
enables continual iteration and learning from exper-
iments.
Step 2: Determine Optimization Points
Identify a step in your funnel that is a prime candi-
date for optimization. It might be your homepage call
to action (CTA), your campaign landing pages, your
checkout flow, or your recommended content.
Make sure to choose an area for optimization that has
a direct correlation to your business goals. The quan-
titative funnel analysis in Section 3 will provide con-
text for which areas of your website can be improved.
Will optimizing this step of your website experience
create a measurable change to the metric you identi-
fied in Section 2?
Step 3: Hypothesize Improvements
A strong hypothesis about how your experiment will
perform is core to running a winning A/B test. Your
qualitative data collected in Section 3 will provide
clear indicators of what should be tested to better
match your visitors’ intent and solve their frustra-
tions with your product or website. We recommend
you take the time to collaborate on hypotheses with
*Reference “Craft your own Experiment Hypothesis” for more detailed instructions on hypothesis creation.

your team, and properly document them along with
the data that you used to inform them.
A hypothesis is the following type of statement:
Hypothesize what types of changes to your website
would produce a positive change in conversions.
Step 4: Create Variations
When setting up an A/B or multivariate test, you
must develop a variation of your website feature that
you would like to test against the current version.
Use your hypothesis and your understanding of your
analytics and qualitative data to isolate a variable that
can be changed. The fewer the variables, the more
straightforward your test will be. Changing multiple
variables at once is possible with multivariate testing,
but complicates the test and confounds the hypothe-
sis, making it difficult to simply prove or disprove.
If ________________________ ,
then _________________ will occur.*
[I make this change to my website]
[the following outcome]
“It’s about coming up with your hypothesis first and then develop-
ing a test from there. Rather than saying, ‘I want a test button cre-
ated,’ or ‘I want to try two different layouts,’ you create a hypothe-
sis first, for example, ‘I think green buttons attract more attention
and therefore will have a higher clickthrough rate.”
—Michael Burk, Senior Online Product Manager, Electronic Arts

Once you have isolated your variable, create varia-
tions of your website based on the modification of
that variable.
Step 5: Run Tests
Prepare an experiment using your A/B testing soft-
ware of choice. For more detailed instructions on how
to set up and run an experiment using Optimizely,
visit our Knowledge Base.
Step 6: Measure the Impact
Allow your experiment to run until it has reached a
statistically significant result. Did your variation win,
lose, or draw even with the current variation?
If the test is a win, congratulations! Your hypothesis
was correct, and you can continue to build upon that
test by applying the data-informed learning to other
areas of your site, or testing another more advanced
hypothesis.
At this point, make sure to extrapolate the value of
your improvement to the website across all of your
traffic over an extended period of time. What will
your lift translate to in sign-ups, orders, et cetera
over the course of a year?
If your test was a draw or your variation lost, investi-
gate why that might be. It could be the case that your
hypothesis needed additional research, or that you
didn’t account for a behavior or event that skewed
your test results. In the event of a losing test or a

draw, it is still possible to gain additional insights
that you had not anticipated through further analysis
and discussion with your testing team. At this point,
discuss what the data couldn’t account for as you
planned the experiment. What would you do differ-
ently in your next hypothesis and experiment?
Putting it all Together
Here’s an example of Data DNA in action:
• Take an e-commerce company that wants to improve the
value of each customer they acquire. This is the top-level
metric that they have identified is an excellent indicator of
the success of their business.
• To improve this metric, the team uses website analytics
to identify the top traffic pages for a high-value customer
segment: returning traffic. They identify conversion points
for returning traffic that differ from other traffic types.
• To collect qualitative data, they survey their customers to
understand why they may not be converting at this point.
• Based on this data, the team hypothesizes a change that
could help to improve the conversion rate for these visitors.
They run an experiment and to determine whether their
hypothesis was correct.

What does a winning test hypothesis look like? It’s a represen-
tation of your Data DNA on a hyper-focused level. Your qualita-
tive and quantitative data collection will help you formulate a
strong, testable prediction.
Hypotheses are statements, not open-ended questions. They
address a question with a proposed solution. Crafting a hypoth-
esis to address an open question or problem on your website
enforces a well-rationalized, thoughtful proposal for how to
address that problem.
To take your hypothesis even further, consider what you would
learn if your prediction was proven correct or incorrect in an
experiment. What would you learn in each scenario?
Craft your own
Experiment Hypothesis
“IF _____________, THEN ___________DUE TO ____________.”
[Variable] [Result] [Rationale]
The Variable:
A website element that
can be modified, added,
or taken away to pro-
duce a desired outcome.
Result:
The predicted outcome.
(More email sign-ups,
clicks on a call to ac-
tion, or another type of
behavior.)
Rationale:
Demonstrate that you
have informed your hy-
pothesis with research:
what do you know about
your visitors from your
qualitative and quantita-
tive research that indi-
cates your hypothesis is
correct?

Assembling your Data DNA
Now, it’s time for you to take action:
• Assess your top metrics with your team and company at
large.
• Take stock of your web analytics, and bring together your
sources of qualitative data from your customers.
• Most importantly, take action by applying an experimental
mentality to your online presence with an intentional, mea-
surable approach to continuously optimizing interactions
with your audience.
By incorporating data into your metrics and goals at
every level of your business, you’ll move your
organization towards a data-driven process for
making decisions and instrumenting change. More
importantly, your data will become actionable at
the customer touchpoint level, where you’ll better
engage and convert your audience into valuable cus-
tomers, repeat visitors, and users.

ABOUT THIS GUIDE
Building your Company’s Data DNA
Written By:
Shana Rusonis
Content Marketing Specialist, Optimizely
@srusonis
Designed By:
Jon Saquing
Communication Designer, Optimizely
@JSaq
Thank you to:
Ural Cebeci, KISSmetrics, Sean Ellis and Morgan Brown.
SOURCES
• “Teradata Data-Driven Marketing Survey,” Teradata.
• “Fostering a Data-Driven Culture,” Tableau Software &
Economist Intelligence Report.
• “A/B Testing and the Benefits of an Experimentation
Culture,” Harvard Business Review.
• Lean Analytics by Alistair Croll and Benjamin Yoskovitz.
• For more detail from Hiten Shah, Co-Founder of KISS-
metrics: “Increase your Testing Success by Combining
Quantitative and Qualitative Data,” OptiCon 2014.
ABOUT OPTIMIZELY
Optimizely is the world’s leading
optimization platform, providing
A/B testing, multivariate testing,
and personalization for websites
and iOS applications. The platform’s
ease of use empowers organizations
to conceive of and run experiments
that help them make better data-
driven decisions. With targeting
and segmentation using powerful
real-time data, Optimizely meets the
diverse needs of any business looking
to deliver unique experiences to their
visitors.
Amsterdam Office
Nes 76
1012 KE Amsterdam
The Netherlands
San Francisco Office
631 Howard Street, Suite 100
San Francisco, CA 94105
OPTIMIZELY.COM/DEMO
To learn more about Optimizely,
schedule a live demo today at
To learn more about best practices for people, process, and technology for a
winning optimization strategy, download a copy of our Roadmap to Building a
Data-Driven Optimization Team.
Optimization can be instrumental in improving the performance of your paid
search funnels. Learn how to maximize your marketing ROI by reading
Optimizing your Conversion Engine: Search Engine Marketing.

Optimizely building your_data_dna_e_booktthh

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (18)

Andere mochten auch

Andere mochten auch (11)

Ähnlich wie Optimizely building your_data_dna_e_booktthh

Ähnlich wie Optimizely building your_data_dna_e_booktthh (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (19)

Optimizely building your_data_dna_e_booktthh