The document is a presentation about getting started with experimentation and optimization. It discusses the importance of experimentation for business growth. It outlines six principles of optimization including strategic business alignment, culture of optimization, customer focus, data-driven research, scientific testing processes, and high velocity testing. It provides tips for the first 90 days of experimentation including creating a 90 day growth plan and research methodology. It also discusses overcoming initial hurdles through iterative testing and scaling testing velocity for high growth.

1. Getting Started with
Experimentation
Presenters
Jeremy Epperson, Director of Optimization, 3Q Digital
Carlos Enriquez, Strategic Partnerships, Optimizely

2. Today’s Speakers
Jeremy Epperson
Director of Optimization
3Q Digital
Carlos Enriquez
Strategic Partnerships
Optimizely

3. Housekeeping
• We are recording
• Slides will be available after the webinar is complete
• There will be time to submit questions at the end of
the presentation

4. Agenda
• Why is experimentation important?
• The 6 Principle of Optimization and Growth
• The first 90 days
• Initial hurdles: turning losses into wins
• Tips for scaling
• Q&A

5. A Tale of Two Companies
• Direct Competitors
• Same industry
• Same size
• Different
approaches to
optimization

6. A Tale of Two Companies
Embraced optimization and worked
for continual improvement
• 57 Tests in first year of CRO
• Took a data driven, customer
focused approach
• RESULT: 112% Growth in
Revenue
Struggled with alignment, silos, and
failure to implement:
• 4 Tests in first year of CRO
• Used opinions and stalled due to
disagreements and perfectionism
• RESULT: Filed for Bankruptcy
Company B
Company A

7. Why is optimization necessary?
• Rising costs of media
• Increased competition and new entrants
• Mitigating risk of pushing new changes live

8. What does the foundation of
successful optimization look like?

9. Digital Growth Study
Monitored growth trends and optimization process of
114 companies for a minimum of 12 months each
Factors Analyzed:
• Conversion Rates
• Testing Velocity
• Validated Lift Percentage
• Revenue Growth
Isolated the differences between each group based
on detailed observations of their process and results
RevenueGrowth
Testing Sophistication
Exponential
Growth
Normal
Growth
Stagnant
Declining
4 Categories of Growth based on Optimization Process
Low
High
Low
High

10. What I Discovered…
There are principles that all exponential growth
companies displayed or developed
Main Takeaways:
• The difference in sophistication between exponential growth companies and
the other 3 categories was astounding, but achievable
• What worked for the highest performing companies can be broken down
into a repeatable process that is universally applicable

11. Growth
1.
Strategic
Business
Alignment
2.
Culture
of
Op<miza<on
3.
Customer
Focused
Approach
4.
Data
Driven
Research
5.
Scien<fic
Tes<ng
Process
6.
High
Velocity
Tes<ng
The 6 Principles of Optimization and
Growth
Each of these principles can be
improved on over time with the
proper process and approach

12. Strategic Business Alignment
Main Takeaways:
• Get all stakeholders aligned
before you start
• Map KPIs to business
objectives
• Create a 90 day written
Growth Plan on weekly
sprints

13. Culture of Optimization
Main Takeaways:
• Only use data driven and
highly growth oriented “A
players”
• Share your wins with
research and results to get
buy in
• Improving culture takes
time

14. Customer Focused Approach
Main Takeaways:
• You are not your customer
• Focus on providing value,
not taking value
• Always do what is right for
the customer and you will
grow

15. Data-Driven Research
Main Takeaways:
• Continually gather data and
insights to drive testing
• Every hypothesis should be
tied to a specific data point
or observation
• The more you understand
the customer, the more you
win

16. Scientific Testing Process
Main Takeaways:
• Structure tests to accurately
measure results
• Every test should prove or
disprove a hypothesis
• Statistics mitigates risk

17. High Velocity Testing
Main Takeaways:
• There’s a positive
correlation between testing
speed and growth.
• Decrease the level of effort
• Start with small wins and
build momentum

18. Now that you know the
principles, where do you start?

19. The First 90 Days

20. 3Q’s Proprietary CRO Process

21. Start with a 90 Day Growth Plan

22. Conversion Research Methodology
Step 1: Heuristic Analysis
• Isolate initial high level findings to guide
research
Step 2: Quantitative Research
• Use analytics, attribution, and DMPs to
pinpoint segments, behaviors, and insights
Step 3: Qualitative Research
• Understand motivation and ”why” of the UX
Step 4: Hypothesis Development
• Create data driven hypotheses for prioritizing

23. Tips to scale your research
• Put the tools in place to quickly and easily capture
data that can be turned into hypotheses
• Focus on moving fast, NOT being a perfectionist
• Crowdsource your data points and insights from
across the company

24. Overcoming initial hurdles:
turning loses into wins

25. Iterative Testing

26. The Power of Iterative Testing
The most common mistake is giving up on your
hypothesis or changing your focus before you
get to the big wins:
You can have an accurate data point, but still be slightly
off on the implementation

27. Iterative Testing Case Study
• Iteration 1: Reduced confusion with
process level copy to clarify
messaging and benefits
• Iteration 2: Localized experience to
remove confusion about offices
• Iteration 3: Removed all identified
content distractions for a “stripped
down version” to remove
objections
Results: 18.2% drop in conversions
Results: 4.5% drop in conversions
Results: 38% increase in conversions
Test Results
Test Strategy

28. Scaling up

29. High Velocity Testing

30. A Tale of Two Companies
(Revisited)
Embraced the 6 principles of
optimization and worked for continual
improvement and high velocity testing:
• 57 Tests in first year of CRO
• Took a data driven, customer
focused approach
• RESULT: 112% Growth in Revenue
Struggled with alignment, silos, and
failure to implement:
• 4 Tests in first year of CRO
• Used opinions and stalled due to
disagreements and perfectionism
• RESULT: Filed for Bankruptcy
Company B
Company A

31. Tips for high velocity testing
• Focus on Minimum Viable Product (MVP)
• Remove ALL wasted effort, time, and resources from the
process
• Outcomes Focus: Results vs. Research Insights
• Prioritization Framework and Roadmap Management

32. Key Takeaways
1. Improve your inputs to testing with a
structured process for research
2. Use Iterative Testing to transform
“losses” into lifts in conversions
3. Accelerate your testing velocity

33. 1200+
Attendees
20+
Sessions
100+
Giveaways
30+
Speakers
Wynn Las Vegas, October 17-19
Registration
Early Bird
$750
Expires on 09/01/2017
Early Early Bird
$550
Expires on 07/01/2017
Regular
$850
Expires on 10/16/2017

34. Questions?
Please submit your questions via the text box on
your screen.

35. Thank you