Every business in the world that is undergoing a digital transformation will benefit by increasing its capacity to run experiments. Bringing teams across product development, growth marketing, and engineering together to create a successful experimentation program is easier said than done. Each team brings unique goals, ways of working, and new sets of challenges. Alek Toumert, lead strategy consultant at Optimizely, shares how to bring teams across the organization together to build and run a best-in-class experimentation program. You’ll get a behind the scenes look at how companies structure their experimentation teams to drive business results. - The five pillars of a successful experimentation program - How to distribute experimentation ownership and which key roles are needed to make a program successful - Which goals and KPIs to track and best practices for sharing results internally

1. 1
Stop the Silos!
How to Bring Teams Together to Build a
Successful Experimentation Program

2. Housekeeping ● Customize the widgets on your page
to your preference
● This webinar is recorded and you will
receive the link with the slides in the
next few days
● We will have time for questions!
Please submit them in the Q&A box
on the left side of the screen

4. 4
The Experimentation Methodology
Avoid investing in building experiences
and features that won’t meet goals
Minimize risk when releasing
changes and new features
Optimize for growth and maximize
performance on the winning ideas
Plan &
Ideate
Iterate
Analyze
Prioritize &
Design
Build & QA
Experiment

5. 5
Today
Outline Optimizely’s Pillars of a Successful Program
Share How Organizations Structure Around Experimentation
Highlight What High Maturity Programs Do

6. 6
Program Pillars

7. 7
Education
Program
Measurement
Workflow
Coaching & Enablement
Community
Pillars of a successful experimentation program

8. 8
Education
Program
Measurement
Workflow
Coaching & Enablement
Community
Pillars of a successful experimentation program

9. 9
Executive Sponsor - Aligns KPIs, sets program direction, and
removes resource / cultural blockers.
Program Manager - Drives initiatives of program forward and is in
charge of measurement of success.
Technical Lead - In charge of ensuring implementation, integrations,
and other technical needs are executed.
Developer / Engineer Lead - Lead developing and QA’ing
experiments along best practices.
Content Lead - Establish processes and best practices of developing
new content for variations.
Analytics Lead - Enables measurement within relevant systems and
educates end users on results analysis.
Design / UX - Instills brand, design, and UX best practices for
ideation and variation development.

10. 10
Ideate
● UX Research
● Analytics: Behavioral, Heatmaps, etc
● Audience Segmentation
● User Testing
Implementation and Test Configuration
● Pages, Tags, Events, Audiences
● Configuration: Custom JS, Tracking
● Integration: Analytics, Custom Segments
● QA Tooling
Experiment Design
● Experiment Design
● Test Plan Documentation
Build, QA & Launch
● Wireframes
● Creative Assets
● Variation Development
● QA
Results Analysis
● Segmentation & Analysis
● Recommendation & Iteration
● Results Reporting
● Results Sharing
Planning & Prioritization
● Roadmap & Prioritization
● Process Management
Analyst
Analyst
Developer
CoE
Program
Manager
Developer
Analyst
Designer
CoE
Program
Manager
Designer
Analyst
Designer
Analyst
Technical
Lead
Analyst
Technical
Lead
&
Executive
Sponsor
Developer
Developer
CoE PM
Data
Analytics
Responsible Accountable Supporting Consulted Informed

11. 11
How Do I Manage All These People?!
A place for review A place for conversation A place for execution

12. 12
Education
Program
Measurement
Workflow
Coaching & Enablement
Community
Pillars of a successful experimentation program

13. 13
Program Metrics
Measure the efficiency and effectiveness of the experimentation
program; these are operational metrics from use of Optimizely’s
platform by the teams
● Velocity
● Conclusive Rate
● Optimizely Results Page Views
Maturity Model
Taking the Maturity Assessment will better help us understand the
gaps and potential areas of improvement against your peers:
https://www.optimizely.com/maturity-model/.
Experimentation Success Metrics
Value Metrics
Measure the financial results and benefits of experimentation on the
business; these are primary product / experience metrics to be
improved upon
● Purchase Conversion Rate
● AOV
● Loyalty Sign-ups
Project Metrics
Measure the quality and timeliness of Optimizely’s delivery against
purchased services
● % Deliverables Complete
● Hours Burn
● # of ODS Experiments Launched

14. 14
Example Success Metric Definitions
Purchase CR
Cumulative average uplift of all experiments that had a
positive impact on Conversion rate.
ATC Rate
Cumulative average uplift of all experiments that had a
positive impact on add to cart rate.
Significance Rate
Percentage of experiments that were run and that reached
statistical significance on the primary metric.
Velocity
Number of experiments started in Optimizely over a specific
time frame.
Variations / Experiment
Average number of variations per experiment. Best practices
based on research suggest at least 4 variations to maximize
win rates.
Win Rate
Percentage of experiments that resulted in a statistically
significant winner on the primary metric
Hours Burned / Hours Planned
The number of days complete divided by the total number
of days in the engagement.
PVR
Cumulative average uplift of all experiments that had a
positive impact on PDP views.
Annual Revenue
Cumulative uplift of all experiments that had a positive
impact on Revenue.
Web Experiments within Scope
Number of experiments completed out of the total number
scoped to be supported by Optimizely Solutions Architect.
Concluded
Number of experiments concluded with a learning and action
taken.
% of Experiments with Audience
The percentage of experiments that contained some defined
targeting at launch.
Average Days Running
Average number of days experiments were in a ‘Running’ in
Optimizely.
ODS Experiments Launched
Number of experiments completed out of the total number
scoped to be supported by Optimizely ODS team.
Purchase CR
Cumulative average uplift of all experiments that had a
positive impact on Conversion rate.
Time to Production
Avg days it takes to productionalize an winning
variation

15. 15
Program Benchmarks
Measure
Velocity - The average number of experiments /
campaigns launched per week.
Why It’s Important
Conclusive Rate - The % experiments that have hit a
statistically significant result on the primary metric (win or
loss).
Win Rate - The % of experiments that have hit a
statistically significant result and uplift on the primary
metric (a win).
As we launch more experiments, we generate more
learnings and more hypotheses/iterations.
As we find more statistically significant results, we drive
more direct actions from our results.
As we find more winners on our most important metrics,
we improve the business’ KPIs more often.
Feature Flags - The average number of feature flags
launched per week.
As we launch more feature flags, we practice safer and
quicker deployment practices.

16. 16
Design Benchmarks
Measure
Duration - The average number of days in an active
state, collecting data per experiment.
Why It’s Important
# of Variations - The average number of variations
launched per experiment.
# of Audience Conditions - The average number of
audience dimensions used per experiment.
# of Goals - The average number of goals configured
per experiment.
Having an understanding of Stats Engine and
consensus on when to call tests.
Utilizing multiple solutions when solving a customer
problem increases likelihood of finding the global
maxima.
A targeted solution to the problem can improve
magnitude of result.
Understanding what behaviors outside of the primary
goal creates additional learnings and hypotheses.

17. 17
Scoring Your Teams
Team Exp Starts Exp. Score Sig. Score Aud. Score Var. Score Day Score TOTAL
Team 1 22 4 4 1 4 3 16
Team 2 7 2 4 2 4 4 16
Team 3 33 4 3 2 3 1 13
Team 4 137 4 2 1 1 4 12
Team 5 4 2 2 2 2 2 10
Team 6 23 4 2 1 1 1 9
Team 7 18 3 2 1 1 1 8

18. 18
Optimizely Program OKRs
To view all of Optimizely’s Experimentation OKRs visit Medium.

19. 19
Education
Program
Measurement
Workflow
Coaching & Enablement
Community
Pillars of a successful experimentation program

20. 20

21. 21
Education
Program
Measurement
Workflow
Coaching & Enablement
Community
Pillars of a successful experimentation program

22. 22
What Can’t We Skip?
Table Stakes for Building a Community
1. Push out your launches and learnings to the rest
of the organization:
○ Email
○ Slack
2. “Pull” people into your history with an organized
way to find past learnings:
○ Confluence
○ Blog
3. Always have a touchpoint for the team to review
what is going out and what is going on!

23. 23
How Can We Do to Increase Participation?
Experimentation Day
● Program review - how are we doing against our
metrics?
● Program share - share with the rest of the
company on your successes and learnings
● Technology changes - what are we doing to allow
more complex and easier testing
● Industry review - what have we seen work in the
industry and how does it apply to us

24. 24
Program Examples

25. 25
Center of Excellence
BU 1 BU 2 BU 2 BU 3
Enablement
Program Measurement
Best Practices
Community
Ideation & Prioritization
Design
Build & QA
Results Analysis

26. 26
PM 1
PM 2
PM 3
PM 4
Testing Council
Meets Weekly for Experiment Review
Analytics
Manages Council and Execution
Ideation & Prioritization
Design
Build & QA
Results Analysis
Best Practices
Community

27. 27
Testing Council
PMs receive general UX tests from Global
Markets discuss upcoming geo-tests
Ideation & Prioritization
Design (UX tests)
Build & QA (UX tests)
Results Analysis & Sharing
US
AU
CA
DE
Ideation & Prioritization
Design (geo tests)
Build & QA (geo tests)
LAUNCH

28. Center of Excellence
Full Stack Testing Council Web Testing Council
PM 1 PM 2
PM 3 PM 4
PM 5 PM 6
Digital Team 1 Digital Team 2
Digital Team 3 Digital Team 4
Digital Team 5 Digital Team 6
Communication via Slack channels, Program
Management, and bi-weekly huddles

29. 29
Where do responsibilities lie across the testing lifecycle?
Stage FS Testing Council Web Testing Council Individual Teams Center of Excellence
Ideate
Shares ideas on other
teams’ backlog
Shares ideas on other
teams’ backlog
Creates ideas for backlog
Encourage best practices
and bring research
Prioritize Discusses overlaps Discusses overlaps
Determines main criteria and
order
Weighs in on overlaps
between Web and FS
Design Creates plan and assets
Reviews test plans and
establishes practice
Build Configures in Optimizely
QA & Launch Reviews and launches
Analyze Determines actions Determines actions Determines success
Share Results Shares with CoE Shares with CoE Shares with Council
Aggregate learnings and
shares with organization

30. 30
Optimizely’s Maturity Model

31. 31

32. BUSINESS VALUE
VELOCITY/VOLUME
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-Functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
Optimizely’s Maturity Model

33. Optimizely’s Maturity Model
BUSINESS VALUE
VELOCITY/VOLUME
LEVEL 1
31%
LEVEL 2
48%
LEVEL 3
20%
LEVEL 4 & 5
1%
Date Range: 2018-2019
N = 2,500+ companies

34. 34
What do Level 4 and 5 programs really focus on?
● Program Metrics - Measure inefficiencies in the program’s output
● Experiment Policies - Institute decision making practices
● ROI Analysis - Understand long term impact of experiments
● Education Program - Create ongoing learning opportunities from
industry
● OKRs - Make experimentation part of individual team member’s goals

35. 35
So what are the takeaways?
Dedicate a program manager and key roles
It’s better to start with something than
something perfect
Measure your program for inefficiencies

36. 36
How does active users impact velocity and stat sig rate?
As customers bring more
people into their program
and Optimizely, they:
● Increase the number
of experiments
started
● Increase share of
statistically
significant
experiments

37. 37
Questions?

38. 38
Thank you!