4. 4
Housekeeping
● We are recording this webinar
● You’ll receive the recording and slides
● We’ll answer questions at the end
5. 5
“Our success is a function of
how many experiments we do
per year, per month, per
week, per day.”
- Jeff Bezos
“Our aim is to create the best
product for our customers, and
we do that through constant
innovation and testing.”
- Gillan Tans, CEO
“Our company culture
encourages experimentation
and free flow of ideas.”
- Larry Page
“We use experimentation and
testing to inform as much of the
business as we possibly can.”
- Gregory Peters, CPO
Today’s Digital Leaders Win By
Using Experimentation At-Scale
6. Spent with Digital Media
19hr
week
2018
Digital-Centric Customer
Spent with Digital Media
41
$396m
Mobile commerce Mobile commerce
$1 8t.
2008
Digital-Convenient Customer
hr
week
Your customer has fundamentally
changed how they engage
7. The digital-centric
customer journey: end-to-
end, multi-device
engagement
Marketing Product & Engineering Support
Views ad
at Checkout
Signs up for
7 day trial
Tracks
packages
Manages
account
Converts to
subscriber
Downloads
App
9. 9
Device coverage Complete
experience
coverage
Open platform
Trusted results
quickly
Consumer-grade
usability
Security &
compliance
Program
Management
Strategic & tactical
partnership
Build
Digital
Products
Market
& Sell
Service
& Retain
What you need
OTT
Voice/IOT
Mobile
Web / SPA
Build
Digital
Products
Market
& Sell
Service
& Retain
DEVICE COVERAGE
10. 10
Device coverage Complete
experience
coverage
Open platform
Trusted results
quickly
Consumer-grade
usability
Security &
compliance
Program
Management
Strategic & tactical
partnership
Build
Digital
Products
Market
& Sell
Service
& Retain
What you need
OTT
Voice/IOT
Mobile
Web / SPA
Build
Digital
Products
Market
& Sell
Service
& Retain
Frontend and
Server-side
Experimentation
& Personalization
EXPERIENCE COVERAGEDEVICE COVERAGE
11. 11
Device coverage Complete
experience
coverage
Open platform
Trusted results
quickly
Consumer-grade
usability
Security &
compliance
Program
Management
Strategic & tactical
partnership
Build
Digital
Products
Market
& Sell
Service
& Retain
What you need
Build
Digital
Products
Market
& Sell
Service
& Retain
TRUSTED RESULTS
Open Platform
Data &
Integrations
Security &
Compliance
OTT
Voice/IOT
Mobile
Web / SPA
Frontend and
Server-side
Experimentation
& Personalization
EXPERIENCE COVERAGEDEVICE COVERAGE
12. 12
Device coverage Complete
experience
coverage
Open platform
Trusted results
quickly
Consumer-grade
usability
Security &
compliance
Program
Management
Strategic & tactical
partnership
Build
Digital
Products
Market
& Sell
Service
& Retain
What you need
Build
Digital
Products
Market
& Sell
Service
& Retain
ENTERPRISE SCALE
Consumer grade
usability for
adoption &
velocity
Program
Management
Open Platform
Data &
Integrations
Security &
Compliance
OTT
Voice/IOT
Mobile
Web / SPA
Frontend and
Server-side
Experimentation
& Personalization
TRUSTED RESULTSEXPERIENCE COVERAGEDEVICE COVERAGE
13. 13
Device coverage Complete
experience
coverage
Open platform
Trusted results
quickly
Consumer-grade
usability
Security &
compliance
Program
Management
Strategic & tactical
partnership
Build
Digital
Products
Market
& Sell
Service
& Retain
What you need
Build
Digital
Products
Market
& Sell
Service
& Retain
Consumer grade
usability for
adoption &
velocity
Program
Management
Open Platform
Data &
Integrations
Security &
Compliance
OTT
Voice/IOT
Mobile
Web / SPA
Frontend and
Server-side
Experimentation
& Personalization
ENTERPRISE SCALETRUSTED RESULTSEXPERIENCE COVERAGEDEVICE COVERAGE
COLLECTDATA
UNIVERSAL
EXPERIENCES
ANALYTICS
ENTERPRISE
SCALE
15. DEVICE & EXPERIENCE COVERAGE
Optimizely Web & Full Stack
FULL STACK.
Test server-side logic &
data to deploy confidently
WEB.
Experiment on & deliver a winning
front-end experience
Copy
Images & Colors
Layout
i.e. booking flow, search bar
location & visual treatment
CTA
Navigation
Pricing or discounts
Personalized content
based on user
Segment offers
based on persona
Search algorithms
New feature
rollouts
16. DEVICE & EXPERIENCE COVERAGE
Universal
Experiences
Cross-Platform Metrics
Share conversions across frontend and
backend with universal attribution
Web Performance
Build experiments in a visual editor, deliver
them with lower latency
Consistent Features
Use Environments in Web projects. Track
Change History in Full Stack.
Full Stack Usability
Do more out-of-the-box with Full Stack.
Easier SDKs running in more places.
17. WEB + FULL STACK
Cross-Platform Metrics
● Reuse client-side event tracking with
server-side tests
● Measure how web experiments impact
downstream backend metrics
● Share event tracking across multiple Full
Stack projects
Share conversion events across Web
and Full Stack experiments
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Coming Soon
22. WEB
Performance Innovation
● Optimizing key platform components for
faster desktop & mobile execution
● Integrated new CDN delivery technology
for lower latency connections, using Full
Stack technology
● New implementation options to give you
full control over your site’s performance
We’re making Optimizely faster than
ever for the modern web
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Coming Soon
23. ● Build an experiment once, and push
through specialized environments
● QA experiments in dedicated environments
before launching to production
● Streamline developer work by setting up
environments in Optimizely that mirror your
own development workflows
Easily integrate experimentation into your
existing development & QA workflows
FULL STACK → WEB
Web Environments
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
DEV
TEST
STAGE
PROD
Coming Soon
24. WEB → FULL STACK
Change History
● Comprehensive records: see exactly who
made a change, what they edited, and
when
● Detailed audit log lets you investigate what
changed at a specific time
● Improved search and filtering
Track all changes made to experiments
and feature flags in your account
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Coming Soon
25. 25
The right experience is not
always the same for everyone
The right experience depends
on context
The right experience requires
continuous iteration
Experimentation
Personalization
Sweet Spot
Why You Need to Experiment to
Personalize Effectively
26. 26
Different experiences
resonate with different
individuals, and
personalization enables you
to ensure you find an
experience that resonates
with your customers ✓x x
Audience
Students
Page
Homepage
Everyone
Banner B
-2%
Students
Banner B
+19%
Students
Banner C
+19%
Variation 1
Students
Banner D
+25%
Variation 2
Students
Banner E
+10%
Variation 3
Audience
Everyone
Page
Homepage
Everyone
Banner A
+5%
Everyone
Banner B
-2%
The right experience
is not always the
same for everyone
27. ● Match users attributes with new number,
string, and boolean comparisons
● Easily adjust targeting conditions without
re-deploying new code
● Compose multiple audiences into new
segments
Target experiments and feature rollouts
using flexible rules and composable
segments
WEB → FULL STACK
Flexible Targeting
Released
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
28. ● Target users based on their predicted
intent, using natural language keywords
● Adapt to changing user behavior and
expectations automatically, without
needing to manually tweak rules
● Experiment your way to the best
personalization strategy - test larger or
smaller audiences, and multiple variations
Build predictive segments based on online
behavior, powered by machine learning
WEB
Adaptive Audiences
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Beta
29. BUILD
LEARN
MEASURE
WEB + FULL STACK
Product Experimentation
Think hypothetically
● Painted doors
● Validating MVPs
● Qualitative research
● Exploratory A/B testing
Quantify product impact
● Feature validation
● Multivariate tests
● Iterative adjustments
● Measuring business impact
Mitigate development risk
● Frequent, small releases
● Feature flagging
● Staged rollouts
● Safe rollbacks
30. FULL STACK
Optimizely Rollouts
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Early Access
● Reduce risk with staged feature rollouts and safe rollbacks
● Powered by the same SDKs as Full Stack. Grow into full
experimentation.
● Generally available next month. Contact your CSM if you’re
interested in early access!
www.optimizely.com/rollouts
Free feature flagging for product
development teams
31. FULL STACK
Developer Efficiency
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Coming Soon
● New languages: Swift, Go
● Datafile Management: automatically sync
configuration between Optimizely and your
application
● Event Dispatching: simplify event tracking
at enterprise scale
● User Profiles: manage user state and
variation history across devices
● New Core: easy access to your
configuration data, improved hooks for QA
and testing, more detailed log messages
Do more, faster. Our redesigned SDKs are
more powerful and easier to use
33. OUR VISION
Universal Experiences
DYNAMIC PRICING,
ALGORITHMS
ON OUR WAY
Your order has just shipped! Track it now.
NOTIFICATIONS,
CONVERSATIONAL
INTERFACES
ADVERTISING
Bringing
experimentation and
personalization to
the full spectrum of
digital experiences
35. 35
MISSION
Help the world’s innovators make
trusted decisions at scale
Experimentation Hero
10’s experiments / year
Experimentation Program
100’s experiments / year
Culture of Experimentation
1000’s experiments / year
36. ANALYTICS
Optimizely Analytics
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Results & Stats Engine
1. Optimize your experiences
and drive business value.
2. Learn how users respond
to your experiment.
3. Control the risk of
launched features.
Machine Learning
1. Accelerate learnings and
impact with ML-powered
Stats Accelerator.
2. Optimize rewards for a
period of time.
3. Identify statistically
significant variants faster.
Integrations
1. Integrate with best-of-breed
technology platforms.
2. Discover powerful ways to
target content.
3. Measure performance in the
tools you already use.
TODAY
Standard
Experiment
Experiment with
Stats Accelerator
Reach statistical significance up to 300% faster
37. ANALYTICS
Easy Event Tracking
Track and optimize every element of
the customer journey with Optimizely.
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Released
E
E
E
Website
Sign ups
Conversions
….
Payments
Revenue and
Refunds
BI
Lifetime Value
● Attribute visitor actions from any
system to your experiments.
● Optimize metrics across the customer
journey.
● Simplify data integrations with other
applications in your technology stack.
38. ANALYTICS
Easy Event Tracking | Customer Use Cases
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Released
E
E
E
E
cross-channel actions
offline conversions
computed metric data
external events
Easy Event
Attribution
(purchased in-store)
(e-mail opened)
(lifetime value)
(notification clicked)
39. ANALYTICS
Results Export
Access, analyze and visualize
experiment performance with
enhanced flexibility.
● Analyze experiment results with
any SQL or Data Science tool.
● Enrich your analysis with data
from your Data Warehouse.
● Monitor experiment performance
using your own dashboards.
Beta
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Reporting
Data Science
Data Warehouse
40. ANALYTICS
Results Export | Customer Use Cases
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Beta
Custom Analysis (SQL, Data Science/Mining...) Reporting Dashboards
41. ANALYTICS
Event Debugger
Inspect and debug your
experiment events faster.
● Verify in real-time that Optimizely
is receiving your events.
● Debug issues with your event
setup to accelerate your
experimentation velocity.
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Coming Soon
42. ANALYTICS
Extensible Analytics Platform
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Query
Data
Compute
Metrics
Statistical
Analysis
Display
Results
Metrics
Object
Results
Object
Data
Object
Experiment Analysis Model
External Data Custom Models Certified Reports
An Experiment Analysis platform designed to be extended and certified by Data Scientists...
… so that it’s used by everyone in the organization to make trusted decisions at scale
Future
EVENTS METRICS BUILDER STATS ENGINE RESULTS
EXTENSIBLE
INTERFACES
44. 10X YOUR PROGRAM
Enterprise Scale
AUTOMATED ORCHESTRATION
Use automation to
accelerate teams into
building high-velocity
experimentation programs.
CUSTOMIZED REPORTS
Easier socialization of an
experimentation through
custom reporting &
dashboards.
ENTERPRISE GOVERNANCE
Roles, controls, and
workflows that give large
teams the ability to scale
with confidence.
Govern, orchestrate, and report on your experience optimization program.
45. ENTERPRISE SCALE
Jira Integration
● Track development in Jira and
experiments from Optimizely with a
unified view across systems
● See the status of Optimizely rollouts
in Jira issues to reduce context
switching and friction in development
● See key information about
experiments and ideas right in Jira
Streamline your workflow by linking
Optimizely ideas, experiments and feature
flags to Jira issues
Released
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
46. ENTERPRISE SCALE
Slack App
● Integrate with Slack to ensure real-
time updates on progress of ideas
and experiments
● Users stay informed and can engaged
with experiments from outside of
Optimizely
● Improve experimentation
communication and collaboration and
increase testing velocity
Integrate Program Management with Slack,
enabling more effective and efficient
collaboration
Released
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
47.
48. Coming Soon
ENTERPRISE SCALE
Experiment Reports
● Easily and quickly report on experiment
results
● Acts as a system of record and cohesive
view of an experiment
● Helps improve prioritization and decision
making over time
Easily create a report summarizing key
details, results, and assets for your
experiments
Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
49. 49
Program Reporting
Core operational metrics like win rate
and velocity across your teams.
Experiment Reports
Shareable summary of your
experiments
ROI
Measure your team’s revenue
impact on your business.
TODAY FUTURE
Make it easier to share results and derive your return on investment in experimentation
ENTERPRISE SCALE
Reporting
50. 50
2019 Optimizely
ROADMAP
WEB & ✔ Support for Dynamic Websites Adaptive Audiences
PERSONALIZATION ✔ Enhanced Multivariate Testing Faster Performance
Environments
FULL STACK ✔ Feature Management ✔ Flexible Targeting
✔ Environments Change History
✔ Multi-language Experiments ✔ Fast Config Sync
✔ Custom Analytics Integrations
PLATFORM ✔ Stats Accelerator Enhanced Export
✔ GDPR ✔ Cross Product Events
✔ Easy Event Tracking Event Debugger
✔ Flexible Segmentation
PROGRAM ✔ Teams Experiment Reports
MANAGEMENT ✔ Program Reporting ✔ Slack App
✔ Jira Integration ROI Reporting
BETA
BETA
Talk to
your CSM
to learn more
Coming Soon
2019
Delivered
2018
54. 58
Easy Event Tracking (Data-In)
1. Track events from any system
into Optimizely.
2. Optimize metrics across the
customer journey.
3. Simplify event integrations with
other applications in your stack.
Results Export (Data-Out)
1. Analyze results with any
SQL or data science tool.
2. Enrich your analysis with
data from your DW.
3. Build dashboards using
tools you are familiar with.
THIS YEAR
This Year | Analytics
Released Beta
55. 59
Easy Event Tracking (Data-In)
1. Track events from any source.
2. Optimize metrics across the
customer journey.
3. Simplify event integrations with
other applications in your stack.
Results Export (Data-Out)
1. Analyze results with any
SQL tool.
2. Enrich your analysis with
external data from your
data warehouse.
3. Build dashboards using
tools you are familiar with.
Event Debugger
1. Verify that Optimizely is
receiving your events in real-
time.
2. Debug issues with your
events faster to accelerate
your experimentation
velocity.
THIS YEAR
Coming SoonBetaLaunched
This Year | Analytics
Hinweis der Redaktion
My name is Adam….
I’m excited to be joined by Olivia...
Olivia leads the digital testing for Wall Street Journal.com and Barrons.com. She’s helped inject testing into all aspects of the customer lifecycle, including acquisition, engagement, product, and retention. Her team has run over 500 experiments over the past 3 years and is continuing to find new ways drive the product roadmap and expand acquisition strategies.
A few housekeeping items for today:
We are recording this webinar
You’ll receive the recording and slides in a follow-up email
and we want today’s session to be interactive. So please ask questions. You can do so by entering them in the chat box of Go to webinar console.
Many of the world’s greatest companies have already broken the code on how to win in the new world order of digital. The answer for them is experimentation.
Companies like Amazon, Google, Facebook and others are all saying exactly the same thing. They have figured out the experimentation is the key to winning in an increasingly digital world.
Jeff Bezos at Amazon is widely quoted about how their success is directly tied to the number of experiments they run per day.
And as you can see here, very similar comments from Larry Page, and Mark Zuckerberg
And if that’s not enough, even the Harvard Business Review is weighing in. In a recent article called the ”Surprising Power of Online Experiments” the authors stated that “if you want to achieve a competitive advantage, you need to master the practice of online experiments.”
So, yes, we do believe experimentation is the next big thing in business management and is rapidly becoming a mainstream business best practice.
Let’s talk about why this is important now more than ever. These stats are to drive home that consumer behavior has fundamentally changed.
Ten years ago we bought physical CDs and DVDs, shopped in brick-mortar-stores, and if we needed to file an insurance claim, we called into a call center. Now all of that is digital. Digital has shifted from being a convenient way for consumers to engage brands to being the central way.
And this shift is not something just affecting marketing. Every single touchpoint - across web, mobile, voice - along the entire customer journey is an opportunity to win or lose your customer.
We see everyday more and more companies adopting digital experience optimization across all business functions because it is key to acquiring more users, driving more and deeper product engagement, and retaining more customers.
Complete coverage of digital channels so you can optimize everywhere your users are
The ability to optimize and personalize the entire experience from the front-end UI to server-side business logic and data
Trusted results so you can make sound decisions quickly
Consumer-grade usability to scale experimentation enterprise-wide
Protection of your user data through enterprise grade security & compliance
Efficient ways to govern and manage your optimization programs at scale - the more you test and learn, the faster you grow
A strategic partner with the hard-won best practices, knowledge and services to build with you into a world-class optimization program.
Complete coverage of digital channels so you can optimize everywhere your users are
The ability to optimize and personalize the entire experience from the front-end UI to server-side business logic and data
Trusted results so you can make sound decisions quickly
Consumer-grade usability to scale experimentation enterprise-wide
Protection of your user data through enterprise grade security & compliance
Efficient ways to govern and manage your optimization programs at scale - the more you test and learn, the faster you grow
A strategic partner with the hard-won best practices, knowledge and services to build with you into a world-class optimization program.
Complete coverage of digital channels so you can optimize everywhere your users are
The ability to optimize and personalize the entire experience from the front-end UI to server-side business logic and data
Trusted results so you can make sound decisions quickly
Consumer-grade usability to scale experimentation enterprise-wide
Protection of your user data through enterprise grade security & compliance
Efficient ways to govern and manage your optimization programs at scale - the more you test and learn, the faster you grow
A strategic partner with the hard-won best practices, knowledge and services to build with you into a world-class optimization program.
Complete coverage of digital channels so you can optimize everywhere your users are
The ability to optimize and personalize the entire experience from the front-end UI to server-side business logic and data
Trusted results so you can make sound decisions quickly
Consumer-grade usability to scale experimentation enterprise-wide
Protection of your user data through enterprise grade security & compliance
Efficient ways to govern and manage your optimization programs at scale - the more you test and learn, the faster you grow
A strategic partner with the hard-won best practices, knowledge and services to build with you into a world-class optimization program.
Complete coverage of digital channels so you can optimize everywhere your users are
The ability to optimize and personalize the entire experience from the front-end UI to server-side business logic and data
Trusted results so you can make sound decisions quickly
Consumer-grade usability to scale experimentation enterprise-wide
Protection of your user data through enterprise grade security & compliance
Efficient ways to govern and manage your optimization programs at scale - the more you test and learn, the faster you grow
A strategic partner with the hard-won best practices, knowledge and services to build with you into a world-class optimization program.
Optimizely has been the site experience laboratory for the better part of a decade. After examining thousands of tests and working with thousands of customers, providing a relevant experience based on previous behaviors and profile data tends to lead to a higher desired conversion rate. We have learned that the winning experience is not the same for everyone. The right experience really is dependent on context and that the right experience requires continuous iteration.
Those that have really excelled at this often have not gotten that conversion rate on their first try and the way that they have gotten this is by experimenting every step of the way, different audience definitions, different experiences/variations and ultimately treating every decision as a test
Let’s look a bit at what we mean when we say that the right experience is not always the same for everyone.
We see that different experiences resonate with different individuals, personalization enables you to ensure you are delivering experiences that resonate with your customers. Let’s walk through an example
+++Click
Here we start out with an A/B test that everyone sees. We can see that Banner B does not do well in this test.
+++Click
However, when we dig into this and look at segments, we can see that while Banner B didn’t resonate with everyone, it does resonate better than the alternative with students.
+++Click
From this information we then decide to experiment and apply an experimentation mindset to this banner, optimising for the student segment, and actually find that banner d is the winner (+++Click)
This is an example of how we can see that by applying an experimentation mindset, we are able to effectively prsonalize.
Trend : product experimentation
As we talk with companies like yours we find that they are seeking ways to accelerate product innovation to meet rising customer expectations, and reach the market faster in order to see immediate impact to revenue.
However, we also know that adding velocity greatly increases the associated risk. That’s why leading companies adopt processes like agile and devops—to mitigate risk while moving quickly.
And finally, these companies also experiment on everything, so they can know the impact of new features on key metrics before rolling out to everyone.
So what does the infrastructure that backs this process look like when implemented at scale?
It starts with feature flags, simple toggles that enable you to separate deploying code from enabling features for customers.
Then these teams do staged rollout of new features, to a small percent of traffic to ensure things work as expected, and can safely roll back if they don’t.
Features are also A/B tested to quantify impact on key business metrics, not just performance.
As these companies iterate, they perform many more A/B/n or multivariate tests to try out different variations on a feature or experience and tune to perfection.
Every feature goes through this process before launch. And it’s deceptively simple. There’s a lot under the hood that these companies have built, including:
Robust data infrastructure – statistical analysis, targeting
Tools to democratize testing across the product org – a usable interface for business users, support for many languages, documentation for onboarding more teams
Enterprise ready capabilities—is the system secure, compliant, and available when they need it?
And integrations to other systems that power the customer experience.
When you look at the multitude of discrete components that make up a robust experimentation stack, it's a daunting task to undertake.
As we talk with companies like yours we find that they are seeking ways to accelerate product innovation to meet rising customer expectations, and reach the market faster in order to see immediate impact to revenue.
However, we also know that adding velocity greatly increases the associated risk. That’s why leading companies adopt processes like agile and devops—to mitigate risk while moving quickly.
And finally, these companies also experiment on everything, so they can know the impact of new features on key metrics before rolling out to everyone.