uShip is an online marketplace that matches consumer and business shippers with transporters. Over the past year plus, they have been working to build their experimentation program, both from a product and marketing perspective. Through this focused program investment, uShip has increased their experiment velocity by over 450%, and has multiple product teams testing and using staged feature rollouts. All of this has minimized risk to product rollouts while ensuring customer adoption. It hasn’t always been easy, but they know that doubling down on experimentation is integral to their success. In this webinar, Jamy Squillace & Brooks Lyford from uShip will share how they started an experimentation program from scratch, gained stakeholder buy-in, and are building a culture of experimentation, focused on testing everywhere. Join us and learn: How to build an experimentation program from the ground up Best practices to balance product and client-side experimentation, leveraging Optimizely’s full platform How to socialize experimentation throughout the organization and begin creating a culture of experimentation

1. 1
Building a Culture Rooted in
Experimentation
Jamy Squillace, Brooks Lyford &
Timothy McDonough
How uShip built an experimentation program from the
ground up
+

2. 2
Jamy Squillace
Director of Product Management
Introductions
Brooks Lyford
Product Manager
Timothy McDonough
Senior Customer Success Manager

3. 3
Housekeeping
● We are recording this webinar
● You’ll receive the recording and slides
● We’ll answer all questions at the end

4. optimizely.com/digital-change-agents/

5. 5
Agenda
● Introduction
● About Optimizely
● uShip’s Journey
● Q&A

6. Revenue, share of wallet, funnel conversion, risk
mitigation, ops efficiency
What
We Do Next gen “Test and Learn” platform for
enterprise-wide digital experience optimization
Over 1B impressions daily
Replaces Digital Guesswork
With Evidence-based Optimization
Digital Experimentation
SaaS Platform
Apply the scientific method to “at-scale”
business decision making.
20X increase in Yield

7. “Our success is a function of
how many experiments we do
per year, per month, per week,
per day.”
Jeff Bezos
“Our aim is to create the best
product for our customers, and
we do that through constant
innovation and testing.”
Gillan Tans, CEO
“Our company culture
encourages experimentation
and free flow of ideas.”
Larry Page
“We use experimentation and
testing to inform as much of the
business as we possibly can” -
Gregory Peters, CPO
Today’s Digital Leaders Win By
Using Experimentation At-Scale

8. What
We Enable
1500+
+51%
+$21M
+19%
+28%
Improved subscription
acquisition
Users Scale and agility
of experiments; users
incremental
in revenue
Improved Driver
Acquisition
increase in cross-sell
performance

9. Experimentation: Testing variations of digital experiences
with real users and deploying what wins
+21%
-5%

10. Frontend UI Backend Business
Logic & Data
The anatomy of an experience
i.e. navigation,
search location
& visual treatment
Copy
Images &
Colors
Layout Search algorithms
Personalize content
based on previous
behavior
Recommendations
Make your headlines
more personal

11. Device coverage Complete
experience
coverage
Open platform
Trusted results
quickly
Consumer-grade
usability
Security &
compliance
Program
Management
Strategic & tactical
partnership
Build
Digital
Products
Market
& Sell
Service
& Retain
What you need
Marketing
Product &
Engineering
Support
The Optimizely Solution
Consumer Grade
Usability
Program
Management
Purpose- Built
Statistics Engine
& Machine
Learning
Open Platform
Data &
Integrations
Security &
Compliance
OTT
Voice/IOT
Mobile
Web / SPA
Frontend
Experimentation
& Personalization
Server-side
Experimentation
Best in class
Services &
Success Teams
SUCCESS ACCELERATIONEXPERIENCE COVERAGEDEVICE COVERAGE TRUSTED RESULTS ENTERPRISE SCALE

12. Success Acceleration
MATURITYANDCOVERAGE
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
Start Anywhere & Scale Up
To a Culture of Experimentation

13. We’re Proud to Work With Great Global Enterprises
24 of the Fortune 100 have chosen Optimizely to drive their digital experience

14. Building a Culture Rooted in Experimentation
How uShip built and experimentation program from the ground up

15. Objectives
How to build an experimentation program from the ground up
Best practices to balance product and client-side experimentation
Creating a culture of experimentation
2
1
3

16. A Brief History
2017 2018 2019
2017
Signed contract
for full stack
and web
NOV 2018
Engaged Optimizely to
partner with
us to reboot our
Experimentation
program
JAN/FEB 2018
Optimizely
full stack
implemented
FEB-DEC 2018
Ran a total of
6 A/B tests
SEP 2018
Attended
Opticon
JAN-MAR 2019
Ran 7 tests and
implemented
new program
Q2 2019
Continuing to
increase testing
velocity with all
development
teams

17. uShip Challenges
Initial implementation helped us to get the product implemented
but didn’t help us create a culture of experimentation
No owner,
process, or
vision
Main focus is
new B2B2C
product
Mythical
expectations
The Million Dollar A/B Test

18. uShip Challenges
Early tests were not run
to statistical significance
Simple understanding of
how to analyze results
was needed to ensure we
did not run futile test

20. Building a Program from the Ground Up

21. Ownership & Accountability
Product owns the
Experimentation Program
(getting buy-in from stakeholders)
Product is accountable to the
business on the state of the program
and reporting on the health of the
experimentation culture.
Testing commitment to the CEO:
6 tests in Q1 (same as what we did in ALL of 2018)
CTO
(Kris)
Director
of PM
(Jamy)
Product
Manager
(Brooks)
Tip #2: The owner needs goals/accountability.Tip #1: Whoever owns the roadmap should
own the program.

22. Become Experimentation Experts
Engagement with Optimizely
for education and support:
(tech and strategy)
Tip #3: Invest in the support you need and
include change management (if needed).
Tip #4: Don’t be afraid to throw out what
you have if it’s not working for you.
● Advisory & Learning
● 6 months of support
● Strategic services:
○ Program goals
○ Best practices
○ Experimentation frameworks
○ Ideation
○ Results analysis
● Technical Services:
○ Instrumentation
○ Implementation review and support

23. ● Shortest path to getting tests running
● Streamlined approval process
● Goal was to get tests running and not
worry about perfect hypotheses
● Failure (with learning) was okay
We removed ALL
roadblocks to testing
Tip #5: Don’t make it more complicated
than it needs to be.
Make it Easy

24. Leveraging Optimizely’s Full Platform

25. Optimizely Full Stack
Tip #6: Get the buy-in of your development
team. Show them the value.
Over 50% of our development teams have used Optimizely in some
way and we are on track to have all teams use either Experimentation
(A/B Tests of Feature Tests) or Feature Flags by the end of the year.
FEATURE
FLAGS
A/B
TESTS
FEATURE
TESTS

26. Our First Multivariate Test
Tested against our control, we wanted to see what copy would engage the most clicks, and which
destination would yield the highest conversion rate.

27. Optimizely Web
Optimizely Web has empowered our Marketing team to
run experiments without having to go through the full
development cycle and has doubled our testing output.
They are now able to dynamically test on our Marketing
and Landing Pages easily.
SOME CHALLENGES:
● Ensuring these quick and easy tests are QA’d and deployed to production safely
● Strategizing and prioritizing tests that can be launched with a very high velocity
● Optimizely the communication channels between Marketing and Development

28. uShip’s Culture of Experimentation

29. uShip Experimentation Program
New Experimentation Program
● Owned by Product
● Bi-weekly Experimentation Guild
● Track all experiments
Rolled out Experimentation Workshop
● Based on our learnings with Optimizely
● Geared towards creating tests aligned with business goals
● Inclusive of all departments, not just product teams
Socialized it throughout the organization
● Company All Hands
● Department All Hands
● Bi-weekly Guild meeting
Tip #7: Create a program that works for
you and your organization.
IS THAT
ALL?

30. We talked about it...ALL. THE. TIME.
Tip #8: Talk about it all the time. Socialize,
socialize, socialize.

31. uShip Experimentation Program - Future
All Product Managers
are building
Experimentation into
their roadmaps in
every quarter.
All teams have
used Optimizely
for Experimentation,
Feature Flags and
Rollouts.
Increase
Experimentation
engagement outside of
Marketing, Product
and Development.
Improve our rigor
around metrics,
analysis and follow-up.
Increase visibility of
the program through
an Experimentation
newsletter and posting
all experiments on a wall.

32. What We Have Learned
Good is better than perfect but it’s important for us that experiments are:
○ Data driven
○ Problem-focused
○ Will unleash learning regardless of the outcome
We needed a goal or challenge (even a random goal) to pick up our pace
Involving the entire organization leads to more engagement and higher velocity
A high testing velocity will reveal a realistic Minimum Detectable Effect (Improvement) very
quickly. We can start to understand our best opportunities to affect change on our platform
Not understanding a problem completely, or having differing thoughts on solutions, is the perfect
reason to start running an experiment

33. uShip Experimentation Program Tips
TIP #1
Whoever owns the
roadmap should own
the program.
TIP #2
The owner needs
goals/accountability.
TIP #3
Invest in the support
you need and include
change management
(if needed).
TIP #4
Don’t be afraid to
throw out what you
have if it’s not working
for you.
TIP #5
Don’t make it more
complicated than it
needs to be.
TIP #6
Get the buy-in of your
development team.
Show them the value.
TIP #7
Create a program that
works for you and
your organization.
TIP #8
Talk about it all the
time. Socialize,
socialize, socialize.

34. Questions