Interested in learning how Optimizely’s latest innovations can help to amplify your testing program? Learn more about how our advanced features and functionality can enable you to uncover breakthrough customer experiences and drive ROI. You will walk away with a better understanding of the advanced capabilities of the platform and tangible ways to bring your experimentation program to the next level.

2. Supercharge Your Testing
Program
Whelan Boyd
Product Manager, Optimizely
Christine Garvey
Manager, eCommerce Testing &
Optimization, Crate and Barrel

3. D E S I G N

4. B U I L DD E S I G N

5. D E S I G N L A U N C HB U I L D

6. P R A YL A U N C HB U I L DD E S I G N

8. B U I L DD E S I G N

9. D E S I G N B U I L D
E X P E R I M E N
T

10. D E S I G N B U I L D
E X P E R I M E N
T
I T E R A T E

11. The New Way:
Learn Through Experimentation

12. :
Search Results Page
Scenario

13. Change to a wine glass example

14. Search Results Page
Updates
1. List vs Grid view
2. Ranking Algorithm
3. Style & Layout
Scenario

15. Change to a wine glass example

16. Change to a wine glass example

17. Challenge #1:
Choose the Right Metrics
Step 1:
Business
Alignment

18. We’re the only website on earth that tries to get rid of
our users as quickly as possible.
“
“

19. • Put yourself in the user’s
shoes
• Ask: What if this metric
went up & nothing else
happened?
• Constantly re-evaluate
Tips for
Choosing the
Right Metrics

20. Example
User
Retention:
33%
Revenue:
$2.0m
Revenue Per
Visitor:
$1.00
Visitors:
2.0m
Average
Order Value:
$50.00
Average
Quantity:
2.00
Conversion
Rate:
2.00%
User
Acquisition:
1.5m
Average per
Unit price:
$25.00
Add to Cart
Cart
Checkout
Rate

21. Example
User
Retention:
33%
Revenue:
$2.0m
Revenue Per
Visitor:
$1.00
Visitors:
2.0m
Average
Order Value:
$50.00
Average
Quantity:
2.00
Conversion
Rate:
2.00%
User
Acquisition:
1.5m
Average per
Unit price:
$25.00
Add to Cart
Cart
Checkout
Rate

22. Challenge #2:
Thinking Too Small
Step 2: Ideation

23. Change to a wine glass example

24. The problem with ”A/B testing”

25. The problem with ”A/B testing”

26. 2 Variations
A B
3 Variations
C
4 Variations
5 Variations
A B
>6 Variations
C D E F

27. Three-fourths of all experiments
only have 2 variations
2 Variations
3 Variations
4 Variations
5 Variations
77%
14%
5%
2%
1%>6 Variations

28. +71%
+48%
+32%
+75%
Testing 5 or more variations can improve
your win rate by 75%
25%
33%
37%
44%
43%
2 Variations
3 Variations
4 Variations
5 Variations
>6 Variations
Significant uplift
Significant reduction
Inconclusive

29. Change to a wine glass example

31. Every day, we run a thousand concurrent
experiments to quickly validate new ideas. These
experiments run across all our products, from mobile
apps and tools used by hoteliers to customer service
phone lines and internal systems. Experimentation has
become so ingrained in Booking.com culture that every
change, from entire redesigns and infrastructure
changes to bug fixes, is wrapped in an experiment.
“
“

32. Challenge #3:
Getting fast, but reliable,
results
Step 3: Experiment
Setup and
Analysis

35. Challenge #4:
A Feature or Design is
Never DONE
Step 4: Iterate,
Iterate, Iterate

36. Change to a wine glass example

39. Challenge #5:
Don’t Hoard Insights
Step 5:
Collaborate &
Learn Together

41. UPDATE WITH C&B IDEAS AND ANIMATIONS

42. ADD ANIMATION WITH IDEA DETAILS &
COMMENTS FOR C&B

44. UPDATE WITH C&B IDEAS AND ANIMATIONS

45. Share
Results
Document
Learnings
Communicate
Insights
Don’t Hoard Results

46. Scaling an Experimentation Program

47. Christine Garvey - Manager, eCommerce Testing & Optimization
Supercharging Our
Testing Program

48. 1. Quick Background
2. Launching a Testing
Program & Setting Best
Practices
3. Scaling Testing &
Advancing the Program
Agenda
eCommerce Strategy and Analytics

49. Bio
• 12 Years of Marketing &
Media experience
• MBA - Northwestern
University, Kellogg
School of Management
• Launched inaugural
Testing Teams at both
Allstate and
Crate and Barrel
eCommerce Strategy and Analytics

50. Establish the Testing Process
Brand Team
Product
Managers
Leadership
Christine
Set up meeting with key groups
(UX, PM, Dev if needed)
Fill out Test Intake Form
Analytics team to run sizing for
test opportunity
Marketing
Consider resources, timing, any
overlap with other tests, etc.
Consider business case for test:
● Will this increase sales?
● Would we implement this?
● Have we done this?
Start design work, if needed.
Ensure test details logged in
test intake form (objective,
success metrics, etc.)
Build and QA test draft (design
and functionality) before
launching test
1.
2.
3.
4.
5.
6.
7.
If all aligned in moving forward
with test...go to step 5
eCommerce Strategy and Analytics
Intake Process

51. Talk about the testing program in multiple
different channels
Bi-Weekly Test E-mail Update Test Summary Deck
Executive: One-Page Summary
(included in Monthly Leadership Meetings)
Bi-Weekly Testing &
Personalization Meetings
eCommerce Strategy and Analytics

52. eCommerce Strategy and Analytics
Test 1
Increase CVR/Sales
PersonalizationMessaging
Test 2 Test 1 Test 2 Test 3
Goal
Strategies
Tactics
Search Enhancements Redesigns
Test 3
Set Your Goals, Strategies & Tactics

53. eCommerce Strategy and Analytics
Test 1
Increase CVR/Sales
PersonalizationMessaging
Test 2 Test 1 Test 2 Test 3
Goal
Strategies
Tactics
Search Enhancements Redesigns
Test 3
Identify & Continue Successful Strategies

54. eCommerce Strategy and Analytics
Goal
Strategies
Tactics
Increase Scale and Run More Advanced
Tests
1. Identify always-on strategies
2. Try other ways to increase scale
Increase CVR/Sales
Expanding scale &
Digital Channels
Always On
HP & Super
Personalization
Product
Recs
App Personalized
PNs
Audiences
Identifying more of
“unknown”
Banners
Feature
Enhancements

55. Baby & Kids Super Personalization: Activating Machine Learning for Baby Audience
Predicted Intent - Baby Audience is showing increases in CVR and RPV vs. holdout
eCommerce Strategy and Analytics
Kids Content Baby Content

56. 1. Establish a testing
process
2. Talk about it!
3. Set goals & identify
winning
strategies….more
testing heroes will follow
4. Pursue opportunities
that increase reach
Supercharging
YOUR Testing
Program

57. Q&A

58. Thank you!