1. Multivariate Testing at Orange
Driving more revenue online
Greg Kelton
Managing Director, Autonomy Optimost EMEA
Interwoven Confidential

2. Quiz!
Why Multivariate Conversion Analysis
Case Studies (Orange, Delta, etc.)
How it works
Q&A
Slide 2 2

3. 1
A
A
Control
B
B
+75%
3

4. Generating a 75% increase in Clickthroughs
• 14 template permutations analyzed
Control (creative 2) Winner (creative 11)
UAR-Shop = x% UAR-Shop = x% +75%
4

5. 2 A
A
+6.97%
B
B
Control
5

6. Generating a 53.42% lift (99% confidence)
Total Recipes Analysed: 2,938
B4
F=1
Control generating a 53.42% lift
(99% confidence)
6

7. 3
Flash (animated)
A
A B
B
Control +4.1%
7

8. Winning creative beat control by 10.0% on UAR-Congratulations
and 8.2% on UAR-Profile Information
Over 1 billion template permutations analyzed
Control (creative 121) Winner (creative 271)
UAR-Congratulations = x% UAR-Congratulations = x% +10.0%
UAR-Profile Information = x% UAR-Profile Information = x% +8.2%
8

9. Who should control your marketing?
Shareholders/
Shareholders/
Investors
Investors
Product
Product Marketing
Marketing
Managers
Managers Dept
Dept
Creative
Creative
CEO
CEO Agencies
Agencies
Executives
Executives Analysts
Analysts
11

10. Who should control your marketing?
Shareholders/
Shareholders/
Investors
Investors
Product
Product Marketing
Marketing
Managers
Managers Dept
Dept
None of the
above Creative
Creative
CEO
CEO Agencies
Agencies
Executives
Executives Analysts
Analysts
12

11. Who should control your marketing?
Your marketing should be driven by your customers
Your
Your
Customers
Customers
13

12. 14

13. Quiz!
Why Multivariate Conversion Analysis
Case Studies (Orange, Delta, etc.)
How it works
Q&A
Slide 15 15

14. The Content Selection Process is Complex & Faulty
How do you make web content decisions?
• Speculation
• HiPPO
• Intuition
• Agency experience
• Argument
• Quasi-objective
• Focus groups
What if Your Customers • Usability testing
Could Tell You What • Web Surveys
Content They Like?
16

15. 17

16. Old Facebook
New Facebook
18

17. Top facebook Applications See 25
Percent Drop In Traffic Since
Redesign
facebook Poll: 94% Of Users
Don’t Like Redesign
19

18. 20

19. 22

20. 23

21. 24

22. 25

23. Quiz!
Why Multivariate Conversion Analysis
Case Studies (Orange, Delta, etc.)
How it works
Q&A
Slide 29 29

24. But…. What’s the point?
• Your online competitive advantage
• Your Sales Funnel:
• Ease of buying & customer experience?
• Multi-channel influences?
• Technology investment (Flash, AJAX, Adobe AIR?)
• Your Operational Advantage:
• Change management?
• Cost reduction?
• Putting business in control of your site?
• Speed to market?
• Your Brand & Offering:
• Regional customers
• Price sensitivity? By region? By demographic?
30

25. Affiliate Marketing Page
Simplify the Buying Process
• Simple version increased
conversion by 3.86%
32

26. Putting business in control of your online channel
Merchandising Challenge - Brands, price
brackets, colours vary depending on product
category
•How many price brackets?
•Should the number of products in stock be
displayed?
•How many brands?
•Which order is best?
•Are collapsible menus preferred?
Benefit –understand how your customers prefer to
interact with site technologies eg. reviews, search,
navigation, shopping carts.
33

27. Shop Home Page
35

28. Shop Home Page
C
A, B
36

29. 30 total Page Recipes analysed
Quiz!
B Control
A +17%
39

30. Product Page
40

31. Product Page
41

32. 60 total Page Recipes analysed
Quiz!
A -9% B +13%
43

33. Quiz!
Why Multivariate Conversion Analysis
Case Studies (Orange, Delta, etc.)
How it works
Q&A
Slide 47 47

34. 1. Pick a Page to Test
48

35. 2. Add Single Javascript Callout to Page
<script
optimost.displayModule("main");
</script>
49

36. 3. Brainstorm ideas to test
1
2
3
vs
None
4
51

37. 4. Optimost Generates Page Recipes
52

38. 5. Run the Experiment Live on your site!
53

39. 6. Optimost declares the Best Recipe
This Version Increases
Original Version
Conversions by 28.4%
54

40. 7. REPEAT
55

41. Summary – MVT Allows you to:
Listen to your Customers
Improve Revenue… consistently
Avoid Cost and Risk of Redesigns
57

42. Thank You!
Greg Kelton
gkelton@interwoven.com
58