My slides from Emerce Conversion where I share Euroflorists first steps into an optimization process with evolutionary artificial intelligence algorithms.

1. How Euroflorist is preparing for AI:
Changes to our optimization strategies.
euroflorist.com

2. There will be Data (& GIFs)

3. How can we make the next step?

4. How can we maximize output
(in terms of learnings and conversion uplift)
with fixed resources?

5. Faster optimization loops!

6. Conversion Optimization is hard!

7. Optimizing for
“the average user”
Who is (s)he anyway?

8. Need more resources!
(but who doesn’t…?)

9. Too many variables impacting conversion!
● Customer persona’s
● Countries
● Budgets
● Customer history
● Regulations
● Colors
● Flower seasons
● Product types
● Holiday peaks
● Time of day/week
● Purchasing power
● Culture
● Devices
● Social media platforms
● Acquisition channels
● Symbols
● Step in Customer Journey
● Tech adoption level
● Accessoires
● Multi-channel funnels
● Etiquette
● Consumer vs Corporate
● Platform
● Different payment systems
● Brand awareness/perception
● Logistic situation
● Aesthetics
● Traditions
● Social context
● Weather
● Competition campaigns
● Consumer state-of-mind
● Competitor A/B test
● ...

10. And to make it even worse...

11. These variables keep changing
all the ffing time!

13. Let’s take a step back...

14. Why does Euroflorist exist?

15. We help people to make others
happy, noticed or remembered
c

16. Who are we?

17. Who is Euroflorist?
● We sell flowers & others gifts
● We’ve been doing this since 1947
● We do B2C & B2B, online and offline
● We are a Swedish company
● Our Dutch office is located in Amsterdam

18. Euroflorist in numbers (1/2)
● > 21 years in e-commerce
● > 60% revenue from online
● > 200 employees: 50% in CS, 35
working in online
● > 2M bouquets sent every year
● 19 sites in 11 countries and 8 languages

19. Euroflorist in numbers (2/2)
● 10M+ users on our sites in 2017
● Sites with up to 20% conversion rate
● Market share ranging from 5-45%
● 8+ score on Trustpilot, NPS > 50
● Growing revenue, teams and taking
market share

20. Guido Jansen
● Chief Psychology Officer @ Euroflorist
● UX/CX Team Lead
● Full Customer Journey optimization
● Wannabee Data Scientist
x@gui.do | @guido

21. What are we doing now (with CRO)?

22. Customer journey teams
● Branding/Marketing
● Traffic
● Landing pages
● PDP
● Product
● Checkout
● Post-order experience
● Delivery
● Customer Support
● E-mail / Retention
1 Business owner + 1 UX/CX team member

23. Live Testing
Complete/Analysis
Deployments
Design
Development
Q&A
Pending approval
Approved
Idea
Hypotheses
Full testing plan
Our optimization loop

26. How can we make the next step?

27. How are others doing this?

28. “[Booking.com’s] utilization of A/B testing …
drives higher conversion across it’s entire
platform, resulting in conversion levels 2-3x
the industry average”
Source: Evercore Equity Research

29. Sadly, when you test more,
the % of tests with an uplift will go down.

30. When A/B testing a fully optimized site,
the % of tests with an uplift is low(er).

31. Euroflorist: 1 in 5 tests with uplift
Booking.com: 1 in 10 tests with uplift
Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017

32. How to get more tests with an uplift?

33. Euroflorist: 10-20 concurrent test
Booking.com: 1000 concurrent tests
Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017

34. Euroflorist: 1 optimization team
Booking.com: 75 product teams
Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017

35. Of course, we are not Booking.com...
!=

36. ...but we do know a thing (or 2) about
experiments...

37. ...so let’s experiment with
the way we experiment!

38. Let’s start looking for a solution...

39. Evolutionary AI?
Tell me more!

40. Possible applications

41. Different applications of AI in E-commerce
● Catalog optimization ← already tried this
● Personalization
● RTB / Advertising
● Chatbots
● Virtual personal shopper
● Improved optimization process
● ...

42. We tried AI catalog optimization

43. Previous experience:
● “Black box”
● No (perceived) control for E-commerce managers
● No proven uplift in CR
● Hard to do for (emotional) consumers buying for
someone else?

44. The AI promise...
● Faster & more efficient
● Time-saving
● Test more
● Adaptive
● Greater impact
● ...

45. Sentient

46. “Fractional Factorial MVT + Evolution”

47. MVT...

48. … + Evolution

49. How the “evolution” part works...
+ 0 - + - + 0 + +

50. How the “evolution” part works...
+ 0 - + - + 0 + +

51. How the “evolution” part works...

52. How the “evolution” part works...

53. How the “evolution” part works...

54. How the “evolution” part works...

55. Always-on continuous optimization

56. Multipage Funnel Optimization
Home PDP Cart

57. Sentient Ascend
Tooling:
Implementation partner:

58. $143M
in total funding
9 YEARS
platform &
technology
development
35 PATENTS
submitted to date; 11 patents
issued
100+
EMPLOYEES
HQ in San Francisco with offices in San
Jose and Hong Kong
Founded by the team that developed
the core technology behind Siri

59. Sentient Ascend
DEEP
LEARNING BAYESIAN
EVOLUTIONARY
ALGORITHMS NEUROEVOLUTION

60. Tested in multiple industries
FINANCE
HEALTH
CARE
INSURANCE
CYBERSECURITY
AGRICULTURE

61. But does it work for E-commerce…?

62. Current brands experimenting with Sentient.ai ßeta

63. Sentient E-commerce ßeta @ Euroflorist

64. Countries selected for trial
The Netherlands Norway Sweden

65. 8 elements on PDP, 2 variants each
● Header - less content
● Image position - left to right
● Add progress bar - feed forward
● Social sentence (under CTA, Hobson’s choice +1)
● Product info - full width / less prominent
● Price partitioning
● USP Bar icons/text - different content
● USP Bar position - to the top

66. 28
= 256 variants

67. (1st) baseline active: end of Feb
Live: March 20th
End: June 3rd

68. NL
3 generations, 23 variants
Sample size
Full factorial MVT requires 1M users
Sentient tested 125K users

69. NL - Time Series

70. NL - Best 20 variants
Best Improvement over control: 2.5%-point (12.7%)

71. Significant?
Nope...
Too many variants/ Not enough visitors...

72. NO
4 generations, 28 variants
Sample size
Full factorial MVT requires 444K users
Sentient tested 75K users

73. NO - Time Series

74. NO - Best 20 variants
Best Improvement over control: 4.6%-point (12.6%)

75. Significant?
Nope...
Too many variants/ Not enough visitors...

76. SE
4 generations, 28 variants
Sample size
Full factorial MVT requires 1.2M users
Sentient tested 351K users

77. SE - Time Series

78. SE - Best 20 variants
Best Improvement over control: 2.8%-point (6.9%)

79. Significant?
Yes!
@ CI 95%, not @ CI 99%

80. Best variants
NL
Variant 8, generation 1
● USP Bar Icons/Text
SE
Variant 26, generation 4
● USP Bar Icons/Text
● Price perception
● Product info
● Call to action
● Progress Bar
NO
Variant 8, generation 1
● USP Bar Icons/Text

81. Best variant:
Common best element: USP Bar Icons/Text
Baseline:

82. But can we trust the numbers?

83. A/B test the winner to validate
Significant (frequentist; 90% confidence; 1-sided)
● Add-to-cart uplift: 3.9% (p-value 0.0161, Power: 95,9%)
● Conversion rate uplift: 5,3% (p-value 0.0361, Power 89,6%)
Not significant
● Average Order Value

84. How much does it add to our
bottom line?
€ € € € € € € € € €

85. Business case:
● Costs: 165K (subscription + support + salary)
● Expected uplift: 500K-1M profit
● Minimal CR uplift needed for break even: CR
increase of 1.25%

86. Final thoughts

87. ● Steered on CRT, not on RPU
● GA data issues
● Sentient dashboard UX issues
● Calculator to determine ideal values would be very helpful
Specific Sentient caveats/ needed improvements to ßeta

88. Regular A/B/MvT testing
Pro:
● More control
● Stricter hypothesis testing
● ...
Con:
● Slow
● Less variants
● Needs a lot of traffic
● Limited test capacity
● ...

89. Testing with AI
Pro:
● “Time per tested variant” much
shorter
● End up with “uncommon”
variation
● Requires less traffic (around
75% less)
● Higher ROI
● ...
Con:
● More extensive setup process
● Black box (?)
● Don’t know exactly how fast
generations will follow...
● Garbage in, Garbage out
● Completely freaks out our local
Ecom managers.
● ...

90. “I just followed instructions”
‘Compu’er says no’

93. Did Sentient deliver on their promise...?
MORE CAPABLE
• Test your entire funnel in a
single experiment
ADAPTIVE
• Evolves with your audience
• Always-on optimization
GREATER IMPACT
• More tests = better results
• Focus on marketing, not test
administration
FASTER AND
MORE EFFICIENT
• Tests dozens of ideas
• Test 1,000,000s of
combinations
• Get to results faster
TIME-SAVING
• Fully automated
• Eliminates tedious test
administration
• Handles the data science

94. Will this make the CRO team obsolete?

95. We’re hiring!
Optimization team:
● Web Analysts
● UX designers
● Growth hackers
● Interns
● AI/CKI background
● (Cognitive/Behavioral) Psychologists
Other online:
● SEA Specialist
● Marketing & communications
specialist
● CRM specialist
● Social Media Manager
● Interns
● ...
Are you smart & curious? Looking for a growing international company where you can
experiment & learn a lot? Join Euroflorist!
euroflorist.com

96. THX!
x@gui.do | @guido