1. Online experimentation

2. ● Introductions
● What is online experimentation
● Methodology
● Metrics
● Showcase of wins & losses
● Q&A
● My email - tshurrock@opentable.com
Agenda
Online experimentation

3. 3
Introductions

4. ● First launch in 1998
● San Francisco HQ
● We have both restaurant facing and
customer facing products
● Acquired by The Priceline Group in 2014
● 40,000 restaurants worldwide
● 21 Million diners seated a month
● 850,000 reviews / month from our
customers
About Opentable
INTRODUCTIONS

5. ● Senior Product Manager at Opentable
● Focused on international expansion
● Development background
● Agency side and client side experience
About me
INTRODUCTIONS

6. CONFIDENTIAL
We want employees who continually challenge
themselves to look at things differently, seek a better
way, and know they have the freedom to do so.
Overview

7. What is online
experimentation?

8. CONFIDENTIAL
An experiment is a controlled study used to test a
hypothesis
Experimentation helps us understand the
impact of our ideas
Overview
A hypothesis is a proposed explanation
made on the basis of limited evidence as
a starting point for further investigation.
In controlled study, subjects are assigned
randomly to different treatment groups, if we see
meaningful differences in the outcomes, then we
can conclude that the treatment caused the
difference

9. CONFIDENTIAL
Overview

10. Experimentation methodology

11. 1. Science!
2. Hypothesise!
3. Minimum run time
4. Statistical significance
5. Validate your experiment
6. Learnings
DO..
.

12. 1. Stop the test mid 7 day cycle
2. Pause the test early because of a negative result
3. Judge your test too early
4. Leave it running because it’s ‘almost there’
5. Massage the results (aka the Texas sharpshooter
fallacy)
DO
NOT!

13. Experimentation is not just UI changes
● Meta analysis completed by Qubit Digital on over 6,700 experiments
● Cosmetic changes have far smaller impact on revenue than treatments grounded in
behavioural psychology.
Experimentation methodology
Treatment Avg Uplift Uplift probability
Scarcity 2.9% 84%
Social proof 2.3% 82%
Urgency 1.5% 70%

14. Experimentation metrics

15. 140-200 A/B tests launched per
month

16. KEY METRICS
● #hashtags are used to categorise experiments
● Allows for the easy reporting on experiment
type
● #socialproof / #global / #UX are the biggest
winners for us
● #scarcity is one of the lowest
Use of hashtags

17. KEY METRICS
Monthly tests launched

18. Showcasing of A/B testing
wins

19. #hashtags
#conversion
#socialproof
#urgency
#yield
#frequency
#crosssell

20. #conversion

21. 21
● Machine learning search results
● Positive result
● Ran for 14 days
● 99% confidence
● Using restaurant attributes to create
quality score and rank accordingly
● Search count, average rating, rating
count, reservation count, price, distance
from city centre
#CONVERSION

22. 22
● Free text search
● Positive result
● 98% significant
● 7 days
● Great insight into what customers are
searching for
#CONVERSION

23. #socialproof

24. ● Social Proof in rows for dine again on the metro homepage
● Positive improvement
● Ran for 14 days
● 92% confidence
#SOCIALPROOF

25. 25
● Showing top dish tags and
top restaurant tags above
the fold on a restaurant
profile page
● Positive improvement
● Information collected while
customers leave reviews of
the restaurant
#SOCIALPROOF

26. 26
● Showing instagram photos on
restaurant profile pages
● Positive improvement
● Internal tool for getting images
● Moderation feature
● Dynamically moderate photos soon!
#SOCIALPROOF

27. #urgency

28. #URGENCY
● Changing countdown timer text to
‘Hurry! We can only hold this table
for you for XX:XX’
● Positive improvement
● 95% confidence level
● Other than this, limited wins with
urgency

29. #frequency

30. 30
● Adding how many points
customer have received when
they made a booking to the past
reservations tab
● Positive improvement
● Ran for 3 weeks (due to low traffic
numbers)
● Increased repeat booking rate
#FREQUENCY

31. #crosssell

32. 32
● Restaurant profile page when
there is no availability for the
timeslot searched for (including
2.5 hours either side)
● Positive improvement
● Ran for 2 months (to get big
enough sample pool)
● 97% significant
#CROSS-SELL

33. 33
● A lot of our customers land on
restaurant profile pages from
Google.
● Adding ‘Find Nearby Restaurants’
button on the bottom of the profile
page
● Positive improvement
#CROSS-SELL

34. Losses (#fail)

35. 35
#FAIL
● More timeslots do not mean more
bookings
● Paradox of choice!
● Negative impact
● 99% significant

36. 36
#FAIL
● Showing images higher up the
page does not help the customer
make a decision quicker
● Negative impact
● 99% significant

37. Further reading

38. ● Influence: The Psychology of Persuasion
- Robert Cialdini
● Pre-Suasion: A Revolutionary Way to
Influence and Persuade - Robert Cialdini
● What works in e-commerce - a meta-
analysis of 6700 online experiments -
http://www.qubit.com
Things to read
Further reference reading list

39. ● You now (hopefully) have a good overview of how
Opentable executes online experimentation
● My email - tshurrock@opentable.com
● Twitter - @shurrock
Recap for the session
A/B TESTING APPROACH AND LEARNINGS

40. Questions?