At Booking.com we have been conducting evidenced based product development using online experiments for more than ten years. Our methods and infrastructure were designed from their inception to reflect Booking.com culture, that is, with democratization and decentralization of experimentation and decision making in mind. In this talk, based on this paper with the same title, we explain how our approach has allowed such a large organization as Booking.com to truly and successfully democratize experimentation.

1. Democratizing online controlled
experiments at Booking.com
Lukas Vermeer | Senior Product Owner for Experimentation

2. TL;DR.
Central repository of successes and failures
Descriptions of all experiment iterations and of the
final decision are available for all experimenters.
Genericity and extensibility
Experimental design is abstracted away. Reports
are automated and product agnostic.
Data which can be trusted
We monitor the validity of the data by computing
common metrics in two separate date pipelines.
Loose coupling
Business logic and experiment infrastructure are
purposefully kept decoupled.
Building safeguards
We encourage sound methodology and provide
data quality checks, but no rules or automation.

3. Focus.

4. This is a story
about culture

5. This is a story
about people

6. 1. Ask PR department for “pretty photos of
employees”.
2. Filter out non-experiment roles (dba,
support, management, etc).
3. Select six at random from the remainder.
4. Email them and ask “how do you use
experiments in your job”.
5. Copy paste replies onto slides.
6. Highlight key phrases.
7. Sprinkle in some context.
Method.
Sample
Survey
Results

7. Lukas Vermeer.
Senior Product Owner
“I help hundreds of people
run experiments to make
Booking.com better.”

8. Founded in 1996
1.600.000+ properties
1.500.000+ room nights per day
15.500+ employees
1.000+ concurrent tests
Everything is a test
Booking.

9. Empower people to
experience the world.

10. Clyde Li.
Client Side Developer
“Data-driven is quite
common these days in tech
industry, however,
empowering everyone to
make data-driven decisions
independently is quite
unique in Booking.com.”

11. Nekeia Boone.
Senior UX Copywriter
“I can come up with an idea
over breakfast, bike to the
office and have it live well
before lunch. I’ve never
worked anywhere else that
gives me this level of
ownership and creative
freedom to validate my
ideas.”

12. Heloisa Biagi.
Client Side Developer
“I think it's great that the
company encourages
everyone to hack and test
ideas among the users. No
higher value opinions, no
centralization of decision
making, everyone is free to
have their own ideas and at
the end of the day, users are
the ones who decide what's
best for them.”

13. You cannot have a democracy
without empowering the people

14. Hadeer Younis.
Full-Stack Software Developer
“Experimentation is a great
way to figure out if
something as small as a
copy change or something
as big as a whole book
process flow will help users,
but you can’t depend on it
as your only source of
product validation as this will
greatly hinder the product
development.”

15. “The plural of
anecdote is not data”
- Lots of misinformed speakers at conferences I’ve attended

16. “The plural of
anecdote is not data”
- Raymond Wolfinger (1969)

17. GreyBlue OrangeWhite
80%60% 70%50%
Data is just data.
To make good decisions, we
need good evidence.

18. A B
Website Optimisation.
Let’s change the button from yellow to blue and see if it increases the magic number.
Buy now! Buy now!

19. A B
Hypothesis Testing.
We observed in user research that some people have difficulty finding the “buy now”
button. We suspect this is caused by the low contrast between the font and the
background. To solve this user issue, we will change the button from yellow to blue. If this
solution works, we expect to see more users hover and click, and eventually purchase.
Buy now! Buy now!

20. Evidence-based
customer-centric
product development

21. Finn Hansen.
Product Owner
“We use experimentation to
help us validate hypotheses
with the goal of addressing
well defined user problems.
It's all about learning as fast
as possible in the most
rigorous way possible.”

22. Based on [prior] we believe [condition] for
[users] will encourage them to [behavior]
We will know this when we see [effects]
happen to [metrics]
This will be good for customers, partners
and our business because [motivation].
Hypothesis template.
Theory
Validation
Objective

23. Diogo Antunes.
Principal Developer & Fellow
“We work in a global scale.
By validating my changes
through experimentation I
continuously get challenged
on my assumptions and the
way I look into the
experience of our
customers.”

24. 9/10 tests fail.
“90% of product decisions have an
inconclusive or negative effect on a
product’s primary metric.”
- VWO (2016)
90%
10%

26. Disclaimer: not what the actual experiment looked like.

27. Take the biggest small step
so you can challenge your
riskiest assumptions quickly

28. The order of items can be improved.
The order of items matters to users.
The feature can be improved.
The feature matters to users.
The page can be improved.
What is on the page matters to users.
Assumptions.
Order
Feature
Page

29. Evidence-based
customer-centric
product development

30. This is a story
about people

31. Empower people to
experience the world.

32. Thank you!
www.workingatbooking.com