There is extensive literature about online controlled experiments, both on the statistical methods available to analyze experiment results as well as on the infrastructure built by several large scale Internet companies but also on the organizational challenges of embracing online experiments to inform product development. At Booking.com we have been conducting evidenced based product development using online experiments for more than ten years. Our methods and infrastructure were designed from their inception to reflect Booking.com culture, that is, with democratization and decentralization of experimentation and decision making in mind. In this presentation, Lukas will explain how building a central repository of successes and failures to allow for knowledge sharing, having a generic and extensible code library which enforces a loose coupling between experimentation and business logic, monitoring closely and transparently the quality and the reliability of the data gathering pipelines to build trust in the experimentation infrastructure, and putting in place safeguards to enable anyone to have end to end ownership of their experiments have allowed such a large organization as Booking.com to truly and successfully democratize experimentation.

1. Democratizing online controlled
experiments at Booking.com
Lukas Vermeer | Senior Product Owner for Experimentation

2. TL;DR.
Central repository of successes and failures
Descriptions of all experiment iterations and of the
final decision are available for all experimenters.
Genericity and extensibility
Experimental design is abstracted away. Reports
are automated and product agnostic.
Data which can be trusted
We monitor the validity of the data by computing
common metrics in two separate date pipelines.
Loose coupling
Business logic and experiment infrastructure are
purposefully kept decoupled.
Building safeguards
We encourage sound methodology and provide
data quality checks, but no rules or automation.

3. Focus.

4. This is a story
about culture

5. This is a story
about people

6. 1. Ask PR department for “pretty photos of
employees”.
2. Filter out non-experiment roles (dba,
support, management, etc).
3. Select six at random from the remainder.
4. Email them and ask “how do you use
experiments in your job”.
5. Copy paste replies onto slides.
6. Highlight key phrases.
7. Sprinkle in some context.
Method.
Sample
Survey
Results

7. Lukas Vermeer.
Senior Product Owner
“I help hundreds of people
run experiments to make
Booking.com better.”

8. Founded in 1996
1.600.000+ properties
1.500.000+ room nights per day
15.500+ employees
1.000+ concurrent tests
Everything is a test
Booking.

9. Empower people to
experience the world.

10. Clyde Li.
Client Side Developer
“Data-driven is quite
common these days in tech
industry, however,
empowering everyone to
make data-driven decisions
independently is quite
unique in Booking.com.”

11. Nekeia Boone.
Senior UX Copywriter
“I can come up with an idea
over breakfast, bike to the
office and have it live well
before lunch. I’ve never
worked anywhere else that
gives me this level of
ownership and creative
freedom to validate my
ideas.”

12. Heloisa Biagi.
Client Side Developer
“I think it's great that the
company encourages
everyone to hack and test
ideas among the users. No
higher value opinions, no
centralization of decision
making, everyone is free to
have their own ideas and at
the end of the day, users are
the ones who decide what's
best for them.”

13. You cannot have a democracy
without empowering the people

14. Hadeer Younis.
Full-Stack Software Developer
“Experimentation is a great
way to figure out if
something as small as a
copy change or something
as big as a whole book
process flow will help users,
but you can’t depend on it
as your only source of
product validation as this will
greatly hinder the product
development.”

15. “The plural of
anecdote is not data”
- Lots of misinformed speakers at conferences I’ve attended

16. “The plural of
anecdote is not data”
- Raymond Wolfinger (1969)

17. GreyBlue OrangeWhite
80%60% 70%50%
Data is just data.
To make good decisions, we
need good evidence.

18. A B
Website Optimisation.
Let’s change the button from yellow to blue and see if it increases the magic number.
Buy now! Buy now!

19. A B
Hypothesis Testing.
We observed in user research that some people have difficulty finding the “buy now”
button. We suspect this is caused by the low contrast between the font and the
background. To solve this user issue, we will change the button from yellow to blue. If this
solution works, we expect to see more users hover and click, and eventually purchase.
Buy now! Buy now!

20. Finn Hansen.
Product Owner
“We use experimentation to
help us validate hypotheses
with the goal of addressing
well defined user problems.
It's all about learning as fast
as possible in the most
rigorous way possible.”

21. Based on [prior] we believe [condition] for
[users] will encourage them to [behavior]
We will know this when we see [effects]
happen to [metrics]
This will be good for customers, partners
and our business because [motivation].
Hypothesis template.
Theory
Validation
Objective

22. Diogo Antunes.
Principal Developer & Fellow
“We work in a global scale.
By validating my changes
through experimentation I
continuously get challenged
on my assumptions and the
way I look into the
experience of our
customers.”

23. Take the biggest small step
so you can challenge your
riskiest assumptions quickly

24. The order of items can be improved.
The order of items matters to users.
The feature can be improved.
The feature matters to users.
The page can be improved.
What is on the page matters to users.
Assumptions.
Order
Feature
Page

25. Evidence-based
customer-centric
product development

26. This is a story
about people

27. Empower people to
experience the world.

28. Thank you!
www.workingatbooking.com