At Booking we have more than a million properties selling their rooms to our customers. We have approximately 1000 events per minute from them leading to total 500 GB of data for partner events alone. In order to make sure we receive the relevant inventory from our partners we A/B test various new features. There were more than 100 experiments focusing on availability alone in one quarter. In my talk I ll be talking about A/B testing at Booking, different technologies like Hadoop, Hbase, Cassandra, Kafka etc that we use to store and process large volumes of data and building up of metrics to measure the success of our experiments.

1. A/B Testing powered by Big
Data in Booking.com
Saurabh Goyal | February 2018
1

2. Structure
● A/B testing
● Big Data
● Example

3. A/B testing
What is it?
3

4. Which button color is
the best?
OR ?
4

5. Let the users
decide!
5

6. Experiment flow.
50%
50%
10% conversion
20% conversion Win!
Lose.
6

7. 10% conversion
20% conversion
What does “the best” mean?
50%
50% Win!
Lose.
7

8. Metrics:
● Conversion
● Click-through rate
● Logins
● Bounce rate
... anything your business
wants the customers to do!
8

9. Additional metrics:
● Clicked on an element
● Clicked on a different
element
● Spent time on the page
● Scrolled down
Helps to learn about user
behavior.
9

10. Building a hypothesis
10

11. “If I implement X,
the users will do Y,
and it’s good because Z”
11

12. “If I... add a cat to my presentation,
the listeners will... say AAAaaw,
and it’s good because... they will
remember my talk”
12

13. 13

14. Success:
Congratulate yourself!
Continue on topic.
Where else can you
apply the same
approach?
14

15. Failure:
That’s unfortunate.
But don’t give up just
yet!
Analyze what went
wrong.
15

16. Repeat.
16

17. 17

18. RepeatAnalysisExperimentMetrics
Steps to a successful experiment:
Hypothesis
18

19. Learning
is the most important thing
19

20. A/B testing:
not only button colors
20

21. Big Data

22. Sold Incremental
Inventory
22

23. ● Over a million properties
● 1000 Inventory events per minute.
● 3.9 TB data from Inventory events.
23
Data Volume

24. ● Mysql replicator
● Kafka
● Hbase
● RTM
● Oozie
24
Tech used

25. 25
Flow chart
Mysql table
changes

26. Example
26

27. User research
27

28. Brainstorm
28

29. Almost sold out
29

30. Our challenges
30

31. Defining
customer problems
is hard.
31

32. Metrics
32

33. Number of Iterations

34. Thank you!
Saurabh Goyal | February 2017
34

35. Questions?
35