I presented this talk on the ItNonStop conference on November 18th, 2017 in Wrocław (https://it-nonstop.net/city/wroclaw). I explained what A/B testing is and how do we in Booking.com apply it to our location-related information, such as location of museums, landmarks, beach, shopping malls, or historical areas.

1. A/B Testing with Geodata
in Booking.com
Elena Nikolaeva | November 2017
@ice_lenorice.lenor@gmail.com | smartpuffin.com |

2. A/B testing
What is it?

3. Which button color is
the best?
OR
?

4. Let the users
decide!

5. Experiment flow.
50%
50%
10% conversion
20% conversion Win!
Lose.

6. 10% conversion
20% conversion
What does “the best” mean?
50%
50% Win!
Lose.

7. Metrics:
● Conversion
● Click-through rate
● Log-ins
● Bounce rate
... anything your business
wants the customers to do!

8. Additional metrics:
● Clicked on an element
● Clicked on a different
element
● Spent time on the page
● Scrolled down
Helps to learn about user
behavior.

9. Building a hypothesis

10. “If I implement X,
the users will do Y,
and it’s good because Z”

11. “If I... add a cat to my presentation,
the listeners will... say AAAaaw,
and it’s good because... they will
remember my talk”

13. Success:
Congratulate yourself!
Continue on topic.
Where else can you
apply the same
approach?

14. Failure:
That’s unfortunate.
But don’t give up just
yet!
Analyze what went
wrong.

15. Repeat.

17. RepeatAnalysisExperimentMetrics
Steps to a successful experiment:
Hypothesis

18. Learning
is the most important thing

19. A/B testing:
not only button colors

20. What is
geodata?

22. 80%
of users think
location is very important
when choosing an accommodation.

23. How do we run
A/B tests on geodata?

24. Step 1. Build a hypothesis

25. User research

26. Street testing

27. Surveys

28. Data analysis

29. “I want to stay in the historical center
of the city,
but I don’t know where it is.”

30. 2. Implement the experiment.

31. Brainstorm

32. Our Geo Team

33. Data collection process.
MVP approach: collect a small piece of data
to experiment with.
Prioritize collection!

34. We collected the data
for 30 cities
where our customers left reviews
mentioning ”Old Town”
the most
Data collection process.

35. Lisbon Old town

36. Old Town

37. 6. Repeat.

38. B.old Town

39. Now grow the data!
We collected 100 more
old towns!
Success?

40. Our challenges

41. Defining
customer problems
is hard.

42. Many places.

43. Many places.

44. Very location-specific.

45. Thank you!
Elena Nikolaeva | November 2017
@ice_lenorice.lenor@gmail.com | smartpuffin.com |