Slides from Rework AI Assistance Summit in San Francisco

1. Personalized assistance at
Booking.com

2. • Data driven culture
• 1000+ simultaneous A/B tests
• Tons of data about 1,550,000 room nights every 24 hours
• Personalized recommendation
• Direct queries: chatbot
Data science in Booking.com

3. • Everything is personalized
• Focus on what is important
• ML models to predict intend
Personalization in Booking.com

5. • Predict relevant hotels
• Bias on what we display
• Hotel was not booked because not seen
• Randomize the position
• Viewed examples as negative
• Learning to rank
Learning to rank: Biased data

6. • Predict the outcome not the intend
• Biased based on our supply
• Predict booking with breakfast
Modeling: correlation vs causation

7. • One feature determines the outcome variable:
• City- most important coefficient
• Saint Petersburg: 2,280 properties, 592 with breakfast (25%)
• Palma de Mallorca: 90 properties, 60 with breakfast (66%)
Modeling: correlation vs causation

8. • Test everything in an experiment!
• Randomize data to reduce bias
• Analyze the output model
Solution

9. Automatic assistance: Chat bot

10. “How many of you remember how late
can you check out of your hotel?”

12. Automatic assistance: Chat bot
● Post-booking
● Menu & conversation interface
● Intent recognition using NLP
● Human-in-the-loop

13. Not so easy

14. AA challenges: short, many topics
● Short message- reduced info
● Many topics- which answer to choose?
● “Can I check in early and do you have parking?”
Solution: More complex iterations; Mini dialogs

15. AA challenges: granularity
● Easy questions:
○ “Does the hotel provides breakfast?”
● Complicated questions:
○ “Does the hotel provide eggs for breakfast?”
Solution: Human in the loop

16. AA: help CS
● Complicated question
● Reduce the number of interactions
● Extract info from the text
● Examples:
○ “Can I change check in date to 15th of December?”
○ “Can you provide shuttle from the train station at 13.00
on Monday”

17. Recommendations:
• Lots of data leads to lots of opportunities for personalization
• Test in production on millions of users
Direct interactions:
• Add human in the loop
• Make interaction with the human smart
Wrap up

18. • Data scientists, back end, front end, product owners, designers
• Amsterdam, Tel Aviv, Shanghai
• Contact me: elena.sokolova@booking.com
We are hiring!

19. Questions?