The Uneasy Marriage of Data Science and Operations Egle Grublyte & Rob Winters

1. Data Science and Operations
T h e U n e a s y M a r r i a g e o f
A C a s e S t u d y i n O p e r a t i o n a l
C h a n g e

2. WHO IS
TRAVELBIRD
Founded in 2010, TravelBird’s focus from
the beginning has been to bring back the
joy of travel by providing inspiration to
explore and simplicity in discovering new
destinations.
Active in eleven markets across Europe and
inspiring three million travelers daily via
email, web, and mobile app.
Beginning in mid-2016, TravelBird
began discussing a potentially new
path, transitioning from a static
daily selection / “flash deal” model
to a managed portfolio model with
personalized daily selections.

3. FROM INDUSTRIAL
E ve r y d ay a m a n u a l s e l e c t i o n o f s i x o f f e r s p r e p a r e d t o b e
s e n t t o t h e e n t i r e e m a i l b a s e . N o a c t i ve p o r t f o l i o
m a n a g e m e n t a n d n o p e r s o n a l i z a t i o n .
Th i s o p e ra t i n g p a ra d i g m h a d b e e n i n p l a ce s i n ce t h e
co m p a ny ’ s f o u n d i n g .
Q 2 / Q 3 ‘ 1 6
TO ENLIGHTENED
E ve r y o n e o f t h e t h r e e m i l l i o n e m a i l s p e r s o n a l i z e d by
o f f e r , s u b j e c t l i n e , t e m p l a t e s t r u c t u r e , a n d t i m i n g .
P o r t f o l i o m a n a g e m e n t s u p p o r t e d v i a m a c h i n e l e a r n i n g
m o d e l s .
A l m o s t a l l o p e ra t i o n a l wo r k d o n e by p l a n n i n g r e p l a ce d
w i t h m a c h i n e l e a r n i n g , a l l ow i n g t h e m t o f o c u s o n m o r e
s t ra t e g i c t a s k s .
Q 4 ’ 1 6 / Q 1 ‘ 1 7
OUR GOAL

4. WHAT IS OUR
PROBLEM DEFINITION?
O U R F I R S T C H A L L E N G E
W h a t d o e s “ p e r s o n a l i z a t i o n ” e n t a i l ?
E m a i l co n t e n t ? E ve r y i n t e ra c t i o n ?
H ow p e r s o n a l i z e d i s e n o u g h ?
F i r s t n a m e i n t h e s u b j e c t ? Co m p l e t e l y
t a i l o r e d co n t e n t ?
To o b r o a d a n d we wo u l d n eve r b e a b l e t o
s h i p ; t o o n a r r ow a n d we wo u l d n o t b e a b l e t o
c h a n g e o p e ra t i o n s i n t h e d e s i r e d fa s h i o n .
M o r e i m p o r t a n t l y , eve r y p e r s o n i n t h e
co m p a ny f r o m t h e C E O d ow n h a d a d i f f e r e n t
i d e a o f w h a t “ p e r s o n a l i z a t i o n ” m e a n t .
Q 3 2 0 1 6

5. WHAT THE HECK IS PERSONALIZATION
F I R S T
We h a d t o g e t a co m m o n d e f i n i t i o n i n s i d e t h e o r g a n i z a t i o n o f w h a t p e r s o n a l i z a t i o n m i g h t e n t a i l .
N o t w h a t w a s i n F o r b e s , n o t G o o g l e H o m e , b u t w h a t l eve r s we m i g h t p u s h g i ve n o u r r e s o u r ce s
a n d o u r co n s t ra i n t s , o p e ra t i o n a l l y a n d t e c h n i ca l l y
C O N T E N T
We defined content as selecting an option out of the
existing portfolio, not creating personal content
P R E S E N TAT I O N O R D E R
We can sort anything but sorting everything would be too
much. We focused on just sorting the top six offers of the
day
I N T E R F A C E
Where we show people the content is important and
different options carry differing levels of complexity. We
picked email only to start

6. WHERE TO START
W E H A D T O E V A L U A T E
S U B J E C T L I N E
Th e f i r s t t h i n g p e o p l e s e e ,
d r i v i n g o p e n ra t e s ca n
m a s s i ve l y l i f t t ra f f i c
C O N T E N T
At t h e e n d o f t h e d ay , t h e
r i g h t o f f e r i s w h a t m a t t e r s
S O R T O R D E R
We k n ow t h a t p e o p l e r e a d
t o p t o b o t t o m , l e f t t o r i g h t .
P o s i t i o n m a t t e r s
T E M P L AT E
Th e s t r u c t u r e a n d n u m b e r
o f o f f e r s ca n h e l p b r i n g
e m p h a s i s o r r e m ove i t
We r e j e c t e d t o t a l p e r s o n a l i z a t i o n d u e t o co m p l ex i t y ; we
w a n t e d t o co m p l e t e o u r p r o j e c t b e f o r e 2 0 2 0 ! Th e f o c u s w a s
o n e m a i l

7. QUICK WINS, MINIMAL IMPACT (AT FIRST)
W E P R I O R I T I Z E D P R O J E C T S F O R
O N E
S O R T O R D E R
T W O
S U B J E C T L I N E S
T H R E E
C O N T E N T
F O U R
T E M P L A T E S
We did this first as it was
easiest, required no
operational change, and
could be tested “in the
background”. An easy
way to prove impact
Subject lines came after
as it required a bit more
effort from our editing
team to create variations
and ensure quality, but
was still minimal impact
This was the end goal
and required massive
operational change,
shifting from daily
planning to category
management.
We started testing
templating after our
operations were stable
and we could focus on
the more technical
opportunities

8. MAKE IT CLEAR
KEEP IT SIMPLE
GET QUICK WINS
MINIMIZE IMPACT AT FIRST

9. HOW DO WE
CONNECT
PEOPLE TO
ANALYTICS?
O U R S E C O N D C H A L L E N G E
Q 4 2 0 1 6

10. JULIE
Julie is our colleague from Team Denmark.
Having a real person with specific examples
helped to get everyone interested and
involved.

11. Cuba Trip
JULIE LOVES
CUBA AND
MOROCCO
Each dot represents a trip: the
darker it is, the more she likes it. The
closer they are, the more alike they
are.
These pictures helped us show what
preferences look like for people.
They also help explain how trips
differ in the customer’s eye.

12. BUT NOT
JUST CUBA
This email contains two Cuban deals.
Everyone could agree that the two
selections are too similar.

13. ABSTRACT CONCEPTS
COULD BE HARD TO UNDERSTAND.
THE EFFORT TO SHARE THE DETAILS PAYS OFF.

14. CAPTURING
EXPERT
PERSPECTIVE
O U R T H I R D C H A L L E N G E
Q 4 2 0 1 6

15. For example, to measure similarity between two trips
we included every observable aspect we had. That
means
price, distance between hotels, image tags, flight and
car information, text features, and more.
What proved to be hard was bringing it all to a single
similarity score. What matters more: distance or price?
397 km
10€
HOW WE SAW IT: MEASURE EVERYTHING

16. AND WE FAILED:
Ever yday we were receiving feedback our recommendations are too similar.

17. EXOTIC
MOROCCO
C U L T U R A L
E XO T I C
SEASIDE
GETAWAY
T H E N E T H E R L A N D S / B E L G I U M
COA S TA L
1 - DAY D R I VA B L E
LONDON
CITY TRIP
FAMILY
HOLIDAY
A M U S E M E N T PA R K S
H O L I DAY PA R K S
FA M I L Y F R I E N D L Y
HOW TRAVELLERS SEE IT…

18. And we created a better selection algorithm
‘’Zig-zag’’ walking algorithm ensured
stable selections quality over time:
Every day we have a “Cuba” deal and
supporting deals for diversity.

19. HUMAN PERCEPTION
IS NOT EASILY TRANSLATABLE TO RULES.
HAVING A GUIDE CAN REALLY HELP.

20. BALANCING
AUTOMATED
AND
HAND-PICKED
O U R F O U R T H C H A L L E N G E
Q 1 2 0 1 7

21. HOW DO YOU RESPOND TO WORLD EVENTS?
Only people can act fast enough,
so we gave them the tools to filter

22. WHAT MAKES A GOOD
CALENDAR?
Human expertise still trumps machine learning here.
Our experts provide rules on what is “good enough.”

23. What are the secrets to a happy marriage
between operations and data science?

24. TALK TO EACH OTHER
LISTEN TO EACH OTHER
LEARN FROM EACH OTHER

25. QUESTIONS?