The document summarizes Skyscanner's approach to paid growth through data-driven experimentation and testing. It discusses how Skyscanner:
1) Conducted an uncontrolled test of paid search ads on branded terms that found both cannibalization of organic traffic but also incremental gains.
2) Developed data science tools to measure the causal impact of changes and calculate adjusted ROI accounting for cannibalization.
3) Learned that while some traffic is cannibalized, paid search recovers this and provides incremental revenue, though returns vary between new and returning users.
4) Continues exploring user behavior and ad formats/extensions to further optimize paid acquisition as search landscapes evolve.
3. Originally
from New
York State
Worked in
agencies in
NY, London &
Edinburgh
Joined
Skyscanner
three years
ago
11
Started in
digital
marketing in
2006
Now Director
of Growth
running Paid
Acquisition
4. A little bit about
Skyscanner
Our awesome
travel products
Our success
story (so far)
14. Judson builds awesome
dynamic HTML5 ads
Craig automates a feed
so each traveller sees
personalised copy
Kate delivers media
targeting travellers
based on intent
Sri’s attribution
model values each
touchpoint in the
media mix
16. Experiments Testing Optimisation
(scientific conditions) (uncontrolled tests) (machine learning)
Or is the outcome the most
important thing?
Do we place a higher value on the
insights that get us to a result?
17. Experiments Testing Optimisation
The difference is important, but all are
data-driven tools for growth
(scientific conditions) (uncontrolled tests) (machine learning)
Or is the outcome the most
important thing?
Do we place a higher value on the
insights that get us to a result?
18. Actually a lot of what we do in Paid
Acquisition is uncontrolled testing.
Here’s an example.
19. We started with an insight:
Travellers who search for
‘Skyscanner’ brand terms in
search engines drive a high
volume of qualified traffic.
‘But don’t we rank
in position one
organically?’
22. Our hypothesis:
If we bid on Google paid search ads
on Branded terms, we will drive
incremental traffic to Skyscanner.
23. We hoped to see this:
Without ads With ads
Organic Paid
Incremental
Cannibalisation
24. The data science bit
We built a causal impact tool that uses
Bayesian structural time series model
technique to validate if a change has an
impact on a metric over time
I.e. does turning on paid ads cause a
change in organic traffic
25. The data science bit
We developed a formula to calculate
adjusted ROI that accounts for paid
cannibalisation in the search engine
I.e. if we pay for sessions we would
acquire in organic, is the cumulative
gain from Brand search profitable?
26. Test design:
Identify a
comparable test
& control market
Phase ad delivery
in the test market
(i.e. turn it on/off)
Change
*nothing* in the
control market
Record what
happens to traffic
& revenue in both
27. It looked like this:
Market Phase 1 Phase 2 Phase 3 Phase 4
Test
Control
Paid ads running for each device type
29. Cannibalisation of organic traffic.1
But we recover all of this in paid.2
And there’s an incremental traffic
gain. Sometimes.
3
30. There’s incremental revenue with paid
ads on, even when net traffic is flat.
4
But revenue varies between new and
returning users!
This is true for all device types.5
6
All users Returning New
Mobile Desktop
31. We can be confident we understand
the right Growth strategy to maximise acquisition
in the search landscape.
Now what?
32. We continue to learn!
To further explore behaviour of new and
returning users
As we grow our product in
various markets
To understand the impact of new ad
formats and ad extensions
To monitor variation in incremental value
as the SERPs evolve
33. Have a clear data-
driven framework to
guide how you act
Know when to
experiment, to test,
and to optimise
Keep learning to
accelerate (paid)
acquisition
Summing up
1 2 3
34. Prioritise experiments
A handy calculator tool created by
Skyscanner’s Principal Growth
Manager Douglas Cook
https://growthhackers.com/articles/
a-simple-ice-score-calculator
Avoid these common pitfalls
Tricks of the trade from
Skyscanner’s Head of
Experimentation Colin McFarland
http://codevoyagers.com/2015/11/
26/common-pitfalls-in-
experimentation/
Tool kit
Design like you’re right…
…test like you’re wrong. An
experimentation toolkit by
Skyscanner’s Rik Higham
http://www.experimentation
hub.com/
35. Thank you for listening.
Find my slides (and much more!) at: http://bit.ly/ActivationPaidGrowth
@SkyscannerGrwth | @ilanamunckton
Hinweis der Redaktion
Hello everyone!
My name is Ilana Munckton.
I’m a director of Growth at Skyscanner.
I’m talking to you from our office in Edinburgh, Scotland.
Today I’m going to tell you about how my Paid Growth squad apply data-driven experimentation to acquisition for Skyscanner.
A little bit about me.
You might be able to detect from my accent that I’m not native to Scotland.
I grew up in the US in New York.
And have been working in the digital marketing industry for over ten years.
I started my career working in media agencies in Manhattan, London, and Edinburgh.
I joined Skyscanner three and a half years ago
And now head up our global paid growth efforts for over 35 markets worldwide.
Let me give you some context about why Skyscanner is so great.
With a little insight into our product and the growth we’ve experienced.
Of course most of you are already loyal Skyscanner fans, right? But for those who are new to our product:
Skyscanner is a travel search engine that compares Flights, Hotels and Car Hire.
Our awesome website and mobile apps help travellers to:
search, plan, and book travel directly - from millions of options.
We pride ourselves on our unbiased and comprehensive coverage, meaning you can always find the option that’s right for you. Whether that’s the cheapest hotel, the most convenient flight, or booking with your favourite car hire company.
And I’m really here today because we have an incredible growth story
We have grown a business to over 50,000,000 monthly web visitors and 60,000,000 app installs
And we now have a team of over 800 employees in 10 offices worldwide
We achieved this in part by partaking on our own journey
From a marketing function to the Growth team we are today
Let me give you a little intro to how we are structured, our principles for growth, and how we apply them.
We structured ourselves into cross-functional teams comprised of central and regional growth tribes.
And each of these tribes contain squads combining marketing, engineering, and data science skillsets.
We embraced data-driven experimentation as a key growth principle
As part of this, we launched a training programme that enables us to ‘growth hack like ninjas’
This is all about becoming T-shaped growth hackers,
layering a wide understanding of growth disciplines with deeper domain specialisms.
Eg I have domain expertise in digital media strategy, & leverage working knowledge of the relevant automation & data science tools.
But we are obsessed with translating this theory into how we *REALLY* operate
So we developed a framework that guides how we act in practice
Going to talk about how we leverage these principles in my team.
What my Paid Growth team are all about
How we incorporate both digital media best practice and scientific experimentation in our work
And I’ll share a case study.
I like to think my team make travel easier for more people.
By surfacing our awesome Apps and website as travellers plan their journeys.
We do this by using digital media advertising to reach people in market for travel with search, display, and social.
And we of course do this by optimising for growth metrics
In particular, ensuring travellers get value from using Skyscanner,
and that our growth is sustainable (by maximising Revenue).
Our Paid Growth team is comprised of:
Digital marketing skillsets with individuals who really know search, display, and social
Working with engineers to automate how we work
With Data scientists to advance the sophistication of our measurement
And with some really talented designers who build our awesome creative
Let me show you some of our awesome display prospecting ads to demonstrate how my team works.
1 We designed some beautiful dynamic display ad templates
2 We automated a feed so each traveller sees tailored ad copy with live pricing for their destination
3 We target these ads to users based on their travel behaviours and interests
4 And measure the performance by valuing each touchpoint with a multi-touch attribution model
I want to talk a little bit about how we blend science and digital media best practice.
Experiments use scientifically robust, controlled conditions. In an ideal world we’d run an A/B test every time. They allow us to be confident we’ll get the same result every time we do the same thing.
In many cases this isn’t possible, so we have to design ‘uncontrolled tests’. We still try to validate statistically significant results when we have less than perfect test conditions.
And sometimes we want to apply algorithmic optimisation, e.g. to let bidding engines improve results.
One way to think about it is by answering: Is the outcome more important? Or do we place a higher value on the insights that get us to that outcome?
I want to talk a little bit about how we blend science and digital media best practice.
Experiments use scientifically robust, controlled conditions. In an ideal world we’d run an A/B test every time. They allow us to be confident we’ll get the same result every time we do the same thing.
In many cases this isn’t possible, so we have to design ‘uncontrolled tests’. We still try to validate statistically significant results when we have less than perfect test conditions.
And sometimes we want to apply algorithmic optimisation, e.g. to let bidding engines improve results.
One way to think about it is by answering: Is the outcome more important? Or do we place a higher value on the insights that get us to that outcome?
And you’ve probably heard this question before:
Why should we bid on paid ads when we’re in the first organic search position for the same keywords?
And here’s just one reason why would bid on paid brand ads.
But brand protection isn’t reason enough
We need to be sure we use data to verify our acquisition of travellers drives sustainable growth.
We expected to see something like this.
The yellow represents organic traffic, and on the left we see how much traffic we get when there are no paid ads turned on.
The red represents the paid traffic gained by turning on ads.
If the traffic on the right side is below the white line, there is no net gain.
If it’s higher, then combining organic with paid ads drives incremental value.
This is the science part. I’m not going to go into detail on this.
In layman’s terms - does turning on paid ads cause a change in organic traffic
And if we pay for sessions we would acquire in organic, is the cumulative gain from Brand search profitable?
As I mentioned we did nothing in the control market.
In the test market we
Had no Brand search ads live for the first 14 days
We then had ads only on desktop
Then ads only on mobile devices
And finally ads on all device types
As we predicted, we had some cannibalisation of traffic.
But happily, we recoup all of the traffic taken from organic in Paid.
And in fact *overall*, adding paid drives incremental traffic as we’d hoped.
So turning on ads and organic gives us more traffic than organic alone.
BUT. Only sometimes. In some cases the total amount of traffic is unchanged. Not a loss, but not a gain.
We also found we generate more revenue when paid ads are turned on.
Even when net traffic is flat, because paid traffic converts at a greater rate.
This is the case across all device types too.
And finally, we observed something we didn’t expect. This is exciting.
We found that the gain in revenue varies between different user types.
We have used data-driven testing to identify
the optimal Growth strategy to maximise acquisition
in the search landscape.
We continue testing!
To inform our bidding strategy for different user types
To validate this relationship as the search landscape changes
To assess things like, whether site links or app ad extensions are best on mobile ads
And as we grow in various markets worldwide.
So that’s where we are today!
To summarise all of that in three key takeaways:
Finally, this is just what worked for Skyscanner.
This isn’t a silver bullet.
But here are three things to get you started!
1 a prioritisation calculator
2 tricks of the trade from our head of experimentation
3 and an experimentation toolkit
Thank you for listening.
Happy to answer any questions.