A/B testing of ad monetisation based on experience from Outfit7 tests (9 billions+ downloads). Which aspects of ad monentisation are easier to test which ones are more difficult? How to look for hypothesis, ideas to test? Which points you should start A/B testing. Which ones require more resources, which ones require less. What kind of improvements can we expect?

1. © Outfit7 Limited 2010 – 2019
AB testing in ad
monetisation
Ante Odić
Deputy VP of Product Management

2. 9+ billion downloads, 22 games,
350 million MAU,
23 billion video content views
<

3. Introduction

4. Analytics
infrastructure
● BigQuery
● Python
● R
Introduction
AB testing
What do you need to do?
Randomly put users into two
or more groups
Have enough
users
BE
User level data
Configuration
Collect user level
data
Change their
settings from BE
Analyse data
Analyse a lot of
data
Understand
scientific
methodology

5. Introduction
Ad monetisation
What it is and why do you care?
● Rewarded videos, banners, interstitials
● Generating opportunity to show ads
● Ad mediation of partners
● Big revenue source for specific type of games

6. Introduction
What are we optimizing
Ad mediation
AD TECH SIDE
● Showing the most valuable ad
● Improving data
● Creating algorithms
Generating opportunity
PRODUCT SIDE
● right position and frequency of
banners
● right timing and frequency for
interstitials
● right motivation of users for
rewarded videos

7. Generating
opportunity

8. Generating opportunity
AB testing
● Number of generated
opportunities
● Requests per user
● Impressions per user
● Percent users with
impressions
● Retention
● Session duration
● Overall monetisation
Focus
“Easy” because user level data is available
Big impact
Good statistical power for most metrics

9. Ad
Mediation

10. Ad Mediation
AB testing
Hard because user level data is NOT available
Big impact when you start
Testing revenue is tricky
● Waterfall algorithms
● Data processing
algorithms
● ARPDAU
Focus

11. Problems
Ad Mediation
No user level data means you depend on reports from partners
Not real AB tests but testing revenue on segments
Time series analysis
Find pragmatic solutions to move quickly

12. Data processing algorithms
Ad Mediation
Use simple logic to understand and solve problems Simulate Implement and test
Data is messy and comes in late.
What do we base our algorithm on?
Predicting current ECPMs
Irregularities on the partner’s side
make your data messy and volatile.
Algorithms should not overreact.
Volatility algorithm

13. Ad mediation algorithms
Ad Mediation
Waterfall algorithm, calculating the
expected value
In which order should we ask partners
for ads?
Impressions
Revenue
Requests
Country
Platform
...
Algorithm Waterfall

14. Now what?

15. Where to start?
Get yourself some data fast
Test opportunity first
Be pragmatic

16. Be careful!
All of this is closely connected to a whole set of metrics
Measure the big picture
Assumptions are not facts, even if you really really believe in them
If you cannot explain it to a kid, you do not understand it

17. What to expect?
Without analytics and AB tests you are not aware of the potential
Without AB tests you can do harm and not even know it
If there is one aspect where you can do huge impact it is
monetisation

18. Thanks!

19. DISCLAIMER
This content and all attachments to this content (“content”) are copyright of
Outfit7 Limited - © 2010–2019 Outfit7 Limited. All Rights Reserved.
Unauthorized use is prohibited. Any redistribution or reproduction of part or
all of the contents in any form is prohibited other than the following: you may
print or download to a local hard disk extracts for your personal and
non-commercial use only. You may not, except with our express written
permission, distribute or commercially exploit the content. Nor may you
transmit it or store it in any other website or other form of electronic retrieval
system.
Third party assets or images are used for presentation purposes only. No
rights reserved.
© Outfit7 Limited 2010 – 2019