How to use statistics and ML for a better user acquisition (UA) with Feature Selection (FS) methodology. Is FS for me? How does it work, how to prepare and how can I benefit from it? Let's take look on examples of gaming companies that used Feature Selection (FS) for improve and scale their business.

1. Proprietary + Confidential
UA focused on LTV
On steroids with Feature Selection
Przemyslaw Modrzewski, Principal Analytical Lead, Google
@modrzewski

2. Proprietary + ConfidentialProprietary + Confidential
We are facing a marketing revolution for…… last 20 years...
Clicks
Installs
ROAS / ROI
Profit
Customer Equity
Customer Lifetime Value

3. Proprietary + ConfidentialProprietary + Confidential
What LTV is after all?
Profits
Acquisition
Retention
Cross,up-selling
Costs
Customers
life-cycle

4. Proprietary + ConfidentialProprietary + Confidential
Problem with game players !

5. Proprietary + ConfidentialProprietary + Confidential
Reality…. (we all know that)
98%
to
analyze
intent
All In-App Users
Most Apps
have only
<2%
Paid Users
Paid Users
>60%
One-time
Purchasers
Big Whales<10%

6. Proprietary + ConfidentialProprietary + Confidential
Acquisition strategy - Options we have….
Paid Users
Will-Pay Users
Active Users
TOPLayer -- to Predict pLTV
● Goal KPI：ROI, to acquire more high value users
● Data Needed：All Paid User in-app behavior & payment data
MIDDLE Layer -- to Predict Conversion Probability
● Goal KPI：Payer Rate, to acquire more paid users
● Data Needed：Feature Selected Active Users in-app behavior data
Baseline -- Feature Selection
● Goal: Select what user behaviors are important to payments
● Data Needed：All Active Users in-app behavior data

7. Proprietary + ConfidentialProprietary + Confidential
Feature selection makes a difference
OBJECTIVE
A mobile advertiser
wants to identify the
best event for UAC
Action where IAP as
the optimization goal
does not work well
FEATURE SELECTION APPROACH
Run through all performance related
features and find the one(s) with the
both high relevance to IAP and high
occurrence itself
PREVIOUS APPROACH
Use install or IAP as UAC
optimization goal
IAP
Install

8. Proprietary + ConfidentialProprietary + Confidential
Events can extend “the pool” of users
Events in game
Whales
Conversions

9. Proprietary + Confidential
What makes Feature Selection

10. Proprietary + ConfidentialProprietary + Confidential
TECHNOLOGY
Use Cutting edge Machine
Learning (e.g.
ExtraRandomForest) to find
optimum feature for action
OUTCOME
Each feature gets an Importance, Frequency and Correlation
scores.
Selected features will need a high score on these 4 criterions.
IMPLEMENTATION
Directly leverage the in-app events in
AdWords
Importance:
Unique User Frequency:
Active User Correlation:
Correlation:
PRIMARY criterion for sorting the list. The higher the number, the
more important the feature is for user classification
Number of unique users triggered this feature/action
Criterion measuring how likely increasing the frequency of
actions can make a user convert to paid user
Criterion measuring how likely a user triggered a
certain action will be converting to a paid user
Feature Selection System

11. Proprietary + Confidential
Some games is community-like
where valuable users are not only
who made in-app-purchase but
also non-paid but highly active.
Select the most suitable
features used in UAC to
enhance the performance of
UAC campaign
Feature Selection - what can do with it?

12. Proprietary + Confidential
Payment Amount
above threshold*
(Y/N)
Paid user
(Y/N)
Pay_Count
above threshold*
(Y/N)
Active Days
above threshold*
(Y/N)
Login Times
above threshold*
(Y/N)
Online Time
above threshold*
(Y/N)
*threshold is determined by clients*threshold is determined by clients
ROI/PURCHASE OBJECTIVE RETENTION OBJECTIVE
Different objectives of conversion

13. Proprietary + Confidential
Segment markets into 4 tiers by
Quadrant Analysis with:
● Install to In-app-Purchase Conversion Rate
● Conversion Lag in Days
X-axis, Install to In-app-Purchase Conversion Rate, Y-axis, Conversion Lag in Days, Each dot represents a
market, Quadrant Segment Threshold, average statics of x and y axis
Pay Rate
ConversionLag
average
average
IAP event is very often difficult to recognize by UA systems

14. Proprietary + ConfidentialProprietary + Confidential
We have to remember about an important criteria for
events in UAC
DELAY
No later than 7 days post
install
Ideally D2/3
The shorter the better
RELEVANCY
Has to be deterministic
and correlated with the
business objective
CONVERSION VOLUME
Min. 20 unique user
converted per day
Ideally 50+
The higher the better
Purchase
Registration
Add to cart

15. Proprietary + Confidential
Identify different proxy events, with Feature Selection, for different markets that are
on the path to purchase
T3/T4 Markets: combine the
upper funnel events
● Android: Buy Item
● iOS: Group and clan
message
All Markets:
● IAP
T1/T2 Markets: lower funnel
event
● IAP
● Start payment
Results
T2 IAP -> BuyItem*
AU/CA/SG
Android
NZ/CH IOS
Tier Optimize
T2
T3
25%
7 Days’ ROI
6X
2X
AU/CA/SG
IOS
None -> Group & Clan
Message
IAP -> Group & Clan
Message *
Approach
* BuyItem : Buy items i.e. weapon, skins, etc. with gaming currency *Group & Clan Message : How many messages sent in group or clan
We can go wider (upper-funnel) but still be careful

16. Proprietary + Confidential
UAC for in-app actions with Feature Selection (3 events)
Campaign 1 for IAP
Campaign 2 for IAP
Campaign 3 for IAP
Campaign 4 for IAP
Campaign 5 for IAP
Campaign 1 for FS events
We can acquire 35 users (ave.) instead of one.

17. Proprietary + Confidential
We can be flexible in picking events if FS shows significance

18. Proprietary + Confidential
UAC for in-app actions with Feature Selection

19. Proprietary + Confidential
Events are 250 cheaper then IAP…… ;)
Event 1
Event 2
Event 3
Event 4
Event 5

20. Proprietary + Confidential
Do we have a correlation in event and revenue?

21. Proprietary + Confidential
Be careful with counting profitability with events

22. Proprietary + Confidential
Last but not least
Deep but less Volume
Condition A AND Condition B
1. Use “AND” to combine
features only if no single
feature is insightful and
you don’t want to explore
new features
2. Use “AND” when you are
ROI driven
3. KPI: ROI, CPA, CVR to pay
Shadow but more
Volume
Condition A OR Condition B
1. Use “OR” to combine
features only if multiple
single features are insightful
2. Use “OR” when you are
Ecosystem driven
3. KPI: ROI, Retention Rate,
Cost per Retention

23. Proprietary + ConfidentialProprietary + Confidential
Thanks to Feature Selection we can “swim in bigger pool” ;)
98%
to
analyze
intent
All In-App Users
Most Apps
have only
<2%
Paid Users
Paid Users
>60%
One-time
Purchasers
Big Whales<10%

24. Proprietary + Confidential
Feature Selection is easy in GCP :)
Cloud AutoML
Model is now trained and ready to make predictions
This model can scale as needed to adapt to customer demands
Upload labeled structured
data
Train your model Evaluate
(Classification/Regression)
Class A
Class B
Class C
Class D
Class E
ID Feature Feature ... Label
...
...
ID Feature Feature ...
...
...

25. Proprietary + Confidential
Thank You Thank
You
Thank
You
Contact: modrzewski@google.com