In this paper we focus on estimating the post-click engagement on native ads by predicting the dwell time on the corresponding ad landing pages. To infer relationships between feature of the ads and dwell time we resort to the application of survival analysis techniques, which allow us to estimate the distribution of the length of time that the user will spend on the ad. This information is then integrated into the ad ranking function with the goal of promoting the rank of ads that are likely to be clicked and consumed by users (dwell time greater than a given threshold). The online evaluation over live traffic shows that considering post-click engagement has a consistent positive effect on both CTR, decreases the number of bounces and increases the average dwell time, hence leading to a better user post-click experience. There are the slides of our WWW 2016 paper. This is work with Nicola Barbieri and Fabrizio Silvestri.

1. Improving Post-Click User
Engagement on Native Ads via
Survival Analysis
Nicola Barbieri, Fabrizio Silvestri and Mounia Lalmas

2. Na$ve ads and user engagement
• Es$mate post-click engagement on
mobile na$ve ads
• Integrate predic$on into ranking func$on
to promote the most engaging ads

3. • We measure
engagement by analyzing
what happens a0er the
user click:
– How much $me the user
spends on the landing
page?
• Two possibili$es:
– the user immediately
comes back (bounces)
– the user stays longer and
“hopefully” convert (high
dwell $me)
looser winner
Dwell $me based user engagement

4. How are na$ve ads selected?
?
?
?
?

5. ?
?
?
• It depends on
poten$al revenue:
eCPM = P(click) * bid
– P(click) is the
probability of user
clicking on an ad.
– Bid is the amount of
money the adver$ser
is willing to pay for
ad being shown.
How are na$ve ads selected?

6. eCPM = 0.1
eCPM = 0.08
eCPM = 0.05
How are na$ve ads selected?

7. Now consider dwell $me
• Suppose we can es$mate
pt = P(dwell_5me > t):
– Probability that a user will
stay for more than t seconds
on the ad landing page.
• We modify eCPM by
mul$plying it by pt
– In prac$ce we are compu$ng
probability of clicking AND
staying for more than t
seconds.

8. How to es$mate pt?
• If t is fixed a-priori it is, basically, a classifica$on
problem:
– Logis$c Regression
– Random Forest
– …
• What if we want to use different thresholds
depending on factors such as, users, ad category,
etc.?
– Different ad-hoc models
– Survival Analysis!

9. Survival Analysis
• Stems from the necessity of making a prognosis
for specific pa$ents, i.e., to es$mate the
probability of surviving a specific amount of $me.
• In other contexts, the response is not ‘survival’,
but a 8me to event:
– How long will a bulb ‘survive’
– Time un$l first tooth is affected with caries
– Time un$l a hard drive will fail
– Time un8l users return to mobile news stream
– …

10. Survival Analysis
• Interest is then in the survival func$on:
S(t) = P(Outcome > t) à probability of being alive a0er 8me t
= 1 – probability of dying before 8me t
S(t) = probability of not returning to stream aZer $me t
= 1 – probability of returning to stream before $me t
• Proper$es of S(t):
– S(0) = 1 à you are absolutely certain to be
alive at the beginning of $me.
– S(+∞) =0 à you will eventually die.
– S’(t) >= 0 à aZer one hour you cannot have
more chances to live than before.

11. Examples of a survival curve
higher post-click
engagement
lower post-click
engagement

12. Hazard and cumula$ve hazard
• Hazard func$on is defined as
rate of occurrence (users returning) at $me t
• Cumula$ve hazard (CHF) is the sum of the
risks (users returning) from dura$on 0 to t
• S(t) and H(t) are related as follows:
h(t) =
d
dt
log (S(t))
H(t) =
Z t
0
h(u)du
S(t) = e H(t)

13. Survival Random Forest
• How to es$mate S(t) from a
popula$on?
– Training data
• Survival Random Forest (SRF):
– Non-parametric
– High performance
– Learning in a mul$-threading
environment
• Is similar to “classic” Random
Forest but at leaf nodes you find
an es$mates of the Cumula8ve
Hazard H(t)

14. Experiments & dataset
• We sampled 46,914 observa$ons (impressions) from our na$ve ad log,
corresponding to:
– 2,438 ads
– Over 850 adver$sers.
• We perform a 80/20 training/test split.
• For each ad we extracted 42 features
• Hyper-parameters:
– 100 trees as we found it to op$mize the tradeoff $me vs generaliza$on
error
– Number of features sampled at each split is √nf, where nf is the number of
features.
• Offline Tests:
– AUC and ROC curves.
• Onine (A/B) Tests:
– Dwell $me upliZ, Bounce rate reduc$on, CTR upliZ.

15. Features (and their importance)
Historical features are most important
Then document object and readability features:
dwell $me is influenced by how much “actual” content is present
within a landing page, and what is the quality of this content.

16. Es$mated dwell $me
80% with dwell $me
≤ 100 seconds
median is 45 seconds
average is 65 seconds
Kaplan-Meier es$mate

17. ROC Curves SA achieve same
accuracy as
counterparts
Non-linear > linear
SRF > RF
SRF does not require
predetermined
threshold

18. UpliZ in online (A/B) tes$ng
Metrics
Dwell $me +13%
Bounce rate -10.3%
CTR +6.8%

19. Variable threshold?

20. Conclusions and future work
• Survival random forest based (slightly) outperforms all the
other compe$ng model and, more importantly, it allows to
compute the survival at different thresholds.
• The A/B test shows an improvement in user experience:
– A posi$ve effect on CTR
– A decrease in the number of bounces, and
– An increases in average dwell $me.
• Future work includes:
– Personaliza$on of threshold per user, per ads, and per ver$cal
(category)
– Integrate other signals related to revenue.

21. Appendix

22. Building $me and OOB error

23. Cumula$ve hazard func$on