4. I listen to ~ 100 Bln ad opportunities daily
I respond with optimal bids within milliseconds
5. I listen to ~ 100 Bln ad opportunities daily
I respond with optimal bids within milliseconds
I petabytes of data (ad impressions, visits, clicks, conversions)
13. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds
I Bayesian Methods for Ad Lift Confidence Bounds
14. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds
I Bayesian Methods for Ad Lift Confidence Bounds
I Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
15. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds
I Bayesian Methods for Ad Lift Confidence Bounds
I Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
I Using Spark for:
16. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds
I Bayesian Methods for Ad Lift Confidence Bounds
I Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
I Using Spark for:
I Monte Carlo sampling for confidence-bounds
17. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds
I Bayesian Methods for Ad Lift Confidence Bounds
I Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
I Using Spark for:
I Monte Carlo sampling for confidence-bounds
I Monte Carlo simulations
18. Application context: ad impact measurement
I Advertisers want to know the impact of showing ads to users.
20. Measuring Ad Impact: Two Approaches
I Observational studies:
I Compare uses who happen to be exposed vs not exposed
21. Measuring Ad Impact: Two Approaches
I Observational studies:
I Compare uses who happen to be exposed vs not exposed
I Bias a big issue
22. Measuring Ad Impact: Two Approaches
I Observational studies:
I Compare uses who happen to be exposed vs not exposed
I Bias a big issue
I Randomized tests:
23. Measuring Ad Impact: Two Approaches
I Observational studies:
I Compare uses who happen to be exposed vs not exposed
I Bias a big issue
I Randomized tests:
I Randomly expose to test, compare with control (un-exposed)
29. Ad Lift: Response Rates
If we see k = 200 conversions out of N = 10, 000 users,
what is a good estimate for the response-rate?
30. Ad Lift: Response Rates
If we see k = 200 conversions out of N = 10, 000 users,
what is a good estimate for the response-rate?
Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . .
31. Ad Lift: Response Rates
If we see k = 200 conversions out of N = 10, 000 users,
what is a good estimate for the response-rate?
Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . .
But how confident are we?
41. Response-Rate: Bayesian Confidence Bounds
Randomly generate response rates that are consistent with the data.
(Sample rates from posterior distribution given data.)
42. Response-Rate: Bayesian Confidence Bounds
Randomly generate response rates that are consistent with the data.
(Sample rates from posterior distribution given data.)
Find the (0.05, 0.95) quantiles of these rates.
43. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
44. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
I Sample from the posterior distribution of the rate r
I conditional on the observed data (k conversions out of N)
P(r | k) Ã P(k | r) · p(r)
45. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
I Sample from the posterior distribution of the rate r
I conditional on the observed data (k conversions out of N)
P(r | k) Ã P(k | r) · p(r)
à rk
(1 ≠ r)N≠k
· Beta(1, 1)
46. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
I Sample from the posterior distribution of the rate r
I conditional on the observed data (k conversions out of N)
P(r | k) Ã P(k | r) · p(r)
à rk
(1 ≠ r)N≠k
· Beta(1, 1)
à rk+1
(1 ≠ r)N≠k+1
47. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
I Sample from the posterior distribution of the rate r
I conditional on the observed data (k conversions out of N)
P(r | k) Ã P(k | r) · p(r)
à rk
(1 ≠ r)N≠k
· Beta(1, 1)
à rk+1
(1 ≠ r)N≠k+1
à Beta(k + 1, N ≠ k + 1)
48. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
I Sample from the posterior distribution of the rate r
I conditional on the observed data (k conversions out of N)
P(r | k) Ã P(k | r) · p(r)
à rk
(1 ≠ r)N≠k
· Beta(1, 1)
à rk+1
(1 ≠ r)N≠k+1
à Beta(k + 1, N ≠ k + 1)
I Compute (0.05, 0.95) quantiles from the generated rates.
49. Response-Rate: Bayesian Confidence Bounds
A simple form of Gibbs Sampling (more later):
I sample M values of r from posterior
P(r | k) ≥ Beta(k + 1, N ≠ k + 1).
I compute (0.05, 0.95) quantiles
50. Response-Rate: Bayesian Confidence Bounds
A simple form of Gibbs Sampling (more later):
I sample M values of r from posterior
P(r | k) ≥ Beta(k + 1, N ≠ k + 1).
I compute (0.05, 0.95) quantiles
from numpy.random import beta
from scipy.stats.mstats import mquantiles
def conf(N, k, samples = 500):
rates = beta(k+1, N-k+1, samples)
return mquantiles(rates, prob = [0.05, 0.95])
56. Response Rates: Example
If we see k = 200 conversions out of N = 10, 000 users,
what is a good estimate for the response-rate?
Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . .
57. Response Rates: Example
If we see k = 200 conversions out of N = 10, 000 users,
what is a good estimate for the response-rate?
Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . .
=∆ 90% confidence region (1.8%, 2.2%)
59. Ad Lift: Simple Example
I control: 10,000 users, 200 conversions
I test: 100,000 users, 2200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
60. Ad Lift: Simple Example
I control: 10,000 users, 200 conversions
I test: 100,000 users, 2200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
This is a great lift !
61. Ad Lift: Simple Example
I control: 10,000 users, 200 conversions
I test: 100,000 users, 2200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
This is a great lift !
Not so fast! Is this a reliable estimate?
62. Ad Lift: Simple Example
I control: 10,000 users, 200 conversions
I test: 100,000 users, 2200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
This is a great lift !
Not so fast! Is this a reliable estimate?
Could true lift ¸ be 0%, or even negative ?
63. Ad Lift: Bayesian Confidence Bounds
Sampling approach:
Observed data: control: (kc, Nc), test: (kt, Nt)
1. Repeat M times:
64. Ad Lift: Bayesian Confidence Bounds
Sampling approach:
Observed data: control: (kc, Nc), test: (kt, Nt)
1. Repeat M times:
I draw control response rate rc from posterior
P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1).
65. Ad Lift: Bayesian Confidence Bounds
Sampling approach:
Observed data: control: (kc, Nc), test: (kt, Nt)
1. Repeat M times:
I draw control response rate rc from posterior
P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1).
I draw test response rate rt from posterior
P(rt | kt) ≥ Beta(kt + 1, Nt ≠ kt + 1).
66. Ad Lift: Bayesian Confidence Bounds
Sampling approach:
Observed data: control: (kc, Nc), test: (kt, Nt)
1. Repeat M times:
I draw control response rate rc from posterior
P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1).
I draw test response rate rt from posterior
P(rt | kt) ≥ Beta(kt + 1, Nt ≠ kt + 1).
I compute lift L = rt/rc ≠ 1
67. Ad Lift: Bayesian Confidence Bounds
Sampling approach:
Observed data: control: (kc, Nc), test: (kt, Nt)
1. Repeat M times:
I draw control response rate rc from posterior
P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1).
I draw test response rate rt from posterior
P(rt | kt) ≥ Beta(kt + 1, Nt ≠ kt + 1).
I compute lift L = rt/rc ≠ 1
2. Compute (0.05, 0.95) quantiles of set of M lifts {L}.
68. Ad Lift: Bayesian Confidence Intervals
I control: nc = 10, 000 users, kc = 200 conversions
I test: nt = 100, 000 users, kt = 2, 200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
86. Ad Lift Estimation
Main ideas:
I observe test-losers response rate RtL
I observe test win-rate w
87. Ad Lift Estimation
Main ideas:
I observe test-losers response rate RtL
I observe test win-rate w
I we show one can estimate
R0
tw =
Rc ≠ (1 ≠ w)RtL
w
88. Ad Lift Estimation
Main ideas:
I observe test-losers response rate RtL
I observe test win-rate w
I we show one can estimate
R0
tw =
Rc ≠ (1 ≠ w)RtL
w
I compute lift L = R1
tw /R0
tw ≠ 1
89. Ad Lift Estimation
Main ideas:
I observe test-losers response rate RtL
I observe test win-rate w
I we show one can estimate
R0
tw =
Rc ≠ (1 ≠ w)RtL
w
I compute lift L = R1
tw /R0
tw ≠ 1
I similar to Treatment E ect Under Non-compliance in clinicial
trials.
93. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
94. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
95. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
I their probabilities
96. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
I their probabilities
I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet)
97. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
I their probabilities
I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet)
I Sample M values of unknown ◊ from posterior: Gibbs Sampler
P(◊ |Data) Ã P(Data | ◊) · p(◊)
98. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
I their probabilities
I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet)
I Sample M values of unknown ◊ from posterior: Gibbs Sampler
P(◊ |Data) Ã P(Data | ◊) · p(◊)
I For each sampled ◊ compute lift L using above
99. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
I their probabilities
I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet)
I Sample M values of unknown ◊ from posterior: Gibbs Sampler
P(◊ |Data) Ã P(Data | ◊) · p(◊)
I For each sampled ◊ compute lift L using above
I Compute (0.05, 0.95) quantiles of sampled L values
101. Ad Lift: Confidence Intervals
Gibbs sampler convergence may depend on prior distribution:
I start with multiple (say 100) priors
I run them all in parallel using Spark.
102. Uses of Monte Carlo Simulations
I confidence intervals
103. Uses of Monte Carlo Simulations
I confidence intervals
I determine “su cient” population sizes for reliably estimating
104. Uses of Monte Carlo Simulations
I confidence intervals
I determine “su cient” population sizes for reliably estimating
I response rates
105. Uses of Monte Carlo Simulations
I confidence intervals
I determine “su cient” population sizes for reliably estimating
I response rates
I lift
106. Uses of Monte Carlo Simulations
I confidence intervals
I determine “su cient” population sizes for reliably estimating
I response rates
I lift
I understand e ect of complex phenomena
107. Uses of Monte Carlo Simulations
I confidence intervals
I determine “su cient” population sizes for reliably estimating
I response rates
I lift
I understand e ect of complex phenomena
I validate/verify analytical formulas
117. Cookie-Contamination Questions
I How does cookie contamination a ect measured lift?
I Does the cookie-distribution matter?
I everyone has k cookies vs an average of k cookies
118. Cookie-Contamination Questions
I How does cookie contamination a ect measured lift?
I Does the cookie-distribution matter?
I everyone has k cookies vs an average of k cookies
I What is the influence of the control percentage?
119. Cookie-Contamination Questions
I How does cookie contamination a ect measured lift?
I Does the cookie-distribution matter?
I everyone has k cookies vs an average of k cookies
I What is the influence of the control percentage?
I Simulations best way to understand this
120. Simulations for cookie-contamination
I A scenario is a combination of parameters:
I M = # trials for this scenario, usually 10K-1M
I n = # users, typically 10K - 10M
I p = # control percentage (usually 10-50%)
I k = cookie-distribution, expressed as 1 : 100, or 1 : 70, 3 : 30
I r = (un-contaminated) control user response rate
I a = true lift, i.e. exposed user response rate = r ú (1 + a).
I A scenario file specifies a scenario in each row.
I could be thousands of scenarios