DDTT11: Ton Wesseling - 21-01-20

Ton Wesseling
How an analyst can add value!
Digital Experiments

TON@ONLINEDIALOGUE.COM
Dale Ha's on Researchgate

A/B-tes5ng mastery course
This talk only makes sense if
you have 10.000 transactions
or more per month – enough to
get experimentation in the
DNA of your organization.!

Data Analyst - The Noun Project icon from the Noun Project

Behavior Analyst Meaning Noun shirt on Amazon.com

DEF!
The task of an analyst within an A/B-testing Culture!
1.  Data!
2.  Effectiveness!
3.  Finance!

Data!
Let there be high quality data!

Make sure all funnels are measured…!

Make sure your testing solution has all users!
Users on template: 42186!
Users in the tool: 37652!
Users with code executed: 34312 !
100%!
89%!
81%!

What if my experiments had 20% more users?!

Recognizing returning users!
Buddhini S. on Jargon Wall

Be able to segment on page interactions!

Be able to segment on who can be inﬂuenced!

Be able to create behavioral segments!
Typical ecommerce ﬂow example:
ü  All users on your website with enough time to take action
ü  All users on your website with at least some interaction
ü  All users on your website with heavy interaction
ü  All users on your website with clear intent to buy
ü  All users on your website that are willing to buy
ü  All users on your website that succeed in buying
ü  All users on your website that return with intent to buy more
Funnel
+
Average
5me

Effectiveness!
Make sure you work on stuff!
with the highest potential outcome!

Statistical Power!
The likelihood that an experiment will
detect an effect, when there is an effect
there to be detected!

Power & Signiﬁcance
New version is
NOT better
New version is
better
New version is
NOT better
New version is
better
Measured
Reality

Do not reject H0 Reject H0
New version is
NOT better
New version is
better
Measured
Reality

H0 is true
H0 is false
Measured
Reality

Signiﬁcance
H0 is true
H0 is false
Correct decision
J
Measured
Reality

Signiﬁcance
H0 is true
Type I
False Positive (α)
H0 is false
Correct decision
J
Measured
Reality

Power
H0 is true
Correct decision
J
Type I
False Positive (α)
H0 is false
Correct decision
J
Measured
Reality

Power
H0 is true
Correct decision
J
Type I
False Positive (α)
H0 is false
Type II 
False Negative (β)
Correct decision
J
Measured
Reality

Power
New version is
NOT better
New version is
better
New version is
NOT better
Correct decision
J
Type I
False Positive (α)
New version is
better
Type II 
False Negative (β)
Correct decision
J
Measured
Reality

Power & Significance rule of thumb
Power
When you start: try to test on pages with a high Power
(>80%) à otherwise you don’t detect effects when there is
an effect to be detected (False negatives).
Significance
When you start: try to test against a high enough
significance level (90%) à otherwise you’ll declare winners,
when in reality there isn’t an effect (False positives).

https://abtestguide.com/abtestsize/!

https://ondi.me/bandwidth!

Prioritize based on MDE to start!

Prioritize based on measured results!!

Finance!
Business case calculations!

What does your calculation look like?!
If signiﬁcant result:!
!
Extra new customers per week!
*!
52 weeks effective!
*!
Average lifetime value!

So this experiment will bring us:!
€412.390!

So this experiment will bring us?!
€412.390 * (100%-Type-M error %)?!

Prioritize based on measured results?!
* (100% - M-Type Error) of course!

What is your false discovery rate?!
Signiﬁcance border: 90%!
100 experiments!
20 signiﬁcant outcomes!
!
50%!* (it’s a little lower, this is the poor man’s calculation)!
(with every real win the number of experiments without wins becomes lower, which leads to less false positives)!

So not really 50%!
FDR* = (Measured Wins - ((Measured Wins - !
((100% - Conﬁdence Level) * Experiments))!
/ Conﬁdence Level)) / Measured Wins!
!
=!
!
(20 – ((20 – ((100% - 90%) * 100)) / 90%)) / 20!
!
=!
!
44%!* (only if your power on all experiments was 100%)!
(Your Power will be lower, which means you had more real wins, but not measured (false negatives).!
This leads to less experiments without an effect, so the number of false positives will be even lower)!

https://abtestguide.com/fdr/!

So all your experiments will bring you:!
Sum of (every winner *!
!
(100% - Type-M error % per winner))!
*!
(100% - FDR%)!
*!
Implementation % (within x months…)!
(assuming every new win is tested on the new default where all earlier wins are implemented)!

You can correct FDR for P-value distribution!

So all your experiments will bring you:!
Sum of (every winner *!
!
(100% - Type-M error % per winner))!
*!
(100% - corrected FDR%)!
*!
Implementation % (within x months…)!
(assuming every new win is tested on the new default where all earlier wins are implemented)!

Maximize your growth with ROI limits:!
Value of A/B-testing for Optimization!
___________________________________!
!
Costs of A/B-testing for Optimization!
= ROI!

Finance: are you above or below your ROI limit?!
1.  Above: increase budgets!
2.  Below: increase knowledge!
3.  Still below: decrease budgets!

Data Analyst - The Noun Project icon from the Noun Project
An A/B-testing for growth analyst:!
1.  Makes sure there is high
quality Data available!
2.  Steers the data chance
on Effect!
3.  Reports on the real
Financial impact!

à https://ondi.me/cxlcourse ß!
Questions:!

Ton Wesseling
https://ondi.me/tonw
Let’s connect on LinkedIn

Latest article on A/B-testing:

DDTT11: Ton Wesseling - 21-01-20

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie DDTT11: Ton Wesseling - 21-01-20

Ähnlich wie DDTT11: Ton Wesseling - 21-01-20 (20)

Mehr von Webanalisten .nl

Mehr von Webanalisten .nl (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

DDTT11: Ton Wesseling - 21-01-20