2. Actually, why these tests?
Everybody…
My child always.... and it is so cute.
No, no, no. That will not be working. Not that way.
Believe me, I am 6 years in this business
....
4. Actually, why these tests?
Let's do a survey
Let's show a model of the experts
Let's do the study focus
"Let our ideas die for us," Karl Popper
„Data driven company”
5. Should I ask them?
Why are you going to buy 50’ TV set?
To watch “National Geographic” and “Discovery”
The real reach of “National Geographic” channel is ...
6. Should I test everything?
Seriously, almost everything
An effectiveness of recommendation systems
An effectiveness of search algorithms
Changes in construction sites
Changes in applications
Each new functionality
7. Should I do tests on all users?
Yes, you can. But...
5 algorithms (5 groups 20% each) mean:
80% non optimal strategies
less money on your account
8. Should I do tests on all users?
Multi-armed bandits :)
9. Should I ensure separation
between groups/experiments?
Yes, definitely.
Two experiments changing construction of this same web
page….
10. Randomness
Randomness: each user should have an equal probability of
being in the group.
Two possible approaches:
1. We share online as users appear - overrepresentation
of more active users
2. We share off-line before - we do not know the next time
of user’s appearance
