A talk I gave at 99designs about website optimisation using Multi Armed Bandit algorithms. It's based on a talk and ebook by John Miles White — "Bandit Algorithms for Website Optimization"

2. A/B testing

3. Multi Armed
Bandit Algorithms

6. Why do we run
A/B tests?

8. Before During After
Logo 1
Logo 1 + Logo 2
Logo 2
ConversionRate
Time

9. Before During After
Logo 1
Logo 2
ConversionRate
Time

11. Explore or exploit

12. Objectively best
Option that will be the best in the future

13. Subjectively best
Option that has been the best in the past

15. Explore
Choose any option
Exploit
Choose the subjectively best option

16. Regret

18. Some classic
Multi Armed
Bandits...

19. Epsilon greedy

20. ((1-e) * 100)% to subjectively best
(e/2 * 100)% to subjectively best
(e/2 * 100)% to subjectively worst

21. Monte Carlo
Run random simulations 1,000’s of times

23. Weaknesses of
ε-greedy
Situation 1:
A: 99%
B: 0.001%
Situation 2:
A: 0.001%
B: 0.002%

24. Softmax

25. P(A) = 0.1
P(B) - 0.2

29. Weaknesses of
softmax
Situation 1:
A: 0.01% after 100 trials
B: 0.02% after 100 trials
Situation 2:
A: 0.01% after 100,000,000 trials
A: 0.02% after 100,000,000 trials

31. UCB
Upper Confidence Bound

36. Weakness of
UCB1
Gotcha: rewards have to be between 0.0 and 1.0
Works best on conversion rates.
Not as well on arbitrary dollar rewards.

38. Further reading..

39. • Other UCB* algorithms
• LinUCB / GLM-UCB
• Exp3 and other Exp* algorithms

40. Thanks!