ABBY is a Django app that helps you manage your A/B tests. The main objective is to document all tests happening in your company, in order to better understand which measures work and which don't. Thereby leading to a better understanding of your product and your customer. ABBY offers a front-end that makes it easy to edit, delete or create tests and to add evaluation results. Further, it provides a RESTful API to integrate directly with our platform to easily handle A/B tests without touching the front-end. Another notable feature is the possibility to upload a CSV file and have the A/B test auto-evaluated, although this feature is considered highly experimental. At Jimdo, a do-it-yourself website builder, we have a team of about 180 people from different countries and with professional backgrounds just as diverse. Therefore it is crucial to have tools that allow having a common perspective on the tests. This facilitates having data informed discussions and to deduce effective solutions. In our opinion tools like ABBY are cornerstones to achieve the ultimate goal of being a data-driven company. It enables all our co-workers to review past and plan future tests to further improve our product and to raise the happiness of our customers. The proposed talk will give a detailed overview of ABBY, which eventually will be open-sourced, and its capabilities. I will further discuss the motivation behind the app and the influence it has on the way our company is becoming increasingly data driven.

1. ABBY
A Django app for A/B test
documentation

2. Who?
● Data Scientist at Jimdo
● Jimdo
○ DIY website builder
○ founded in 2007 in Hamburg, Germany
○ > 180 employees in 3 countries (DE, US, JP)
○ > 10 million websites
Twitter: @datenheini

3. Medicine:
● placebo = control group
● drug(s) = test group(s)
A/B Testing Basics
Internet company:
= old version
= new version(s)

4. A/B Testing Basics
Rules:
● Frame your hypothesis.
● Keep it simple.
● Set a timeframe.

5. A/B Testing Basics
Best practices:
● Be realistic.
● Grab low-hanging fruits.
● Don’t get frustrated.

6. A/B Testing Basics
2 minutes of science...

7. Significance (p-value):
A/B Testing Basics
→ should be 0% (at most 5%)
How often will failed tests lead to positive results?

8. Statistical power:
A/B Testing Basics
→ should be 100% (at least 80%)
How often will you recognize a successful test?

9. Evaluation metrics:
● p-value
● statistical power
● effect size
● confidence interval
A/B Testing Basics

10. Let’s do a little A/B test
And see what we need documentation for.

13. Why?
Communicating
the results

14. Why?
Persisting the results
→ Knowledge Base

15. Why?
Avoid duplicated tests

16. Why?
Everybody has this problem, there needs to be a solution
already!
NO!

17. Why?

18. What?

19. What?
Central place for A/B test documentation

20. What?
Tests and results self-explaining

21. What?
Keep track of
evaluation metrics

22. What?
Tries to encourage
best practices

23. What?
Productivity gain

24. What? Productivity gain!
Reference for daily work
Cross-functional effects
Better understanding of customers

25. How?

26. How?
CRUD app (create-read-update-delete)
RESTful API (WIP)
Test evaluation logic

27. Demo time!

28. It’s your turn!
ABBY is open-source. Have a look:
https://github.com/Jimdo/abby

29. Thank you!