Experiences from defining lean analytics and for running AB tests in a web startup. 1. Lean Analytics requires a Lean project 2. Specify success metric for every feature 3. You need lots of users for an A/B test 4. Make big changes 5. Measure your tests correctly 6. Push through radical changes with A/B 7. Use the right tools

1. 7 Practical Tips
on Lean Analytics
Janne Aukia — 17 Feb 2015

2. 1. Lean Analytics
requires a Lean project
Illustration from commons.wikimedia.org/wiki/File:Kepler-solar-system-1.png

3. Challenging for a consultancy?
Lean Analytics works well with
Lean Startup, Lean UX and
Agile projects
Question designs, question
authorities, ask difficult questions,
be prepared to kill features.
Requires freedom to experiment

4. 2. Write down a success
metric for every feature

5. Was this feature used more than
Google+ login?
If feature isn’t successful,
just kill it.
Measure only actionable
metrics.
Is the feature for acquisition,
retention or revenue?

6. 3. You need lots of
users for an A/B test

7. Typically at least 10k events,
such as visits or people using a
feature.
Doesn’t work for small
startups.

8. 4. Make big changes
Illustration from http://quod.lib.umich.edu/m/moa/agb8710.0001.001/26

9. If you cannot see the changes
from 2 meters away, most likely
your AB test won’t either.
Google A/B tested 41 shades of
blue. Don’t do that — leads to
local optimum.
Illustration from http://www.doxsey.net/blog/too-practically-minded-to-be-of-any-theological-good/

10. 5. Measure your tests
correctly

11. Never terminate early
a running test
Use a split test calculator,
for example:
http://getdatadriven.com/ab-significance-test

12. 6. Push through radical
changes with A/B

14. 7. Use the right tools

15. Google Analytics can be used for
tracking events but it is
painful.
Mixpanel more suited for the
job. Consider Keen.io, Heap as
well.

16. 1.Lean Analytics
requires a Lean
project
2.Specify success
metric for every
feature
3.You need lots of
users for an A/B
test
4.Make big changes
5.Measure your tests
correctly
6.Push through radical
changes with A/B
7.Use the right tools
Summary