This document provides tips and best practices for using data-driven product development. It outlines a 4-step process: 1) understand business objectives and define key performance indicators, 2) build mock reports and define analytics events, 3) launch experiences and continuously improve through A/B testing and reports, 4) rethink the entire product to achieve global maxima. It also provides 7 tips for effective A/B testing including looking at leading indicators, managing expectations, adding common sense, getting clean samples, using "ghost links" to understand demand, using analytics for debugging, and identifying targeted audiences.
2. 2
A Bit About Myself
General Manager,
Consumer
Experience
Business Unit
CEO & Founder,
SafetyNest
VP Product & Eng,
Sonicbids
(acquired by
BackStage)
Head of Next
Generation Product
Team, Yahoo
Search
Strategy Consultant,
Business Consulting
Services, IBM
Marketing Strategy
Manager, Consumer
Electronics Division,
msystems
8200, Bachelor in
Computer Science
MBA, Harvard
Business School
9. 9
First you should know your KPIs –
what are you trying to measure?
STEP ONE
(This is actually the second step after understanding your business (-: )
12. 12
Then, before you define which BI
events to add, build a mock report
based on your hypothesis
STEPTWO
13. 13
Example
64% 82% 12 Months
Premium
Wix users
Wix users already
used ShoutOut
AverageWix users’ age
1.
2.
3.
4.
5.
14. 14
After you launch your product,
continue to improve via abTests &
product analysis reports
STEPTHREE
15. 15
What is an abTest? (1 min overview)
A/BTesting = Running a simultaneous
experiment between two or more
product variants to see which
performs better
statistically significant better
16. 16
Change is signal and not noise
Use of 4 numbers
(sample sizes for a & b, converted in a & b)
Confidence >95% to say that A or B won
A. B.
19. 19
To reach a global max
(e.g. 3x conversion), re-think
your entire product
STEP FOUR
20. 20
Continuing to improve the current experience / funnel will
get you close to an optimal local max (i.e. Evolution)
In order to reach a global max, you need to do
something significantly different (i.e. Revolution),
and not to continue to improve the current experience
24. 24
Understand your
business / product
objectives
Define your KPIs
Continue to improve and iterate
(via abTests & Reports) till you
find a better baseline
Define your
BI events
Launch the experience,
and measure it;
this is your baseline
Then, it will become
your new baseline
1
2
3
Build a mock report
25. 25
Understand your
business / product
objectives
Define your KPIs
Build a mock report
Continue to improve and iterate
(via abTests & Reports) till you
find a better baseline
To reach a global max, re-
think your entire product
Define your
BI events
Launch the experience,
and measure it;
this is your baseline
Then, it will become
your new baseline
1
2
3
4
26. 26
Tip #1
Don’t have enough traffic?
Look at leading-indicators
and/or delta trend over time
30. 30
Unless you add many features / new product
/ top funnel / payment / package picker
Sanity tests are totally fine, but set your
expectations accordingly
Tests (in most cases) won’t show you an
impact on churn ( use reports / monthly)
32. 32
Remember, 95% confidence, can still mean that
5 out of 100 tests can show a false-positive or
false-negative result
If the results don’t make sense, try to give it
more time (and use the tip #1)
34. 34
The more precise your target audience is, the
less you “dilute” your signal and risk seeing
nothing even when there is a change
Usually start only from “new users”
Run the test on the specific page (vs. the entire
product) – i.e. only users that were actually
exposed to your new feature
35. 35
Tip #5
BI can help you understand
“demand” prior to actually building
the product/feature via “Ghost
Links”
41. 41
Tip #7
If you want to perform interviews /
surveys / observations with a
targeted audience, BI reports will
help you identify it
42. 42
For instance, only users who used
feature X but not featureY
Better understand the rational
behind usage data
Investigate what is the most needed
feature or biggest pain point