One thing that separates great product teams from the rest is how well they use analytics and experimentation to make decisions. Product teams live in a data-driven world and the bar is rising. Analytics increasingly offer greater transparency into understanding how your customers are using your product, what’s working, and what isn't. A strong experimentation mindset enables you to systematically drive improvement. In this webinar, product management expert Dan Olsen shares his Lean Product Analytics Process: a methodical, step-by-step approach to improving your product with analytics and experimentation. Dan also shares advice from his book The Lean Product Playbook, illustrating the concepts with real-world examples and case studies. Watch the recording to learn how to harness the power of analytics and experimentation to improve your product.

1. How to Use Experimentation to
Optimize Your Product and Business
Dan Olsen 3-28-19

2. n Engineering background: EE, submarine design
n Stanford MBA
n Product Management leader at Intuit & startups
n CEO & Cofounder, TechCrunch winner YourVersion
n Product Management consultant & trainer
n Founder, Lean Product Silicon Valley
Twitter: @danolsen
My slides: https://dan-olsen.com
My Background
Copyright © 2019 @danolsen

3. Spiderman’s motto:
“With great power comes
great responsibility”
Copyright © 2019 @danolsen

4. Product Manager’s motto:
Spiderman’s motto:
“With great power comes
great responsibility”
“With great responsibility
comes NO power”
Copyright © 2019 @danolsen

5. Product Teams Are Responsible For
Achieving
Product-Market Fit
AND Growth
Copyright © 2019 @danolsen

6. The 3 Phases of Your Product
Copyright © 2019 @danolsen
BEFORE
Product-Market Fit
AFTER
Product-Market Fit
BEFORE your
product is live
Phase 1
AFTER your
product is live
Phase 2
#ofCustomers
Time
0
lots
Phase 3

7. Quantitative vs. Qualitative Learning
QualitativeQuantitative
OprahSpock

8. Qualitative Complements Quantitative
Copyright © 2019 Olsen Solutions
Qual
Why?
Quant
What?
How many?

9. The 3 Phases of Your Product
BEFORE
Product-Market Fit
AFTER
Product-Market Fit
Growth
Mainly
Quantitative
BEFORE your
product is live
Phase 1
Test hypotheses,
gain confidence
before building
Mainly
Qualitative
AFTER your
product is live
Phase 2
Improve product-
market fit
Qualitative &
Quantitative
Goal:
Methods:
Phase 3

10. The 3 Phases of Your Product
BEFORE
Product-Market Fit
AFTER
Product-Market Fit
Growth
Mainly
Quantitative
BEFORE your
product is live
Phase 1
Test hypotheses,
gain confidence
before building
Mainly
Qualitative
AFTER your
product is live
Phase 2
Improve product-
market fit
Qualitative &
Quantitative
Goal:
Methods:
Phase 3
What phase are you in?

11. That’s Why I Wrote
Copyright © 2019 @danolsen

12. What is
Product-Market
Fit?

13. The Product-Market Fit Pyramid

14. The Product-Market Fit Pyramid

15. The Product-Market Fit Pyramid

16. The Product-Market Fit Pyramid

17. The Lean Product Process

18. The Lean Product Process

19. The Lean Product Process

20. The Lean Product Process

21. The Lean Product Process

22. The Lean Product Process

23. The Lean Product Process

24. The Lean Product Process

26. Don’t Just Throw Spaghetti

27. Local vs. Global Maximum

28. Theory Experiment Data
Evidence
Supports
or refutes
A/B Testing Isn’t Enough
Revise
You need to understand WHY
A outperformed B
Your mental model
of causality

29. Copyright © 2019 @danolsen

30. Identify highest
ROI idea
Design and
Implement
Analyze How
the Metric
Changes
Brainstorm
Ideas to
Improve Metric
Copyright © 2019 @danolsen
Lean Product Analytics Process
Identify What
Your Metrics Are
Measure Metrics
Baseline Values
Evaluate Metrics
Upside Potential
Global
Level
Metric
Level
Select
Top Metric
Learn
& Iterate

31. Valuable to Have a Holistic Analytics
Framework: Dave McClure’s AARRR!

32. Valuable to Have a Holistic Analytics
Framework: Dave McClure’s AARRR!

33. Valuable to Have a Holistic Analytics
Framework: Dave McClure’s AARRR!

34. Valuable to Have a Holistic Analytics
Framework: Dave McClure’s AARRR!

35. Valuable to Have a Holistic Analytics
Framework: Dave McClure’s AARRR!

36. At any point, one metric offers the
highest return-on-investment (ROI)
“Metric that Matters Most”
(MTMM)
Focus on right metric at right time
Copyright © 2019 @danolsen

37. If you just launched a new product,
which would you optimize first?
Acquisition: attracting prospects
Conversion: turning prospects into customers
Retention: ensuring customers remain active
Copyright © 2019 @danolsen

38. If you could track only 1 metric to measure
Product-Market Fit, which metric would it be?
Copyright © 2019 @danolsen

39. Retention Rate
n Retention rate tracks what % of your
customers are still active over time
~80%
never use
app again
Curve either
goes to zero
or flattens out

40. Cohort Analysis
Copyright © 2019 @danolsen

41. Improving Retention Rate Over Time=
Increasing Product-Market Fit

42. Improving Retention Rate Over Time=
Increasing Product-Market Fit

43. Improving Retention Rate Over Time=
Increasing Product-Market Fit

45. Profit = Revenue - Cost
Unique Visitors x Ad Revenue per Visitor
Impressions/Visitor x Effective CPM / 1000
Visits/Visitor x Pageviews/Visit x Impressions/PV
New Visitors + Returning Visitors
Unique Visitors in Last Period x Monthly Retention Rate
Define the Equation of your Business
Peeling the Onion
Advertising Business Model:
Copyright © 2019 @danolsen

47. View Each Metric as a Gauge
Copyright © 2019 @danolsen
Minimum
Possible
Value
Maximum
Possible
Value
Current
Value

48. Identifying Your Metric that Matters Most
n What is the upside potential of each metric?
n How much do we think we can “move the needle”?
n What would the revenue impact be?
n How many resources will it take to move the needle”?
n Developer-days, time, money
n Which metric offers best Return on Investment (ROI)?
Copyright © 2019 @danolsen

49. Copyright © 2019 @danolsen
Prioritizing Product Ideas by ROI
Investment (developer-weeks)
Return(ValueCreated)
Idea C
Idea B
Idea D
Idea A
Idea F
1
1
2 3 4
2
3
4
?

50. Types of Metric ROI SituationsReturn
Investment
Return
Investment
Return
Investment
Metric A
Good ROI
Metric B
Bad ROI
Metric C
Great ROI
Copyright © 2019 @danolsen

51. Case Study
Copyright © 2019 @danolsen

52. • Which metric has highest ROI opportunity?
Case Study:
Optimizing Friendster’s Viral Loop
Active
Users
Prospective
Users
Invite Click
Succeed
Invite
click-through rate
Conversion
rate
Don’t
Click
Fail
Invites per
sender
% of users
sending
invites
• Multiplied together, these metrics determine your viral ratio
Users
% of users
who are
active
= 15%
= 2.3
= 85%
Registration
Process
Copyright © 2019 @danolsen

53. The Upside Potential of a Metric
0
100%
0
100%
0
?
Registration
Process Yield
% of users sending
invitations
Avg # of invites
sent per sender
2.3
85%
15%
Max possible
improvement
0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
Copyright © 2019 @danolsen

54. Is anyone feeling a
sense of déjà vu
right now?

55. The Upside Potential of a Metric
0
100%
0
100%
0
?
Registration
Process Yield
% of users sending
invitations
Avg # of invites
sent per sender
2.3
85%
15%
Max possible
improvement
0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
Copyright © 2019 @danolsen
Metric B
Bad ROI
Metric A
Good ROI
Metric C
Great ROI

56. Okay, so how can we improve the metric?
n How do we increase the average number of
invites being sent out per sender?
n For each idea:
n What’s the expected benefit? (how much will it
improve the metric?)
n What’s the expected cost? (how many engineer-
days will it take?)
n You want to identify highest ROI idea
Copyright © 2019 @danolsen

57. After Launching Address Book Importer…
Copyright © 2019 @danolsen

58. After Launching Address Book Importer…
Copyright © 2019 @danolsen

59. After Launching Address Book Importer…
Copyright © 2019 @danolsen

60. Identify highest
ROI idea
Design and
Implement
Analyze How
the Metric
Changes
Brainstorm
Ideas to
Improve Metric
Copyright © 2019 @danolsen
Lean Product Analytics Process
Identify What
Your Metrics Are
Measure Metrics
Baseline Values
Evaluate Metrics
Upside Potential
Metric
Level
Select
Top Metric
Learn
& Iterate
Global
Level

61. Upping Your Experimentation Game
n Simultaneous tests vs. before-and-after
n Rollout to any % of users vs. only 100%
Feature flags
n Testing multiple variants vs. 1 alternative
n Experimentation platform: buy vs. build
n Experimentation velocity learning velocity
Copyright © 2019 @danolsen

62. Ken Norton, Google Ventures
Today: Thu March 28th in Mountain View
What Product Teams Can Learn from Jazz Musicians

63. Thanks!
@danolsen
https://dan-olsen.com
Lean Product Meetup:
meetup.com/lean-product
Happy to connect on
LinkedIn