1.
Using Lean Analytics
for Startups and
Enterprises
Ben Yoskovitz | @byosko

2.
Introduction
@byosko
I am a
product guy
entrepreneur
author
angel investor

3.
Find me online
Blog: http://instigatorblog.com
Slideshare: http://slideshare.net/LeanAnalytics
Book: http://leananalyticsbook.com
Email: byosko@gmail.com
@byosko

4.
CORPORATE PARTNERS
VENTURE-BACKABLE FOUNDERS
PRE-SEED FUNDING
BETTER STARTUPS
+
+
=
Highline BETA is a startup co-creation company that
launches new ventures with leading corporations.
http://highlinebeta.com @byosko

5.
Metrics:
The Fundamentals

6.
Metrics: The fundamentals
● How data fits in
● What makes a good metric
● Types of metrics
● Analytical superpowers
@byosko

7.
How to get things built properly (in theory)

8.
Everyone has great ideas,
right?
People love this part
(but that’s not always
a good thing!)
This is where things start to
fall apart.
No data, no learning.
Build Measure Learn seems so easy!

9.
INTELLECTUALLY HONESTY
Follow the Lean model and it becomes
increasingly hard to lie, especially to yourself.

10.
FOCUS
Don’t chase shiny objects. You might
succeed without focus, but it’ll be by accident.

11.
BETTER DECISION MAKING
Everyone has data. The key is figuring
out what pieces will improve your learning
and decision making.

12.
USE YOUR GUT PROPERLY
Instincts are experiments.
Data is proof.

13.
So what makes a good metric?

14.
Question: What are the metrics you’re tracking?
● Take 2 minutes to write down the key
metrics you’re tracking (or your business is
tracking) right now.
● These could be at a business level or project
level.
● At the end of this section we can re-evaluate
if the metrics you’re tracking are still the right
ones.
@byosko

15.
WHAT IS ANALYTICS?
Analytics is the measurement of movement
towards business goals.

16.
A good metric is:
Understandable
If you’re busy
explaining the data,
you won’t be busy
acting on it.
Comparative
Active Users vs.
Active Users/
month
Ratio / Rate
% Monthly Active
Users
Behavior
Changing
You’ll know how
you’ll change your
business based on
what the metric tells
you.
@byosko

17.
If a metric won’t
change how you
behave, it’s a
bad
metric.
THE
GOLDEN
RULE OF
METRICS
http://www.flickr.com/photos/circasassy/7858155676/

18.
Acquisition1-15%
Low cost of
acquisition, high
checkout
Customers that buy
>1x in 90d
Then you are in this
mode
Your customers will
buy from you
You are just
like Focus on
15-30%
>30%
Hybrid
Loyalty
Once
2-2.5
>2.5
per year
per year
70%
20%
10%
of retailers
of retailers
of retailers
Increasing return
rate, market share
Loyalty, selection,
inventory size
(Thanks to Kevin Hillstrom for this.)
Metrics help you know yourself:

19.
Types of Metrics

20.
Vanity vs. Actionable metrics
Vanity Actionable
Makes you feel good but
doesn’t change how
you’ll act.
Helps you pick a
direction and change
your behavior.
“Up and to the right.” These are good.
@byosko

21.
Beware of vanity metrics:
Users
Follows / friends
/ likes
Logins
This tells you nothing about what they did, why
they stuck around, or why they left.
Count actions instead. Count how many followers
will do your bidding.
What are they actually doing when they login?
Logins don’t tell you about actions and value.
Downloads
Sure, people need to download your app in order
to use it, but so what?
@byosko

22.
The best (worst!) vanity metric of all time…
# of Features
@byosko
https://www.flickr.com/photos/pinoyed/5009440499

23.
Qualitative vs. Quantitative metrics
Qualitative Quantitative
Unstructured, anecdotal,
revealing, hard to
aggregate.
Numbers and stats; hard
facts, but less insights.
Warm and fuzzy. Cold and hard.
@byosko

24.
Discover qualitatively.
Prove quantitatively.

25.
Do Airbnb hosts get more
business if their property
is professionally
photographed?

26.
Gut instinct (hypothesis)
Professional photography helps Airbnb’s business
Concierge MVP
Sent 20 photographers out into the field
Measure the results
Compared photographed listings to control group
Make a decision
Launched photography as a new feature to all hosts
CASE STUDY
Do professional
photos make a
difference?

27.
Exploratory vs. Reporting metrics
Exploratory Reporting
Speculative. Tries to find
unexpected or interesting
insights. Source of unfair
advantages.
Predictable. Keeps you
abreast of normal, day-to-
day operations. Can be
managed by exception.
Cool. Necessary.
@byosko

28.
! Started as Circle of Friends
! Leveraged Facebook early
! Grew to 10M users fast
ENGAGEMENT SUCKED!
CASE STUDY
Finding insights in the data

29.
ENGAGEMENT SOLVED.
CASE STUDY
Moms are crazy! (in a good way)
! Messages to one another were ~50% longer
! 115% more likely to attach a picture to a post
! 110% more likely to engage in a threaded conversation
! Invited friends were 50% more likely to become engaged users
! 60% more likely to accept invitations to the app

30.
Lagging vs. Leading metrics
Lagging Leading
Historical metric that
shows you how you’re
doing: reports the news.
Number today that shows
a metric tomorrow:
makes the news.
Start here. Try and get here.
@byosko

31.
Examples of leading metrics
A Facebook user reaching 7 friends within 10 days of signing up.
(Chamath Palihapitiya)
A Dropbox user who puts at least 1 file in 1 folder on 1 device. (ChenLi
Wang)
A Twitter user who follows a certain number of people, and a certain
percentage of those people follow the user back. (Josh Elman)
A LinkedIn user getting to X connections in Y days. (Elliot Schmukler)
@byosko

32.
1. People who install the Chrome extension
2. People who connect more than 1 social account
3. People who share 15 pieces of content in 7 days
CASE STUDY
Buffer discovered 3 leading metrics

33.
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Correlation vs. causation

34.
Correlated vs. Causal metrics
Correlated Causal
Two variables that are
related (but may be
dependent on something
else.)
An independent variable
that directly impacts a
dependent one.
Ice cream and
drowning.
Summertime and drowning /
Summertime and eating ice cream
@byosko

35.
A leading, causal metric
is a superpower.

36.
Causality is a superpower because it lets you change the future.
Correlation lets you
predict the future
Causality lets you
change the future
“I will have 420 engaged users and
75 paying customers next month.”
“If I can make more first time visitors
stay for 17 minutes I will increase
sales in 90 days.”
Pick a metric
to change
Find
correlation
Test for
causality
Optimize the
causal factor
@byosko

37.
Cohort analysis
https://blog.kissmetrics.com/cohort-and-multi-touch-attribution/
@byosko

38.
Ricky (product manager) has
some ideas for improving the
“Proposal Send Screen” (based
on qualitative feedback & his
gut), but before prioritizing this
work, he digs into the data.
http://proposify.biz
Putting basic data to use

39.
50% of people send
proposals through Proposify
(50% don’t) (quantitative)
— Is this good or bad?
Putting basic data to use
http://proposify.biz

40.
Ricky isn’t sure. So he’s going
to need to look at additional
data (exploratory):
• Churn
• Proposal won rate
• Any correlations here?
Putting basic data to use
http://proposify.biz

41.
@byosko
• Also needs to do more direct
customer development to
learn more (qualitative)
• All of this work might lead to
additional, meaningful
product dev (actionable)
Putting basic data to use
http://proposify.biz

42.
Look back at the metrics you’re tracking
● Remember the metrics you wrote down
earlier? How do they stack up now? Are
they good metrics?
● What might you change about the
metrics you’re tracking as a business
and/or on a project/feature level?
@byosko

43.
Quick summary on the basics of analytics
● Analytics is about measuring movement
towards business goals
● Analytics is about simplifying not complicating
● Analytics is about helping you focus on what
really matters
● Remember the Golden Rule: A good metric
has to change your behaviour
@byosko

44.
Measuring Success:
An introduction to
Lean Analytics

45.
Lean Analytics Framework
● The five stages of business & product development
● Mapping business models
● The One Metric That Matters (KPIs)
● The Lean Analytics Cycle
@byosko

46.
Two keys: the Business you’re in & the Stage you’re at
What business are
you in?
What stage are you
at?
! E-Commerce
! SaaS
! Free Mobile App
! 2-Sided Marketplace
! Media
! User-Generated Content
! Empathy
! Stickiness
! Virality
! Revenue
! Scale
@byosko

47.
Big companies need one more thing.
An understanding of what type
of innovation they’re doing.

48.
Core Adjacent Transformative
Do the same thing
better.
Nearby product, market,
or method.
Start something
entirely new.
Regional
optimizations.
Innovation, go-to-market
strategies.
Reinvent the business
model.
• Get there faster
• Smaller batches
• Solution, then testing
• Increased accountability
• Customer development
• Test similar cases
• Parallel deployment
• Analytics & cycle time
• Fail fast
• Skunkworks/R&D
• Focus on the search
• Ignore the current model &
margins
Many models for enterprise innovation

49.
Know the problem
(customers tell you it)
Know the solution
(customers/regulations/
norms dictate it.)
Know the problem (market
analysis)
Don’t know the solution
(non-obvious innovation
confers competitive
advantage.)
Don’t know the problem (just
an emerging need/change)
Don’t know the solution.
Waterfall:
Execution
matters
Agile/scrum:
Iteration matters
Lean Startup:
Discovery
matters
Another way to look at it
Core Adjacent Transformative

50.
Current
state
Business
optimization
Product,
market,
method
innovation
Business
model
innovation
You can convince
executives of this
because some of it
is familiar.
This terrifies them
because it eats the
current business.
A three-maxima model for enterprise
innovation

51.
Improvement Adjacency Remodelling
Do the same,
only better.
Explore what’s
nearby quickly
Try out new
business models
Lean approaches apply, but the metrics vary widely.
Sustain /
core
Innovate /
adjacent
Disrupt /
transformative

52.
Sustaining Adjacent Disruptive
Next year’s car Electric car,
same dealer
On-demand, app-based
car service

53.
So the metrics that matter to a big
company are dependent on the
type of innovation being done.

54.
Stages of business & product
development

55.
Eric’s three engines of growth
Stickiness Virality Price
Approach
Math that
matters
Keep people
coming back.
Get customers
faster than you
lose them.
Make people
invite friends.
How many they
tell, how fast they
tell them.
Spend money to
get customers.
Customers are
worth more than
they cost.
@byosko

56.
Dave McClure’s Pirate Metrics

57.
Dave McClure’s Pirate Metrics
Acquisition
Activation
Retention
Referral
Revenue
How do your users become aware of you?
Do drive-by visitors subscribe, use, etc.?
Does a one time user become engaged?
Do users promote your product?
Do you make money from user activity?

58.
The Lean Analytics Stages
Empathy You’ve found a real, poorly-met need that a reachable
market faces.
You’ve figured out how to solve the problem in a way that
users will adopt, keep using and pay for.
Your users and features fuel growth organically and
artificially.
You’ve found a sustainable, scalable business with the
right margins in a healthy ecosystem.
STAGE GATE
Stickiness
Virality
Revenue
Scale

59.
The Lean Analytics Stages
Empathy You’ve found a real, poorly-met need that a reachable
market faces.
You’ve figured out how to solve the problem in a way that
users will adopt, keep using and pay for.
Your users and features fuel growth organically and
artificially.
You’ve found a sustainable, scalable business with the
right margins in a healthy ecosystem.
STAGE GATE
Stickiness
Virality
Revenue
Scale
Most products (and
startups) fail at this
point.

60.
CASE STUDY
! Stage: Empathy/Stickiness
! Model: E-Commerce
! Originally tied to Instagram with
an “Insta-Order” feature
Jumping the gun on product development

61.
Optimize for 1st time purchases or repeat
orders?
WITH INSTA-ORDER
Click checkout
Confirmation page
Confirm order
Success page
Sign in to PayPal
Back to PayPal
Authorized pre-approved
payments
WITHOUT INSTA-ORDER
Click checkout
Sign in to PayPal
Confirmation page
Confirm order
Success page
● 2x transactions
● Lower bounce rate
● Sign-in goals increased

62.
“THERE ARE NO
SHORTCUTS TO ANY
PLACE WORTH GOING.”
- Beverly Sills

63.
Mapping business models

64.
Does recurring
revenue work
for everyone?
CASE STUDY
@byosko

65.
The leader in predictive analytics for people. Clearfit
helps thousands of companies build better teams. As
featured in:
CASE STUDY
10x
revenue increase
off of 3x in sales
volume
“People don’t do subscriptions for haircuts, hamburgers or
hiring. You have to understand your customer, who they are,
how and why they buy, and how they value your product or
service.” - Ben Baldwin

66.
The goal is to understand the customer’s
lifecycle / journey through every
touchpoint with your product.

67.
Paid Direct WOM Search
Inherent
virality
Customer Acquisition Cost
VISITOR
User
FORMER USERS
Engaged user
Reactivate Trial over
Invite others
Paying customer
Disengaged
Account cancelled
Freemium / trial offer
Enrollment
Disengaged user
Cancel Cancel
Reactivate
FORMER CUSTOMERS
Billing info exp.
Resolution
Dissatisfied
Capacity Limit
Upselling
Signup conversion
rate
Free user disengagement
Freemium churn
Reactivation
rate
User lifetime value Customer lifetime value
Trial abandonment rate
DAU/WAU/MAU
Paid
conversion
Viral coefficient
Viral rate
Paid churn rate
Support data
Tiering
Upselling rate
SaaS Customer
Lifecycle

68.
Returning Paid Direct Search Viral
Customer Acquisition Cost
VISITOR
E-Commerce
Customer
Lifecycle
Navigation Search Reco Engine
1-time buyer
Cart
Additions
Conversion
Logistics, delays
Delivery
Enrollment
Call to Action
Sharing
Unsocial buyer
Sharing rate
Returning
rate
Customer Lifetime Value
Open rate, engagement
Transaction size
Emphasis on maximizing cart
value, minimizing acquisition
costs
Bounced
Not interested
Abandoned
Bounce rate
Unsatisfied
Ratings, delivery
issues
Feature usage,
product
discovery

69.
CASE STUDY
A
A/B testing what really matters
B

70.
CASE STUDY
B
! 41% increase in revenue
per customer! (People
bought a lot more product.)
! Conversion also went up,
but was secondary in
importance.

71.
All business models have issues
CAC vs. LTV -- margins are usually very small. A $10M e-commerce business is
small.
Freemium requires tens of millions of free users. They can be expensive to
support. Will enough convert?
The average # of apps downloaded by North Americans per month is now 0.
Monetizing is incredibly hard. Popularity is fleeting.
Chicken & egg problem. Supply and demand. How do you build up both
enough?
Real monetization requires hundreds of millions of engaged visitors. People’s
attention is hard to capture and keep.
Content creation. Will it be good enough? Will enough people do it? Why?
E-Commerce
SaaS
Mobile Apps
2-sided Marketplace
Media
UCG
@byosko

72.
You know what business you’re in.
You know what stage you’re at.
NOW WHAT?

73.
The One Metric That Matters
The business you’re in
E-Com SaaS Mobile 2-Sided Media UCG
Thestageyou’reat
Empathy
Stickiness
Virality
Revenue
Scale
THE ONE METRIC
THAT MATTERS
@byosko

74.
What really matters when you’re
backing up your car?

75.
Moz cuts down on metrics to track
SaaS-based SEO toolkit in the Scale stage.
Focused on net adds.
Net adds up:
Was a marketing campaign successful?
Were customer complaints lowered?
Was a product upgrade valuable?
Net adds flat:
Can we acquire more valuable customers?
What product features can increase engagement?
Can we improve customer support?
Net adds down:
Are the new customers not the right segment?
Did a marketing campaign fail?
Did a product upgrade fail somehow?
Is customer support falling apart?

76.
Timehop only cares about virality
! Focused on % of daily active users
that share content
! Aiming for 20-30% of daily active
users to share content
“All that matters now is virality. Everything else--be it
press, publicity stunts or something else--is like pushing a
rock up a mountain: it will never scale. But being viral
will.” -- Jonathan Wegener, founder

77.
# of transactions (for
merchants)
# of nights booked sales
total time reading
https://medium.com/data-lab/mediums-metric-that-matters-total-time-reading-86c4970837d5#.tidx5bunj
http://quibb.com/links/metrics-to-inform-your-model-lessons-from-square-stripe-and-quora
http://500.co/aircall-growth-uber/
monthly active users monthly recurring revenue
(MRR)
Examples of OMTM
@byosko

78.
www.flickr.com/photos/connortarter/4791605202/
METRICS ARE
LIKE SQUEEZE
TOYS

79.
Better: http://bit.ly/BigLeanTable

80.
The Layer Cake of Metrics
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Department OMTM Department OMTM Department OMTM
OMTM: Business Help Indicator

81.
What’s your OMTM?
● So what’s your OMTM? Do you know?
Can you write it down? Is it available to
everyone at your company?
● Can you see how your work matters to the
overall health of the business and how you
might measure that value creation?
@byosko

82.
Drawing lines in the sand

83.
Growth
5% / week (revenue or
active users)
Time on site
17 minutes
Free to paid
2% of free users
Mobile file size
< 50MB
Engaged visitors
30% monthly users
10% daily users
Paid load time
< 5 seconds
Churn
2% / month
CLV:CAC
3:1
Some benchmarks
@byosko

84.
CASE STUDY: Solare draws a line in the sand
@byosko

85.
50 reservations by 5pm 250 covers that night
=
CASE STUDY: Solare discovers a leading indicator
@byosko

86.
The Lean Analytics Cycle

87.
Identify a key business problem,
pick the OMTM, draw a line in
the sand, and get started.

88.
Draw a new line
ZxLERATOR | NYC | SUMMER 2016 89
LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure the
results
Make changes in
production
Design a test
Hypothesis
With data:
find a commonality
Without data:
make a good guess
Find a potential
improvement
Draw a linePick a OMTM
Lean Analytics Cycle

89.
Quick summary on the Lean Analytics framework
● What you track depends on what type of innovation
you’re doing: core, adjacent or disruptive
● What you track depends on your business model and
stage (for a startup, project, product or even at a
feature-level)
● Find the One Metric That Matters so you can focus as
much as possible
● The more holistically you can assess your business, the
better off you’ll be (map it all and find the hot spots!)
@byosko

90.
The value of data
in building
better products.

91.
Data is a key input and filter in
building better products.

92.
COMPETITION,
OTHER PRODUCTS,
BEST PRACTICES
BUILDLEARN
IDEAS
CORPORATE
GOALS (SOME GOOD,
SOME BAD)
GUTS & INSTINCTS
PARTNERS
OTHER
DEPARTMENTS
INDUSTRY
TRENDS, ETC.
DATA
CUSTOMER INPUT
DATA

93.
COMPETITION,
OTHER PRODUCTS,
BEST PRACTICES
PARTNERS
INDUSTRY
TRENDS, ETC.
GUTS & INSTINCTS
OTHER
DEPARTMENTS
CORPORATE GOALS
DATA
AS A
FILTER
BETTER
DECISIONS
CUSTOMER INPUT

94.
Product & Design
(defining goals /
objectives)
User & customer
feedback
Sales
Marketing
Customer Support
Etc.
! In-person interviews
! Surveys
! Customer support
inquiries
! Real-time online
Supported
by data
Your gut
Company vision
Collecting Input & Customer Discovery
Your own ideas

95.
Data is also a
communication tool.
http://www.instigatorblog.com/data-common-language/2016/09/22/

96.
@byosko @byosko

97.
Data is complex.
How we communicate it
doesn’t have to be.

98.
Alistair Croll
acroll@gmail.com
@acroll
Ben Yoskovitz
byosko@gmail.com
@byosko