The document provides an overview of using metrics to make better decisions for startups. It discusses moving from tracking vanity metrics that are easy to measure to actionable metrics that matter. An example is given of a SaaS web app that tracks key metrics like sign-ups, activations and upgrades at each step. The actual baseline results are compared to assumptions, and different tests are discussed, like a video test that decreased conversions and an inspired test that increased revenue. The importance of learning faster through tests and data is emphasized, as is measuring faster and minimizing the time to get customers through the process.
12. Original Hypothesis for KISSinsights
“ Our hypothesis is that product manager type people
have a problem doing fast/effective/frequent
customer research.”
13. Are the metrics you are
tracking helping you make
decisions?
14. Example SaaS Web App
• People sign up and pay online
• They first sign up for a 7 day free trial
• Then they pay $97 per month
15. Example SaaS Web App
• People sign up and pay online
• They first sign up for a 7 day free trial
• Then they pay $97 per month
What are your assumptions?
16. Example SaaS Web App Macro Steps
Visit Site Sign Up Activate Upgrade
17. Software as a Service Product
(Assumptions)
Key Actions % of people # of people
Visit Site 100% 1000
Sign Up 10% 100
Activate 5% 50
Upgrade 1% 10
Revenue $970 per month
19. Software as a Service Product
(Actual Baseline)
Key Actions % of people # of people
Visit Site 100% 1000
Sign Up 3.6% 36
Activate 1.2% 12
Upgrade 0.4% 4
Revenue $388 per month
20. What would you try to improve?
Key Actions % of people # of people
Visit Site 100% 1000
Sign Up 3.6% 36
Activate 1.2% 12
Upgrade 0.4% 4
Revenue $388 per month
21. Software as a Service Product
(Video Test)
Key Actions % of people # of people
Visit Site 100% 1000
Sign Up 6.5% 65
Activate 1.8% 18
Upgrade 0.3% 3
Revenue $291 per month vs. $388 baseline :(
23. Software as a Service Product
(Inspired Test)
Key Actions % of people # of people
Visit Site 100% 1000
Sign Up 7.7% 77
Activate 2.8% 28
Upgrade 1.2% 12
Revenue $1164 per month vs. $388 baseline :)
24. Software as a Service Product
(Compare all 3)
Key Actions Baseline Video Test Inspired Test
Visit Site 100% 100% 100%
Sign Up 3.6% 6.5% 7.7%
Activate 1.2% 1.8% 2.8%
Upgrade 0.4% 0.3% 1.2%
Revenue $388 per $291 per $1164 per
month month month
25. The Startup Pyramid
The Race to the Top
Growth
Transition to Growth
Product / Market Fit
Sean Ellis: Startup-Marketing.com
26. AARRR
Created by Dave McClure - 500startups.com Designed by
1. ACQUISITION
Customers come from various channels
2. ACTIVATION
Customers use product for first time
YOUR PRODUCT
3. RETENTION 4. REFERRAL
Customers come back multiple times Customers refer others
5. REVENUE
Customers make your business money
27. Minimize TOTAL time through the loop
IDEAS
Learn Faster Build Faster
LEARN BUILD Unit Tests
Split Tests
Customer Interviews Usability Tests
Customer Development Continuous Integration
Five Whys Root Cause Incremental Deployment
Analysis Free & Open-Source
Customer Advisory Board Components
Falsifiable Hypotheses DATA CODE Cloud Computing
Product Owner Cluster Immune System
Just-in-time Scalability
Accountability
MEASURE Refactoring
Customer Archetypes Measure Faster
Cross-functional Teams Developer Sandbox
Semi-autonomous Teams Split Tests Funnel Analysis Minimum Viable Product
Smoke Tests Clear Product Owner Cohort Analysis
Continuous Deployment Net Promoter Score
Usability Tests Search Engine Marketing
Real-time Monitoring Real-Time Alerting
Customer Liaison Predictive Monitoring
Eric Ries: StartupLessonsLearned.com
29. No Departments, Just Two Teams
?
Problem Team
What are the
☼
Solution Team
Find the
most important minimum solution
problems to solve? for a given problem.
How many of your know how to drive?\nHow many of you would like to drive this car?\n
How about blind-folded at 152 miles per hour?\nWell, that’s what most of us have been doing with our startups!\nSo...\n
We crash and burn.\nYou might even crash and burn if you aren’t blind-folded!\nSame with a startup, it’s EASY to fail. We all know that.\n
What’s easy?\n
It’s easy to add Google Analytics to your site and see reports.\nI am sure, most of our Google Analytics graphs don’t look like this.\nTraditional web analytics shows you page views, visits, page per visit, time on site, etc...\nThese are all easy metrics to get.\n
It’s also easy to track click through rates on ads.\n
Measuring what’s easy will increase you likelihood of failure.\nYou’ll crash and burn and end up with a startup just like this Ferrari.\n
Measuring what matters will increase your likelihood of success.\n
Vanity metrics are easy, they make you feel good.\nBest case = Up and to the right\nEveryone trying to take credit for it.\nEither way, what do you do next?\n
Actionable metrics point out problems and lead to solutions.\nThey make your learn faster and smarter about your business.\n
“Our hypothesis is that product manager type people have a problem doing fast/effective/frequent customer research.”\nQuestion: Is this hypothesis in fact true or not?!\nTake the least amount of effort to find out...\nWe did everything from customer interviews, fake landing pages, building a minimum viable product, pivoting and now have reach product / market fit and are optimizing the business.\n
How do you know what metrics matters? \nWhen I hear a metric from someone, I always try to ask them what decision that metric helps them make.\n
People sign up and pay online\nSign up for a 7 day free trial\nPay $97 per month\n
What would you measure?\nWhat would you optimize?\nSign up? Page views? Time on site? Bounce rate? CTR on ads?\nNext step is to write these down...\n\n
\n
Create a funnel out of the user actions and add numbers.\nThese user actions and numbers are your assumptions.\n
Get real data!\n
This is the funnel after implementing tracking for each step.\nBased on the original assumptions of 1% of people upgrading...\nWe’re not there yet. So...\n\n
Try to understand your riskiest point.\nUsually if people aren’t purchasing you want to focus on that.\nIn this example, people were purchasing, but the sign up is really low.\nSo we tried...\n
Adding a video to our homepage.\nIt increased the sign up rate, which was great.\nBut it decreased revenue :(\nFocusing on all the actionable metrics enabled us to see this.\nSo...we got inspired...\n
\n
The sign up rate increased, and the revenue increased dramatically.\n\n
You want to make sure you can compare the groups of people for these tests.\n
Sean Ellis and his 3 stages of a startup\n\nWe use the Startup Pyramid to remind us where we are and what our success metrics should be.\n\nProduct / Market Fit, Transition to Growth, Growth\n
AARRR!!! Dave McClure’s pirate metrics.\n\nHelps you frame key areas of your business based on a customer’s lifecycle.\n\nWe use this for idea generation.\n
And of course, Eric’s lean startup graphic.\n\nIt’s about validated learnings about customers and increasing cycle time.\n\nSo we can build something a LOT of people want to buy.\n
The principles of build, measure, learn help us think about how to get answers.\nThe methods and process of communication and information exchange are much different in a data-driven startup.\nThe tactical stuff to build, measure and learn faster is one side of things.\nNow I’m going to talk about the business processes and what we have learned trying to apply them.\nBefore continuing...clarify business process for a startup.\n\n
Company is divided into two teams. No sales, marketing, etc...\nProblem team identifies the most important problems to solve.\nSolution team finds the minimum solution for each problem.\nProblem team is more non-technical, solution team is more technical / “makers”. There are cross-functional folks too.\n
3 team meetings every week\nMonday - Team meeting to describe most important problems and goals for the week. Metrics + Few tasks, determine if solution team has input and let deadline for ideas on Tuesday.\nWednesday - Check in on how are we doing, solutions already getting implemented? What have we learned so far?\nFriday - Postmortem on learning and peek into next week from the Problem Team.\nProblem team is 1-week ahead.\n
To do this effectively we built a tool we are internally call KISSfocus.\nA tool we built to increase autonomy and communication, focused on our validated learnings driven process.\n5 tasks per person, only 1 is active (green-lighted), not in KF then not exist, backlog.\nWe set weekly goal and metrics based on most important problems.\nWhole team has visibility across all tasks, keeps solution team focused on most important problems.\n