Lean Analytics for Startups and Enterprises

1. Using Lean Analytics for Startups and Enterprises Ben Yoskovitz | @byosko

2. Introduction @byosko I am a product guy entrepreneur author angel investor

3. Find me online Blog: http://instigatorblog.com Slideshare: http://slideshare.net/LeanAnalytics Book: http://leananalyticsbook.com Email: byosko@gmail.com @byosko

4. CORPORATE PARTNERS VENTURE-BACKABLE FOUNDERS PRE-SEED FUNDING BETTER STARTUPS + + = Highline BETA is a startup co-creation company that launches new ventures with leading corporations. http://highlinebeta.com @byosko

5. Metrics: The Fundamentals

6. Metrics: The fundamentals ● How data fits in ● What makes a good metric ● Types of metrics ● Analytical superpowers @byosko

7. How to get things built properly (in theory)

8. Everyone has great ideas, right? People love this part (but that’s not always a good thing!) This is where things start to fall apart. No data, no learning. Build Measure Learn seems so easy!

9. INTELLECTUALLY HONESTY Follow the Lean model and it becomes increasingly hard to lie, especially to yourself.

10. FOCUS Don’t chase shiny objects. You might succeed without focus, but it’ll be by accident.

11. BETTER DECISION MAKING Everyone has data. The key is figuring out what pieces will improve your learning and decision making.

12. USE YOUR GUT PROPERLY Instincts are experiments. Data is proof.

13. So what makes a good metric?

14. Question: What are the metrics you’re tracking? ● Take 2 minutes to write down the key metrics you’re tracking (or your business is tracking) right now. ● These could be at a business level or project level. ● At the end of this section we can re-evaluate if the metrics you’re tracking are still the right ones. @byosko

15. WHAT IS ANALYTICS? Analytics is the measurement of movement towards business goals.

16. A good metric is: Understandable If you’re busy explaining the data, you won’t be busy acting on it. Comparative Active Users vs. Active Users/ month Ratio / Rate % Monthly Active Users Behavior Changing You’ll know how you’ll change your business based on what the metric tells you. @byosko

17. If a metric won’t change how you behave, it’s a bad metric. THE GOLDEN RULE OF METRICS http://www.flickr.com/photos/circasassy/7858155676/

18. Acquisition1-15% Low cost of acquisition, high checkout Customers that buy >1x in 90d Then you are in this mode Your customers will buy from you You are just like Focus on 15-30% >30% Hybrid Loyalty Once 2-2.5 >2.5 per year per year 70% 20% 10% of retailers of retailers of retailers Increasing return rate, market share Loyalty, selection, inventory size (Thanks to Kevin Hillstrom for this.) Metrics help you know yourself:

19. Types of Metrics

20. Vanity vs. Actionable metrics Vanity Actionable Makes you feel good but doesn’t change how you’ll act. Helps you pick a direction and change your behavior. “Up and to the right.” These are good. @byosko

21. Beware of vanity metrics: Users Follows / friends / likes Logins This tells you nothing about what they did, why they stuck around, or why they left. Count actions instead. Count how many followers will do your bidding. What are they actually doing when they login? Logins don’t tell you about actions and value. Downloads Sure, people need to download your app in order to use it, but so what? @byosko

22. The best (worst!) vanity metric of all time… # of Features @byosko https://www.flickr.com/photos/pinoyed/5009440499

23. Qualitative vs. Quantitative metrics Qualitative Quantitative Unstructured, anecdotal, revealing, hard to aggregate. Numbers and stats; hard facts, but less insights. Warm and fuzzy. Cold and hard. @byosko

24. Discover qualitatively. Prove quantitatively.

25. Do Airbnb hosts get more business if their property is professionally photographed?

26. Gut instinct (hypothesis) Professional photography helps Airbnb’s business Concierge MVP Sent 20 photographers out into the field Measure the results Compared photographed listings to control group Make a decision Launched photography as a new feature to all hosts CASE STUDY Do professional photos make a difference?

27. Exploratory vs. Reporting metrics Exploratory Reporting Speculative. Tries to find unexpected or interesting insights. Source of unfair advantages. Predictable. Keeps you abreast of normal, day-today operations. Can be managed by exception. Cool. Necessary. @byosko

28. ! Started as Circle of Friends ! Leveraged Facebook early ! Grew to 10M users fast ENGAGEMENT SUCKED! CASE STUDY Finding insights in the data

29. ENGAGEMENT SOLVED. CASE STUDY Moms are crazy! (in a good way) ! Messages to one another were ~50% longer ! 115% more likely to attach a picture to a post ! 110% more likely to engage in a threaded conversation ! Invited friends were 50% more likely to become engaged users ! 60% more likely to accept invitations to the app

30. Lagging vs. Leading metrics Lagging Leading Historical metric that shows you how you’re doing: reports the news. Number today that shows a metric tomorrow: makes the news. Start here. Try and get here. @byosko

31. Examples of leading metrics A Facebook user reaching 7 friends within 10 days of signing up. (Chamath Palihapitiya) A Dropbox user who puts at least 1 file in 1 folder on 1 device. (ChenLi Wang) A Twitter user who follows a certain number of people, and a certain percentage of those people follow the user back. (Josh Elman) A LinkedIn user getting to X connections in Y days. (Elliot Schmukler) @byosko

32. 1. People who install the Chrome extension 2. People who connect more than 1 social account 3. People who share 15 pieces of content in 7 days CASE STUDY Buffer discovered 3 leading metrics

33. Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Correlation vs. causation

34. Correlated vs. Causal metrics Correlated Causal Two variables that are related (but may be dependent on something else.) An independent variable that directly impacts a dependent one. Ice cream and drowning. Summertime and drowning / Summertime and eating ice cream @byosko

35. A leading, causal metric is a superpower.

36. Causality is a superpower because it lets you change the future. Correlation lets you predict the future Causality lets you change the future “I will have 420 engaged users and 75 paying customers next month.” “If I can make more first time visitors stay for 17 minutes I will increase sales in 90 days.” Pick a metric to change Find correlation Test for causality Optimize the causal factor @byosko

37. Cohort analysis https://blog.kissmetrics.com/cohort-and-multi-touch-attribution/ @byosko

38. Ricky (product manager) has some ideas for improving the “Proposal Send Screen” (based on qualitative feedback & his gut), but before prioritizing this work, he digs into the data. http://proposify.biz Putting basic data to use

39. 50% of people send proposals through Proposify (50% don’t) (quantitative) — Is this good or bad? Putting basic data to use http://proposify.biz

40. Ricky isn’t sure. So he’s going to need to look at additional data (exploratory): • Churn • Proposal won rate • Any correlations here? Putting basic data to use http://proposify.biz

41. @byosko • Also needs to do more direct customer development to learn more (qualitative) • All of this work might lead to additional, meaningful product dev (actionable) Putting basic data to use http://proposify.biz

42. Look back at the metrics you’re tracking ● Remember the metrics you wrote down earlier? How do they stack up now? Are they good metrics? ● What might you change about the metrics you’re tracking as a business and/or on a project/feature level? @byosko

43. Quick summary on the basics of analytics ● Analytics is about measuring movement towards business goals ● Analytics is about simplifying not complicating ● Analytics is about helping you focus on what really matters ● Remember the Golden Rule: A good metric has to change your behaviour @byosko

44. Measuring Success: An introduction to Lean Analytics

45. Lean Analytics Framework ● The five stages of business & product development ● Mapping business models ● The One Metric That Matters (KPIs) ● The Lean Analytics Cycle @byosko

46. Two keys: the Business you’re in & the Stage you’re at What business are you in? What stage are you at? ! E-Commerce ! SaaS ! Free Mobile App ! 2-Sided Marketplace ! Media ! User-Generated Content ! Empathy ! Stickiness ! Virality ! Revenue ! Scale @byosko

47. Big companies need one more thing. An understanding of what type of innovation they’re doing.

48. Core Adjacent Transformative Do the same thing better. Nearby product, market, or method. Start something entirely new. Regional  optimizations. Innovation, go-to-market strategies. Reinvent the business model. • Get there faster • Smaller batches • Solution, then testing • Increased accountability • Customer development • Test similar cases • Parallel deployment • Analytics & cycle time • Fail fast • Skunkworks/R&D • Focus on the search • Ignore the current model & margins Many models for enterprise innovation

49. Know the problem (customers tell you it) Know the solution (customers/regulations/ norms dictate it.) Know the problem (market analysis) Don’t know the solution (non-obvious innovation confers competitive advantage.) Don’t know the problem (just an emerging need/change) Don’t know the solution. Waterfall:  Execution matters Agile/scrum:  Iteration matters Lean Startup: Discovery matters Another way to look at it Core Adjacent Transformative

50. Current  state Business optimization Product,  market,  method innovation  Business model innovation  You can convince executives of this because some of it is familiar. This terriﬁes them because it eats the current business. A three-maxima model for enterprise innovation

51. Improvement Adjacency Remodelling Do the same,  only better. Explore what’s  nearby quickly Try out new  business models Lean approaches apply, but the metrics vary widely. Sustain / core Innovate / adjacent Disrupt / transformative

52. Sustaining Adjacent Disruptive Next year’s car Electric car,  same dealer On-demand, app-based  car service

53. So the metrics that matter to a big company are dependent on the type of innovation being done.

54. Stages of business & product development

55. Eric’s three engines of growth Stickiness Virality Price Approach Math that matters Keep people coming back. Get customers faster than you lose them. Make people invite friends. How many they tell, how fast they tell them. Spend money to get customers. Customers are worth more than they cost. @byosko

56. Dave McClure’s Pirate Metrics

57. Dave McClure’s Pirate Metrics Acquisition Activation Retention Referral Revenue How do your users become aware of you? Do drive-by visitors subscribe, use, etc.? Does a one time user become engaged? Do users promote your product? Do you make money from user activity?

58. The Lean Analytics Stages Empathy You’ve found a real, poorly-met need that a reachable market faces. You’ve figured out how to solve the problem in a way that users will adopt, keep using and pay for. Your users and features fuel growth organically and artificially. You’ve found a sustainable, scalable business with the right margins in a healthy ecosystem. STAGE GATE Stickiness Virality Revenue Scale

59. The Lean Analytics Stages Empathy You’ve found a real, poorly-met need that a reachable market faces. You’ve figured out how to solve the problem in a way that users will adopt, keep using and pay for. Your users and features fuel growth organically and artificially. You’ve found a sustainable, scalable business with the right margins in a healthy ecosystem. STAGE GATE Stickiness Virality Revenue Scale Most products (and startups) fail at this point.

60. CASE STUDY ! Stage: Empathy/Stickiness ! Model: E-Commerce ! Originally tied to Instagram with an “Insta-Order” feature Jumping the gun on product development

61. Optimize for 1st time purchases or repeat orders? WITH INSTA-ORDER Click checkout Confirmation page Confirm order Success page Sign in to PayPal Back to PayPal Authorized pre-approved payments WITHOUT INSTA-ORDER Click checkout Sign in to PayPal Confirmation page Confirm order Success page ● 2x transactions ● Lower bounce rate ● Sign-in goals increased

62. “THERE ARE NO SHORTCUTS TO ANY PLACE WORTH GOING.” - Beverly Sills

63. Mapping business models

64. Does recurring revenue work for everyone? CASE STUDY @byosko

65. The leader in predictive analytics for people. Clearfit helps thousands of companies build better teams. As featured in: CASE STUDY 10x revenue increase off of 3x in sales volume “People don’t do subscriptions for haircuts, hamburgers or hiring. You have to understand your customer, who they are, how and why they buy, and how they value your product or service.” - Ben Baldwin

66. The goal is to understand the customer’s lifecycle / journey through every touchpoint with your product.

67. Paid Direct WOM Search Inherent virality Customer Acquisition Cost VISITOR User FORMER USERS Engaged user Reactivate Trial over Invite others Paying customer Disengaged Account cancelled Freemium / trial offer Enrollment Disengaged user Cancel Cancel Reactivate FORMER CUSTOMERS Billing info exp. Resolution Dissatisfied Capacity Limit Upselling Signup conversion rate Free user disengagement Freemium churn Reactivation rate User lifetime value Customer lifetime value Trial abandonment rate DAU/WAU/MAU Paid conversion Viral coefficient Viral rate Paid churn rate Support data Tiering Upselling rate SaaS Customer Lifecycle

68. Returning Paid Direct Search Viral Customer Acquisition Cost VISITOR E-Commerce Customer Lifecycle Navigation Search Reco Engine 1-time buyer Cart Additions Conversion Logistics, delays Delivery Enrollment Call to Action Sharing Unsocial buyer Sharing rate Returning rate Customer Lifetime Value Open rate, engagement Transaction size Emphasis on maximizing cart value, minimizing acquisition costs Bounced Not interested Abandoned Bounce rate Unsatisfied Ratings, delivery issues Feature usage, product discovery

69. CASE STUDY A A/B testing what really matters B

70. CASE STUDY B ! 41% increase in revenue per customer! (People bought a lot more product.) ! Conversion also went up, but was secondary in importance.

71. All business models have issues CAC vs. LTV -- margins are usually very small. A $10M e-commerce business is small. Freemium requires tens of millions of free users. They can be expensive to support. Will enough convert? The average # of apps downloaded by North Americans per month is now 0. Monetizing is incredibly hard. Popularity is fleeting. Chicken & egg problem. Supply and demand. How do you build up both enough? Real monetization requires hundreds of millions of engaged visitors. People’s attention is hard to capture and keep. Content creation. Will it be good enough? Will enough people do it? Why? E-Commerce SaaS Mobile Apps 2-sided Marketplace Media UCG @byosko

72. You know what business you’re in. You know what stage you’re at. NOW WHAT?

73. The One Metric That Matters The business you’re in E-Com SaaS Mobile 2-Sided Media UCG Thestageyou’reat Empathy Stickiness Virality Revenue Scale THE ONE METRIC THAT MATTERS @byosko

74. What really matters when you’re backing up your car?

75. Moz cuts down on metrics to track SaaS-based SEO toolkit in the Scale stage. Focused on net adds. Net adds up: Was a marketing campaign successful? Were customer complaints lowered? Was a product upgrade valuable? Net adds flat: Can we acquire more valuable customers? What product features can increase engagement? Can we improve customer support? Net adds down: Are the new customers not the right segment? Did a marketing campaign fail? Did a product upgrade fail somehow? Is customer support falling apart?

76. Timehop only cares about virality ! Focused on % of daily active users that share content ! Aiming for 20-30% of daily active users to share content “All that matters now is virality. Everything else--be it press, publicity stunts or something else--is like pushing a rock up a mountain: it will never scale. But being viral will.” -- Jonathan Wegener, founder

77. # of transactions (for merchants) # of nights booked sales total time reading https://medium.com/data-lab/mediums-metric-that-matters-total-time-reading-86c4970837d5#.tidx5bunj http://quibb.com/links/metrics-to-inform-your-model-lessons-from-square-stripe-and-quora http://500.co/aircall-growth-uber/ monthly active users monthly recurring revenue (MRR) Examples of OMTM @byosko

78. www.flickr.com/photos/connortarter/4791605202/ METRICS ARE LIKE SQUEEZE TOYS

79. Better: http://bit.ly/BigLeanTable

80. The Layer Cake of Metrics Project OMTM Project OMTM Project OMTM Project OMTM Project OMTM Project OMTM Department OMTM Department OMTM Department OMTM OMTM: Business Help Indicator

81. What’s your OMTM? ● So what’s your OMTM? Do you know? Can you write it down? Is it available to everyone at your company? ● Can you see how your work matters to the overall health of the business and how you might measure that value creation? @byosko

82. Drawing lines in the sand

83. Growth 5% / week (revenue or active users) Time on site 17 minutes Free to paid 2% of free users Mobile file size < 50MB Engaged visitors 30% monthly users 10% daily users Paid load time < 5 seconds Churn 2% / month CLV:CAC 3:1 Some benchmarks @byosko

84. CASE STUDY: Solare draws a line in the sand @byosko

85. 50 reservations by 5pm 250 covers that night = CASE STUDY: Solare discovers a leading indicator @byosko

86. The Lean Analytics Cycle

87. Identify a key business problem, pick the OMTM, draw a line in the sand, and get started.

88. Draw a new line ZxLERATOR | NYC | SUMMER 2016 89 LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics Pivot or  give up Try again Success! Did we move the needle? Measure the results Make changes in production Design a test Hypothesis With data:  ﬁnd a commonality Without data: make a good guess Find a potential improvement Draw a linePick a OMTM Lean Analytics Cycle

89. Quick summary on the Lean Analytics framework ● What you track depends on what type of innovation you’re doing: core, adjacent or disruptive ● What you track depends on your business model and stage (for a startup, project, product or even at a feature-level) ● Find the One Metric That Matters so you can focus as much as possible ● The more holistically you can assess your business, the better off you’ll be (map it all and find the hot spots!) @byosko

90. The value of data in building better products.

91. Data is a key input and filter in building better products.

92. COMPETITION, OTHER PRODUCTS, BEST PRACTICES BUILDLEARN IDEAS CORPORATE GOALS (SOME GOOD, SOME BAD) GUTS & INSTINCTS PARTNERS OTHER DEPARTMENTS INDUSTRY TRENDS, ETC. DATA CUSTOMER INPUT DATA

93. COMPETITION, OTHER PRODUCTS, BEST PRACTICES PARTNERS INDUSTRY TRENDS, ETC. GUTS & INSTINCTS OTHER DEPARTMENTS CORPORATE GOALS DATA AS A FILTER BETTER DECISIONS CUSTOMER INPUT

94. Product & Design (defining goals / objectives) User & customer feedback Sales Marketing Customer Support Etc. ! In-person interviews ! Surveys ! Customer support inquiries ! Real-time online Supported by data Your gut Company vision Collecting Input & Customer Discovery Your own ideas

95. Data is also a communication tool. http://www.instigatorblog.com/data-common-language/2016/09/22/

96. @byosko @byosko

97. Data is complex. How we communicate it doesn’t have to be.

98. Alistair Croll acroll@gmail.com @acroll Ben Yoskovitz byosko@gmail.com @byosko