Detecting and Building Habits: A Data-Driven Approach - Robert Moore, CEO, RJ Metrics - 2016 Habit Summit
•
29 likes•1,510 views
Bob Moore is the co-founder and CEO of RJMetrics, a SaaS business intelligence company with the mission of inspiring and empowering data-driven people. Moore is nationally known for his startup advice and data-driven research, which has been featured in The New York Times, The Wall Street Journal, Inc Magazine, and other publications. In this presentation, Bob helps us "Detect and Build User Habits"
Recommended
Recommended
More Related Content
Similar to Detecting and Building Habits: A Data-Driven Approach - Robert Moore, CEO, RJ Metrics - 2016 Habit Summit
Similar to Detecting and Building Habits: A Data-Driven Approach - Robert Moore, CEO, RJ Metrics - 2016 Habit Summit (20)
Recently uploaded
Recently uploaded (20)
Detecting and Building Habits: A Data-Driven Approach - Robert Moore, CEO, RJ Metrics - 2016 Habit Summit
- 1. Detecting and Building Habits: A Data-Driven Approach Habit Summit 2016 | Stanford University | March 22nd, 2016 Robert Moore CEO, RJMetrics
- 3. rjmetrics.com “During digestion, your body breaks down the nutrients in food. This causes a rise in blood sugar that triggers your pancreas to release insulin into the bloodstream. Insulin increases the amount of serotonin and melatonin that flood the brain, two chemicals associated with drowsiness.” -Science Hope You Enjoyed Lunch
- 4. rjmetrics.com Willpower Depletion “The longer we work on mentally strenuous tasks, the more mental energy we expend, and eventually we’ll run out and start falling back to these easy — and often wrong — default decisions.” -Also Science
- 5. rjmetrics.com Willpower Depletion “The longer we work on mentally strenuous tasks, the more mental energy we expend, and eventually we’ll run out and start falling back to these easy — and often wrong — default decisions.” -Also Science
- 6. rjmetrics.com If you feel yourself getting sleepy during this talk… Disclaimer
- 7. rjmetrics.com If you feel yourself getting sleepy during this talk… If you feel, with each minute that goes by, you are somehow dumber than before I started talking… Disclaimer
- 8. rjmetrics.com If you feel yourself getting sleepy during this talk… If you feel, with each minute that goes by, you are somehow dumber than before I started talking… blame science! Disclaimer
- 9. Detecting and Building Habits: A Data-Driven Approach Habit Summit 2016 | Stanford University | March 22nd, 2016 Robert Moore CEO, RJMetrics
- 10. Why We’re Here
- 11. rjmetrics.com Storage in the Early 1980s
- 12. rjmetrics.com Practical Implications Data Storage Type Space Required Date and Time TIMESTAMP 4 bytes Transac7on Amount DECIMAL (10,2) 4 bytes Item ID SMALLINT 2 bytes Customer ID INTEGER 4 bytes Loca7on SMALLINT 2 bytes Total 16 bytes = 32¢
- 13. rjmetrics.com Practical Implications Data Storage Type Space Required Date and Time TIMESTAMP 4 bytes Transac7on Amount DECIMAL (10,2) 4 bytes Item ID SMALLINT 2 bytes Customer ID INTEGER 4 bytes Loca7on SMALLINT 2 bytes Total 16 bytes = 32¢ 50¢
- 14. rjmetrics.com Data Analysis Was Irresponsible For Most Companies Not long ago: • 1 byte: $0.02 • 1 kb: $20 • 1 mb: $20,000 • 1 gb: $20,000,000
- 16. rjmetrics.com What A Time To Be Alive! That’s 62.5 million cups of coffee! Not to mention inflation (avg 2016 price)
- 17. rjmetrics.com Changing Times: It’s Not Just Storage
- 18. rjmetrics.com
- 19. rjmetrics.com
- 20. rjmetrics.com
- 21. rjmetrics.com
- 22. rjmetrics.com
- 23. rjmetrics.com What This Means for Business Behavioral Data Attribution Data Transaction Data Customer Data Why? Where? What/When? Who? Time
- 24. rjmetrics.com What This Means for You It is easier than ever to use data to understand the behavior and habits of your customers. Here’s how.
- 25. Statistical Groundwork Part I: Correlation and Causation
- 26. rjmetrics.com Correlation and Causation Just because two things consistently move in the same direction, that doesn’t mean that one causes the other.
- 27. rjmetrics.com Correlation and Causation A B Nick Cage Films Drowning Deaths
- 28. rjmetrics.com Explanation 1: Direct Causation A B Nick Cage Films Drowning Deaths
- 29. rjmetrics.com Explanation 2: Indirect Causation A B Nick Cage Films Drowning Deaths
- 30. rjmetrics.com Explanation 3: Lurking Variable A B Nick Cage Films Drowning Deaths C
- 31. rjmetrics.com Explanation 4: Cyclical A B Nick Cage Films Drowning Deaths
- 32. rjmetrics.com Explanation 5: Unrelated A B Nick Cage Films Drowning Deaths X
- 33. rjmetrics.com Focus on Real Improvements, Not Correlations An iterative process based on data allows you to find real results, even in the face of false positives
- 34. Statistical Groundwork Part II: Statistical Significance
- 35. rjmetrics.com Statistical Significance: Go Big Experiment A Old Conversion Rate: 1.0% New Conversion Rate: 1.1% Trials Required: ?
- 36. rjmetrics.com Statistical Significance: Go Big Experiment A Old Conversion Rate: 1.0% New Conversion Rate: 1.1% Trials Required: 159,650
- 37. rjmetrics.com Statistical Significance: Go Big Experiment A Old Conversion Rate: 1.0% New Conversion Rate: 1.1% Trials Required: 159,650 Experiment B Old Conversion Rate: 1.0% New Conversion Rate: 2.0% Trials Required: ?
- 38. rjmetrics.com Statistical Significance: Go Big Experiment A Old Conversion Rate: 1.0% New Conversion Rate: 1.1% Trials Required: 159,650 Experiment B Old Conversion Rate: 1.0% New Conversion Rate: 2.0% Trials Required: 2,226 Lesson: If you’re small, go big or die waiting Get the numbers at www.testsignificance.com
- 39. rjmetrics.com Side Benefit of Going Big
- 40. Techniques Part I: Golden Motion Analysis
- 41. rjmetrics.com Our Golden Motion Experiment What am I trying to learn? RJMetrics has a free trial. What behaviors (“golden motions”) increase the likelihood of user conversion? How will this change my behavior? If we identify actions that are linked to strong increases in conversion, we can change our onboarding process to that more users will take those actions. How will we measure success? Increase in free trial conversion rate.
- 42. rjmetrics.com Step One: Instrumentation Just a few years ago, this was extremely hard. Today, it is trivially easy! What might we want to track: • Logins • Chart Creation • Chart Editing • Adding Users • Sharing Dashboards • Using APIs • Connecting Multiple Data Sources • Time Spent In Tool
- 43. rjmetrics.com Step Two: Consolidation Get that event data right alongside everything else you know about your customer or prospect. Again, third party tools have made this very easy! (my only plug, I swear!)
- 45. rjmetrics.com Step Two: Consolidation User ID Did they convert? Charts Created Users Created Data Connected … 1 No 0 1 1 … 2 Yes 10 3 1 … 3 No 4 5 1 … 4 No 2 4 2 … 5 No 0 3 3 … 6 No 0 4 2 … 7 Yes 14 7 1 … 8 No 0 5 1 … … … … … … …
- 46. rjmetrics.com Step Three: Run Tests Example: Does connecting more data sources increase conversion rate? Data: Result: No statistically significant difference. (two-tailed p = 0.71) Data Sources Conversion Rate Sample Size 1 30% 90 More than 1 34% 110
- 47. rjmetrics.com Step Three: Run Tests Example: Does creating more charts increase conversion rate? (this was WAY down on my list of guesses)
- 48. rjmetrics.com Step Three: Run Tests Example: Does creating more charts increase conversion rate? Data: Result: YES! Statistically significant difference. (two-tailed p < 0.01) Charts Conversion Rate Sample Size 0 30% 70 1 or More 62% 130
- 49. rjmetrics.com Step Three: Run Tests Example: Does creating more charts increase conversion rate? Data: Charts Conversion Rate Sample Size Significant? 0 30% 70 Yes 1 48% 55 Yes 2 66% 35 Yes 3 70% 20 No 4 or more 71% 20 No
- 50. rjmetrics.com Step Four: Change Behavior Objective: Make sure every client creates at least two charts on their own. Changes: • Introduce self-guided tour of report builder • Give new clients “homework” after launch calls • Beef up documentation on report builder • Give our Account Management team tools to track this metric
- 51. rjmetrics.com Step Four: Measure Again Nothing is promised! Your work isn’t done. Don’t get Nicholas Cage’d.
- 52. rjmetrics.com Step Four: Measure Again 0% 10% 20% 30% 40% 50% 60% 70% 1/1/13 2/1/13 3/1/13 4/1/13 5/1/13 6/1/13 7/1/13 8/1/13 9/1/13 10/1/13 11/1/13 12/1/13 Conversion Rate Change implemented here BOOM!
- 53. Techniques Part II: Cluster Analysis
- 54. rjmetrics.com Cluster Analyis We can start with that same event data from the last example
- 55. rjmetrics.com Cluster Analysis Decide on attributes to cluster on: Maybe user profiling is more complex than just “viewers” and “creators.” And where do you draw the line?
- 56. rjmetrics.com Cluster Analysis Algorithms (in this case the k-means algorithm) group our users into groups that are most statistically similar to one another based on the behaviors we’ve observed
- 60. What’s Left? Opportunity! We know where to focus.
- 61. rjmetrics.com The Opportunities Are Enormous Give data a seat at the table! It won’t answer everything, but it’s an important voice when you’re studying user habits.