Stop refreshing vanity metrics & start focusing on the metrics that inform decisions
•Download as PPTX, PDF•
3 likes•821 views
Stop Refreshing Vanity Metrics & Start Focusing on the Metrics that Inform Decisions There is a propensity to focus on vanity metrics; metrics that show you the score: How many new views, new daily active users, how much revenue last week. You may slice these by different attributes - geography, platform, user demographics. While this can help you understand the high level trends in your business, it does little to tell you how to get better. This slide deck looks at how vanity metrics can distract you from focusing on the analysis that matters, which is identifying and measuring the metrics that drive decisions. There are several real examples of how companies (Venmo, Simply Business, and Looker) have used event data in highly customized ways to make better decisions about their products.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Stop refreshing vanity metrics & start focusing on the metrics that inform decisions
Similar to Stop refreshing vanity metrics & start focusing on the metrics that inform decisions (20)
More from Looker
More from Looker (15)
Recently uploaded
Recently uploaded (20)
Stop refreshing vanity metrics & start focusing on the metrics that inform decisions
- 1. Colin Zima Chief Analytics Officer, Looker Beyond Vanity Metrics
- 4. The Case for Dashboards Increased Engagement Using visual to draws in the less data inclined Clarity and Priority Teams driving towards a singular goal set “Data Culture” Data as the source of truth
- 5. Too Broad Too Rigid Too Static The Not-So-Good Individuals don’t feel empowered to drive top-level metrics The difficult-to-quantify is often eschewed for the simplest metrics Can’t creating alignment around metrics that are changing
- 6. Define Behavior Quantify Act and Refine “Custom” Analytics • How can we understand this behavior with data? • Find correlated action if necessary. • Set up tracking as needed • Define ongoing metrics • What are we actually trying to measure? • Create more targeted, individualized dashboards and views • Track over time and monitor performance • Build better metrics
- 7. Four Examples 1. Quantifying bad user experience at Venmo 2. Prioritizing new market launches at a delivery start-up 3. Better understanding of the best performing content at Upworthy 4. Driving deeper retention in mobile gaming
- 8. Venmo’s Design Challenge Before: After:
- 9. The Result
- 10. Custom Analytics To The Rescue
- 13. Now Quantify It
- 15. Attention Minutes • What action are we really trying to drive – Engagement • Then measure engagement • Upworthy’s attention minutes aim to do just that • Total Attention on Site (per hour, day, week, month, whatever) — that tells us (like total uniques or total pageviews) how good of a job Upworthy is doing overall at drawing attention to important topics. • And Total Attention per Piece, which is a combination of how many people watch something on Upworthy and how much of it they actually watch. Pieces with higher Total Attention should be promoted more. Source: http://blog.upworthy.com/post/89621755036/the-code-literally-to-what-lies-between-
- 16. The Results
- 17. Gaming Analytics • DAU and MAU are de facto success in gaming • Getting underneath the surface is where operational success comes from • Cohort analysis is a crucial tool
- 18. Level Balancing
- 19. Level Balancing
- 20. Level Balancing
- 21. TwoDots TwoDots: 15 app releases, 40+ level rebalances Before: After:
- 22. Takeaways Define the behaviors you want to quantify Measure them and create tangible progress And keep the TV Dashboards, at least you’ll get people excited about data
Editor's Notes
- At HotelTonight we transitioned from a focus on bookings to a focus on room nights – everyone is driving at the same team goals
- Who created the metrics – developers weren’t the ones doing this, analysts and product were doing this on their own. Observing a problem and then quantifying.
- For services growing the business locally, finding the right markets to serve can be a challenge. Any scaled back service in a given market leads to impatient customers that may expect better coverage. This is acutely true for applications in the app store, where ratings are a crucial growth mechanism. When folks download the app and can’t get service, poor ratings follow.
- For services growing the business locally, finding the right markets to serve can be a challenge. Any scaled back service in a given market leads to impatient customers that may expect better coverage. This is acutely true for applications in the app store, where ratings are a crucial growth mechanism. When folks download the app and can’t get service, poor ratings follow.
- Knowing we now have a proxy for bad user experience, we can set goals of driving these down to zero. In this example, we look at % of users that open the app more than 5 miles from a service provider, on the upper left on different continents, ranging from the purple (not launched in South America), to the Green – coverage in the United States. Obviously all of this can be sliced up and down, even to sub-regions. On the lower right we have the launch in Puerto Rico, where coverage went from 0% quickly to near 100%. This can be replicated everywhere, and even moving the quality bar from 5 miles down to 1 mile, etc. This could also be done for things like response time by location on web, or search page load time. We can simply look at each users response time rather than something less granular than average conversion rate.
- How many of you have run in to a online slideshow recently? How many of you thought it was a good user experience? Why do they exist? Pageviews Unfortunately advertisers are actually paying for views sometimes This is changing and ad performance is improving both lots of businesses are still using these blunt metrics
- Chartbeat, Medium have also talked about this at length – engagement over vanity metrics
- Level success rates actively rebalancing the game to drive continued usage
- Level success rates actively rebalancing the game to drive continued usage
- Level success rates actively rebalancing the game to drive continued usage