Drew Dillon gave a presentation on running a data-driven product organization. He discussed collecting data on key metrics like user growth and retention. Insights from data can show what areas need improvement and whether experiments worked. Dashboards and reports help visualize trends and funnels. Multivariate testing allows testing different features to see what improves metrics. Machine learning can power recommendations and natural language processing. While data provides insights, intuition is also important, and not everything can be measured quantitatively.

1. www.productschool.com
How to Run a Data Driven Product Dev
Organization by Skedulo CPM

2. FREE INVITE
Join 30,000+ Product Managers on

3. COURSES
Product Management
Learn the skills you need to land a product manager job

4. COURSES
Coding for Managers
Build a website and gain the technical knowledge to lead software engineers

5. COURSES
Data Analytics for Managers
Learn the skills to understand web analytics, SQL and machine learning concepts

6. COURSES
Digital Marketing for Managers
Learn how to acquire more users and convert them into clients

7. COURSES
Blockchain for Managers
Learn how to trade cryptocurrencies and build products using the blockchain

8. Drew Dillon
TONIGHT’S SPEAKERS

9. Running a Data Driven
Product Organization

10. Hi, I’m Drew
@drewdil
CPO at Skedulo
VP of Product at AnyPerk
Director of Product at Yammer

11. I’m Not a Statistician

12. Overview
● The Character of Data
● Collecting Data
● What to Measure?
● Insights
● Dashboards and Reporting
● Multivariate Testing
● Machine Learning

14. The Character of Data

21. What Can Data Tell Us?
● What areas of the product are getting the most use and how
● How we're doing (key performance indicators, "KPI")
● The outcome of an experiment
● Whether a feature "sucks"
● And… what it can’t
○ What to do next
○ Causal relationships between past events
○ When to make larger bets
○ Whether you’ve hit relative maxima

22. Collecting Data

24. Segment

26. What to Measure?

27. Growth
● Users/customers - how many are coming in? Where do they come
from?
● Retention - are people coming back 48 hours, one week, one month
after joining? Different apps may care about different retention
intervals.
● Invites (k-factor) - how many invites are you getting out of each user? If
you have multiple viral elements, which is most effective?

28. App Categories
Transactional Engaging

29. Transactional Apps
• Conversion
• Funnel
2 M
250 K
hit the site purchase
conversion rate:
12.5%
1 M user opportunity

30. Engaging Apps
• Engagement
– Sign-in, page views, time on site
• Interactions
– App-specific
– Prioritize driving further engagement / growth
What are the two most important interactions on Instagram?

31. Successful apps
prioritize one

32. Transactional or Engaging?
Amazon
Netflix
Sports Betting

33. Core Metrics
activities believed to lead to long
term success
Feature Metrics
usage of an individual feature

34. What Not to Measure
● Clicks—clicks aren’t always positive, clicking around may signal user
distress or confusion
● Page views—a painfully obvious perverse incentive.
● Time on site—not terrible data, but over-emphasized and not
objectively positive

35. Precision vs. Trends

36. Insights

38. Ad Hoc Questions
Can empower other teams:
● What are behaviors that correlated to paid upgrades?
● Which customers haven't logged in in a while?
And tell you what's worth working on:
● How many people view this page?
● Should we build an Android app? (How might you determine this?)

39. Cohort Analysis
AHA!

40. Dashboards & Reporting

42. Vanity Metrics

43. There are only two honest
visualizations, a line graph and a
bar chart. Anything else is a sales
pitch.

44. Funnel Visualization

45. Multivariate Testing

47. It’s not Stats, it’s
Econ

48. What to Test
in a perfect world,
everything

50. Testing Has Costs
1. Time to delivery
2. Code complexity
3. QA has to test multiple versions
4. Other PMs and Designers have to know about tests
5. Users don’t love the idea
Early heuristic: only test features that impact 30% of users.

51. A/B Testing Tools

52. Test Flows
still need to roll-your-own

53. Overtesting and Intuition

54. Million Dollar Bet
Here are the ground rules:
1. The competition will last approximately two years
2. Once a flag is on the object, the other competitor can no longer put their flag
there
3. We both start in San Francisco with $10,000. We can borrow more, but then
we don’t win as much from the bet.
What would be your strategy?

60. This Journey is Your Product

61. Frequentist Probability

62. Flavors of Test Data
Aggregates
Number of times a certain action was taken
● Number of invites sent
● Total replies
● Days Engaged
Binaries
Whether a certain action was taken
● Invite sent
● Replied
● 24 hour retention

63. P-values

64. Sample Size, Again

66. Bayesian Probability

67. Credible Intervals
incorporate problem-specific
contextual information from the prior
distribution

68. Multivariate testing
doesn’t replace
intuition, it just makes
outcomes clearer.

69. Experimentation Culture
start by understanding bias

70. Experiment Outcomes at Google
Failure SuccessFlat
33.333333% 33.333333% 33.333333%
The very best product managers at Google are “wrong” 68%
of the time.

71. Redefining Engineering Success

72. Machine Learning

74. Recommender Systems (Collaborative Filtering)

75. Natural Language Processing

76. Dimensionality Reduction

77. Quick Note on “Overfitting”

78. Review
● The Character of Data
● Collecting Data
● What to Measure?
● Insights
● Dashboards and Reporting
● Multivariate Testing
● Machine Learning

79. Data “Driven?”

80. "The most important things
cannot be measured."
"The most important things are
unknown or unknowable."
- Edward Deming

82. Further Reading
bit.ly/DDProduct
Stay in Touch!
linkedin.com/in/drewdil
twitter.com/drewdil
medium.com/product-man
quora.com/profile/Drew-Dillon

83. www.productschool.com
Part-time Product Management, Coding, Data, Digital
Marketing and Blockchain courses in San Francisco, Silicon
Valley, New York, Santa Monica, Los Angeles, Austin, Boston,
Boulder, Chicago, Denver, Orange County, Seattle, Bellevue,
Toronto, London and Online