The document discusses data-driven product management and provides practical ideas and tools for PMs to use to make decisions. It emphasizes that PMs should optimize decisions for enterprise value and focus on gathering knowable facts from customer data and usage metrics, rather than opinions. PMs are encouraged to measure key metrics from the start, investigate problems, create dashboards, delay decisions until necessary while learning from experiments, and use tools like SQL, text editors, Excel, and Tableau to integrate different data sources and identify patterns to inform prioritization. The overall message is that PMs should make decisions based on analyzing multiple quantitative and qualitative data sources, rather than relying solely on opinions or anecdotes.
2. About Me
Who here lives in Arlington? (Vote Dunn!)
MIT mechanical engineer (but I never used it)
7 startups in 15 years
Career path from support to implementation to QA to
PM
<date> – Confidential
2
4. Most PMs Aren’t Visionaries
Ideas come from customers, colleagues, and prospects
Steve Jobs isn’t walking into this product meeting
PMs probe, interpret, and synthesize
4
5. Ideas Are Not the Scarce Resource
Ideas come in sizes: markets, features, bug fixes, and
optimizations
They have different motivations
Increased sales
Higher retention
Lower cost of goods
Unlimited resources, you could do it all – but we don’t
have that
Someone has to decide what is next
This is why PMs get paid the big bucks
5
6. Optimize for Enterprise Value
The PM’s job is to prioritize
What’s the North Star for your company?
Stars are directional – you can’t make a map to get to star
How do you know if you are pointed in the right direct?
How do you know if you are making progress?
How do you compare apples to oranges?
And compare that to bacon?
6
7. “That is a knowable fact.”
What the advocate says
“No one uses that feature”
“Everyone wants this!”
“That breaks all the time”
“You're not fixing enough bugs”
“This problem happens to everyone!”
“I’ve heard this request a million times”
What the data says
15% of users click that every week
We’ve had 3 customers ask for this feature
5% of support calls are associated with a bug
7
8. Know Which Facts Are Knowable
Carefully separate opinion from fact, known from
unknown
Huge, immediate reduction in complexity of the decision
Develop a third and fourth category
1.
2.
3.
4.
We really don’t know
Knowable fact
We can know if we do . . .
Before we decide, we really should know
A good PM uses all 4 categories to make a decision
This talk is more about 3 and 4
8
10. Your Application Database Knows
Your customers using your app are telling you how they
use it.
You need to get the data reproducibly
You need data, not reports
Know what you need to change
Know if your changes actually worked or not
10
11. Measure It From The Start
Your application database can’t tell you everything
Make an early change that adds data and measurement
Pipeline speed
Funnel shape
Daily activity
12. Measure the Good and the Bad
You have to know what the problems are
You have to know when they get worse
14. When Do You Have to Decide?
Most of the time, the answer is “later”
Don’t decide until you have to
This is where the art meets the science
Know your downsides and worst-case scenarios, and mitigate them
Watch, and monitor
Agile (“agile”) really shines here
You will have the development bandwidth when you need it
Unfortunately frustrating for many customers and colleagues
14
17. How Do You Decide?
Most decisions aren’t reduced to a time series
Comparing apples, oranges, and bacon
Your company needs all three
Collect all the data you can
Read what the customer said (or potential customer). Talk to them directly.
Talk to the people who interacted with them (support, consultant, sales rep,
account manager)
Look at the usage
Look at the market and the competition
17
18. Find a way to order the data
Whiteboards and stickies
What themes can you find
What time ordering can you find
What pre-requisites can you find
Which ideas are both cheap and enable discovery
18
19. Build a framework
Whiteboards and stickies – and Excel
Just make one up
10 points for data loss
1 point for annoying
1 point per customer affected
3 points per big customer
You are the most qualified person to do it
See what maps to your intuition, what doesn’t
Know the limitations of what you built
Iterate
19
21. My Tools
SQL
You need access to the data, not reports
NoSQL has query tools, too
Text editor
UltraEdit. Python, Perl work too
Turn dross into data
Excel
You do know how to make a pivot table, right? Find the lumpy parts.
Can you do vlookups in your sleep? Integrate your data sources
Tableau
Whiteboards and stickies
TheBrain mind-mapping software
21