Eli Holder at Product School, New York

1. How to build amazing Products through
Customer Feedback
/Productschool @ProdSchool /ProductmanagementNY
Venue Sponsored by

2. Eli Holder
- Product Leader, Designer and
Entrepreneur
- Founder of Notch.me and Unblab
www.productschool.com
How to build amazing Products
through Customer Feedback

3. Good Qs,
Good
P’s.31 Good Questions Product Managers
can ask to build Good Products.
by Eli Holder
eli.holder@nodd.co

4. “That’s a great
question.”

5. Work Work Work.Eli’s Background

6. unblab. “Spent about 10 minutes training it to
recognize my important emails and it's already
saved me hours and hours of sorting throughacquired by

7. “The SMS like model of having in and
outbound messages keeping a conversation
timely is the killer feature.”
“I found it very refreshing and energizing to
see creative products such as this one at

8. “Until I saw your graphics and your ‘result
stories’, I really wasn’t that dissatisfied.
Now, I want to view all of my data this

9. “Eli has been instrumental in bringing an
organized process to studying users"

11. Nodd helps team
leaders get honest,
constructive feedback
from their teams at
work.
"I loved the emojis and
the survey format was
awesome. I was
engaged the whole time.
Excellent platform.”
Nodd.co

13. Dumb.
2009

14. Dumb. Slightly less dumb.
2009 2016
The power of questions.

15. Goal of this talk:
Hooray
Questions!

16. Why Questions?
“The
Unknown
Unknowns”
i.e. the epistemology
of Donald Rumsfeld

17. Unknown
Unknowns
Known
Unknowns
When you learn a new question:

18. Goals of the talk:
1. The power of
2. Asking the right
3. Product Questions

19. Audience Q:
Warmup: “How are you?”

20. 1. Who are we? Why are we here?
(strategy & existential crises)
2. Who’s our audience? What do
they want? (user research)
3. What problem should we solve?
(opportunity assessment)
4. How should we solve the
problem? (design sprinting)
5. Are we approaching Product /
Market Fit? (build, measure, learn loop)
Big Product
Q’s
(for each Product Lifecycle
Stage)

21. Related Product Q’s.
Who are we? What’s our mission? What’s our aligning metric? What are our organizational objectives for the quarter/year (OKRs)? Who are
the stakeholders? What are their goals? Why invest more in the product (e.g. instead of sales?) Who’s going to be building it? What are their
personal goals and interests? What are their strengths and weaknesses? How much of their time do we have (e.g. are they cross functional? do
they have other projects? are there churn risks?) What are the team’s strengths and weaknesses? What are people’s expected communication
styles? What’s the organization’s culture (e.g. Risk tolerance)? What’s the organization’s perceived brand? What are the expectations for a
product manager? What’s the (target) business model? What stage is the company at? What’s the prevailing role within the company? Who are
the other companies in the space? How is your company different? Tech advantages / debt? Design advantages / debt? Data advantages? What
are the religious issues that everyone’s tired of discussing? External requirements? Whats your support team like? How do you (think) you make
money? Who’s our audience? What do our users think about on the subway? Who are our primary (and secondary) users? What does a
typical day look like for them? What problems are they currently experiencing? (What are the 5 whys for the problems? What are they doing now
to solve them? What do they like about the current solution? What do they dislike about the current solution? What are they feeling?) What are
the most important dimensions to describe them? (e.g. demographics? behavior? company role? values? physical attributes? ) Who are the
extreme users? What do our users value? What are their goals? What are their obligations? What are their frustrations? What are their
stresses? What gets them excited? Who are the people in their lives? How do they typically interact with them? What’s the users value to us?
What language do they use to describe their experiences? What other products / tools are they using? What are they reading? Who are they
following? How do they find / buy related products? How do they fit into purchasing decisions? What can they spend? When do they spend?
What’s their organization’s culture like? How do they fit in? What’s their organization’s business model? How do competitors perceive them?
What does academia think about them? What problem are we solving? (value prop) What problems are incidental (non-goals)? For
whom do we solve this problem? (target market) How big is the opportunity? (market size) How will we measure success? (metrics)
What alternatives are out there now? (competitors) Why us? What’s our differentiator? Why now? (market window) Why might this a
bad idea? How will we market this? Other considerations for success? How should we solve this? What are all the different ways we
could possibly solve this? How are our competitors solving this? How have academics solved this? How have people solved similar problems in
different industries? Which solution is the best for us to pursue? Which solution is most interesting to the team? Which solution has the most
learning potential? Is it technically feasible? (What’s the relative complexity?) Is it consistent with the behavior / data we’ve collected so far?

22. Who are we?
Why are we
here?Product Lifecycle Stage #1:
Q:
Strategy & Existential Crises

23. What’s the mission?Q:
Who are the
stakeholders? What do
they want?
Q:
What are our strengths &
weaknesses relative to
competitors?
Q:
Strategy & Existential Crises

24. Why it’s importantFollow your dreams, idiot.

25. Team Plan:
Follow my dreams.

26. “What’s our one metric that matters”?
“What are our near-term OKRs”?
“What are our product principles”?
Ask:
What’s the mission?Q:

27. Who are the
stakeholders?
What do they want?
Q:

28. This is not you.

29. This is you.

30. 1. Conformity bias
2. Positivity bias
3. They don’t know
4. Don’t want to look dumb
challenges:
Who are the
stakeholders?
What do they want?
Q:

31. The Hopes / Fears
Exercise:
Who are the
stakeholders?
What do they want?
Q:
“What are your 3 biggest hopes for this
project? (e.g. what do you hope this project
accomplishes for [company]?)
“What are your 3 biggest fears about this
project? (e.g. what do you think will go wrong?)
“What are your other considerations for this
project? (anything else on your mind?)
Ask:

32. Hopes:
• That project X opens up a new market for us
• I’m optimistic about the chance at a new market
• That project X opens up a substantial new
revenue stream for us
• That project X has interesting technical
challenges
Fears:
• That project X is much more technically difficult
than we expected
• That we get distracted and don’t give project X a
fair shot
• That we’re more excited about the revenue
opportunity than solving a real customer problem.
Hopes / Fears Exercise Results:
Who are the
stakeholders?
What do they want?
Q:
Answers:

33. Hopes, Fears and
Psychological Safety.

34. What’s our mission? What’s our aligning metric? What are our
organizational objectives for the quarter/year (OKRs)? Who are the
stakeholders? What are their goals? Why invest more in the product
(e.g. instead of sales?)
Who’s going to be building it? What are their personal goals and
interests? What are their strengths and weaknesses? How much of their
time do we have (e.g. are they cross functional? do they have other
projects? are there churn risks?) What are the team’s strengths and
weaknesses? What are people’s expected communication styles?
What’s the organization’s culture (e.g. Risk tolerance)? What’s the
organization’s perceived brand? What are the expectations for a product
manager? What’s the (target) business model? What stage is the company
at? What’s the prevailing role within the company?
Who are the other companies in the space? How is your company
different? Tech advantages / debt? Design advantages / debt? Data
advantages?
Strategy: Relevant Q’s.

35. Audience Q:
Any questions so far?

36. Who’s our
audience?
What do they want?Product Lifecycle Stage #2:
Q:
User Research

37. “Set up at least 30 meetings,
or you won’t have a good
Michael Sippey
Former VP Product
Twitter
User Research: Why?

38. Allison Smith*
27, Cleveland
New Mom
Yahoo Mail User
*Not really Allison But
thank you to this person
for CC Flickr photo.
How does Allison
use email?

40. What does a typical day look
like?
What’s important to them?
What’s holding them back?
What do they think about on the
Qs:
Who’s our Audience?Q:

41. What do users want?Q:

42. Audience Q:
Why shouldn’t you directly ask
users “What do you want?”

43. “If I had asked people what they
wanted, they would have said:
Faster Horses.”
Henry Ford*
Model-T Inventor
*Not really

44. What do users want?Q:
1. They can’t say what they want
2. They won’t know what’s
possible
3. It’s not their job to know
challenges:

45. What do users want?Q:
“How do you currently do [area you
want to explore]?”
“What are the top 3 things you love
about how you’re doing it now?”
“What are the 3 biggest frustrations
with how you’re doing it now?”
Ask:

46. Happy Event
Frustrating Event
Emotion Timeline
Exercise.
What do users want?Q:
"If you were to draw a chart of your
emotions throughout your experience
[doing X], with peaks when you're most
happy and valleys when you're most
unhappy, what would that look like?"
Ask:

47. A designer describes the 360
performance review experience.
Choose my
reviewers.
Find out it’s perf
review time.
Find out I have to
write 4 reviews
I write the reviews,
feel relieved
A long time goes
by and I forget
about it.
Boss emails to
say it’s time to
have the talk.
Talk day.
Promotion!
Emotion Timeline Exercise
Results:
What do users want?Q:
Answer:

48. Who are your competitors
customers? What do they
value?
Q:

49. “[your industry]” (testimonials|our
clients|our customers)
site:[competitor_site] filetype:pdf
site:twitter.com "apple maps" ("is
a”|"sucks"|"can't"|"fuck"|"shit")
https://twitter.com/search?q=to
%3A[COMPETITORS_TWITTER]%20%3A)
:
Who are your competitors
customers? What do they
value?
Q:

50. User Research: Relevant Q’s.
What do our users think about on the subway?
Who are our primary (and secondary) users? What does a typical day look like
for them? What problems are they currently experiencing? (What are the 5
whys for the problems? What are they doing now to solve them? What do they
like about the current solution? What do they dislike about the current
solution? What are they feeling?)
What are the most important dimensions to describe them? (e.g.
demographics? behavior? company role? values? physical attributes? ) Who
are the extreme users? What do our users value? What are their goals? What
are their obligations? What are their frustrations? What are their stresses?
What gets them excited? Who are the people in their lives? How do they
typically interact with them? What’s the users value to us?
What language do they use to describe their experiences? What other products /
tools are they using? What are they reading? Who are they following? How do they
find / buy related products? How do they fit into purchasing decisions? What can
they spend? When do they spend? What’s their organization’s culture like? How do

52. What problem
should we solve?
Product Lifecycle Stage #3:
Q:
Opportunity Assessment

53. Audience Q:
What is the #1 Most Important
Question about a Product?

54. The Most Important Question:
What problem are weQ:
*Independently of how we might solve it.

55. *Independently of how we might solve it.
Advil.
Features are for suckers..
What problem are we
solving?
Q:
310M Americans experience
mild headaches and body
pain due to inflammation at
least once per year. We’ll
make that pain go away.
This:
We’re manufacturing doses of
ibuprofen. They’ll come in
blue or brown pills. The bottle
is child proof. We’re
advertising the hell out of
them.
Not This:

56. “Shiny feature X.”
CEO: It needs to be social. Add share buttons everywhere!
PM: Sir, it’s a porn site. How many people share porn on Facebook?
CEO: The investors say we’re not growing fast enough. And some site
called “Buzzfeed" is really taking off because of the sharing.
PM: It sounds like you’re saying the real problem is growth? Are
there other ways we could solve that?
*Independently of how we might solve it.
What problem are we
solving?
Q:
Get to the root of the problem.
Avoid building things that don’t add value.

57. What problem are we solving?*Q: *Independently of how we might solve it.
80M Americans have have a
hard time finding things on
the internet.
Finding information on the internet.
Problem
Idea #1 Portals and directories (yahoo)
Idea #2 Search by phrase matching
(altavista)
Idea #3 search by phrase matching
and back links signal (google)
Problem framing leaves room for multiple solutions.

58. What problem are we solving?*Q: *Independently of how we might solve it.
PM:
Designer:
*Quits. Goes to Facebook.*
Problem framing helps you avoid irritating
your teams’ designers and engineers.
The signup button must
be big and red so more
people will see it and click
on it.

59. What problem are we solving?Q:
Problem Statements:
1. Focus on value, not features.
2. Avoid building useless shit.
3. Allow for alternate solutions.
4. Aren’t prescriptive.
5. Force clearer thinking.
*Independently of how we might solve it.

60. From The PM Handbook
(mostly).

61. Opportunity Assessment:
The 11 most important questions for any product.
1. What problem are we solving? (value prop)
2. What problems are incidental (non-goals)?
3. For whom do we solve this problem? (target market)
4. How big is the opportunity? (market size)
5. How will we measure success? (metrics)
6. What alternatives are out there now? (competitors)
7. Why us? What’s our differentiator?
8. Why now? (market window)
9. Why might this a bad idea?
10.How will we market this?
11. Other considerations for success?
From The PM Handbook
(mostly).

62. Big Product
Q’s
(for each Product Lifecycle
Stage)
1. Who are we? Why are we here?
(strategy & existential crises)
2. Who’s our audience? What do
they want? (user research)
3. What problem should we solve?
(opportunity assessment)
4. How should we solve the
problem? (design sprinting)
5. Are we approaching Product /
Market Fit? (build, measure, learn loop)

63. “How can I
improve?”
Q:
How to get Honest Team Feedback at work.

64. 1. Fear of Retribution
2. Timing
3. Management Middleman
challenges:
“How can I
improve?”
Q:
How to get Honest Team Feedback at work.

65. Problem:
People at work- especially managers - have
difficulty getting honest, constructive
feedback from their teams at work, causing
missed opportunities for learning and
development. Root causes include:
Reviewers’ fear of retribution, feedback
infrequency, and concerns about how
criticism can negatively impact coworkers’
careers.

66. Nodd helps team leaders ask the right questions to get
honest, constructive feedback from their teams at work.
It solves feedback’s honesty problem with 3 simple
rules. Feedback through Nodd is:
Nodd keeps reviewers’
identities confidential and
responses are only ever shown
in aggregate from 3+
reviewers.
Nodd helps create a feedback
ritual by prompting teams with
simple, reviewer friendly micro-
surveys every two weeks or at
the end of each month.
Feedback is sent directly from
person-to-person and never
through management. So
reviewers can feel safe
knowing it won’t affect the
person’s pay.
Anonymous. Frequent. Private.
Nodd.co

67. https://nodd.co

68. Summary:
1. The power of
2. Asking the right
3. Product Questions
4. Nodd is in beta.
Signup at: https://

69. Thanks!
eli.holder@nodd.co
http://bit.ly/good-product-questions

70. Upcoming Courses
www.productschool.com
www.productschool.com
brad@productschool.com
APPLY AT
Weekdays: June 7th
Weekends: June 4th

71. Three Rules Every Mobile Product Needs
to Be Successful With The Author Of
Mobilized
UPCOMING WORKSHOP
www.productschool.com
RSVP ON EVENTBRITE

72. Appendix

73. “How could we solve this problem AND REALLY DO
AN AWFUL JOB & EMBARRASS OURSELVES?
“How would we solve this problem IN SPACE?!
“How would we solve this problem IF OUR USERS
WERE EXCLUSIVELY 19th CENTURY PIRATES?
“How would we solve this problem WITH
BENEVOLENT, AUTONOMOUS NANO-BOTS?
“How would you solve this problem BY DOING THE
OPPOSITE OF YOUR LEFT-NEIGHBOR’S IDEA?
What are all the different ways we could
possibly solve this?
Q:
Ask:

74. To assess team alignment, how many different ways do people answer:
“What problem are we solving for our users over
the next 2 months?”
When someone shows you work:
“What were your goals for this?”
To assess team’s understanding of a story:
“On a scale of 1, 2, 3, 5, 8, 13, 21, how complex
does everyone think this story is?” (planning
poker: okay for planning, great for
communication).
Are we on the same page?
Q:
Ask:
(probably not)

75. How can we improve our process?
Q:
Ask: Ask each team member to write:
“What are 3 things that we don’t currently
do, that we should start doing?”
“What are 3 things that we currently do, that
we should stop doing?”
“What are 3 things that we currently do, that
we should continue doing?”