Join Nickey Skarstad, Director of Product at Duolingo, as she discusses why it’s important to actively gather customer feedback, how to build customer feedback loops into your product planning, the best ways to decipher different types of feedback, and different frameworks for how/when to apply feedback processes.

3. Hi!
I’m Nickey 👋

4. WHAT ARE
FEEDBACK
LOOPS?

5. WAYS OF COLLECTING FEEDBACK
(typically automated, or systematized so it
happens frequently)

6. CUSTOMER FEEDBACK
LOOPS
Gather feedback
Share feedback
Analyze feedback
Action feedback

7. WHAT DOES IT
MEAN TO BE
CUSTOMER-
CENTRIC?
1. Building
empathetic teams
2. Putting customers
at center of your
process

9. WHY INVEST IN
FEEDBACK
LOOPS?

10. Two customer
interviews
before launch
Months of
time and
resources
spent
shipping a
product
feature

11. HOW MANY OF YOU HAVE AN OKR
RIGHT NOW ABOUT CHURN?
(literally me )

12. 25X MORE EXPENSIVE TO
ACQUIRE A NEW CUSTOMER
(so why risk losing them?)

13. THE IMPORTANCE OF YOUR 100
TRUE FANS

14. KEEP YOUR WHALES
(1-2% of users in the gaming industry drive
~80% of the revenue)

15. BUILDING
EMPATHY
FOSTERS
INNOVATION
of people with highly empathic senior leaders
report often or always being innovative at
work compared to only 13% of people with less
empathic senior leaders.
61%

16. EMPATHETIC
LEADERSHIP
CREATES MORE
ENGAGEMENT
76%
of people with highly empathic senior leaders
report often or always being engaged,
compared to only 32% of people with less
empathic senior leaders.

17. YOU JUST BUILD BETTER
(Great > Good > Basic)

18. HOW TO BUILD
FEEDBACK
LOOPS?

19. FEEDBACK LOOPS ARE A SYSTEM
OF INPUTS THAT YOU PROCESS &
APPLY AT THE RIGHT MOMENTS

20. CUSTOMER FEEDBACK
LOOPS
Gather feedback
“the what”
Share feedback
“the when”
Analyze feedback
“the why”
Action feedback
“the how”

21. GATHER
FEEDBACK
(the what)

22. EAT THE DOG
FOOD
1. Create programs
that incentivize
using the product
2. Celebrate when
people use the
product

23. EAT THE DOG FOOD
● At Airbnb Experiences, we
reimbursed people when they
used the product
○ Had people share real
stories at all hands and
celebrated people who used
the product often
● Duolingo runs “language
learning challenges” where you
compete/get paid

24. LET
CUSTOMERS
HELP YOU
BUILD
1. Beta programs can
help you co-create
with customers
2. Giving customers
direct influence
over product
decisions can be
powerful

25. “Prototypes are worth
1,000 meetings.
-IDEO adage

26. LET CUSTOMERS HELP
● At Etsy we had 10k sellers in
a prototype before we
shipped
● Got feedback daily on new
changes
● Sellers felt ownership when it
shipped

27. LET CUSTOMERS HELP

28. LET CUSTOMERS HELP

29. LET CUSTOMERS HELP

30. LET CUSTOMERS HELP

31. LET CUSTOMERS HELP

32. LET CUSTOMERS HELP

33. LET CUSTOMERS HELP

34. LET CUSTOMERS HELP

35. LET CUSTOMERS HELP

36. BUILD LOOPS
INTO THE USER
JOURNEY 🔁
1. Find ways to ask
customers for
feedback while
they are using your
product
2. Catch people at
moments while
feedback is fresh
and relevant

37. INTO THE USER JOURNEY
● At Etsy, asked sellers
why they went back to
an old version
● Got much better
feedback because we
caught them right as
they were making a
decision

38. INTO THE USER JOURNEY

39. INTO THE USER JOURNEY

40. MIX THE RIGHT
QUAL / QUAN
COCKTAIL 🍸
1. Come up with the
right feedback mix
for your
product/team
2. Both/and will help
get better holistic
data, which you can
then process

41. QUAL / QUAN COCKTAIL

42. ANALYZE
FEEDBACK
(the why)

43. DEVELOP A
SYSTEM FOR
INCOMING
FEEDBACK
1. Build process and
ritual around
reviewing incoming
feedback
2. Do this frequently,
more than once a
quarter

44. DEVELOP A FEEDBACK SYSTEM
● At Duolingo, we collect feedback
through in-product reviews, UXR
studies, A/B testing and more
● It’s collected at almost all steps
in product development process
● We review in a dedicated meeting
weekly, and build into sprint
planning to action the most
timely and relevant info

45. INVEST IN YOUR
TOOLS
1. Good tools help you
organize high
impact feedback
2. Without shared
places to store
info, it gets lots in
the Google Doc
space and time
continuum

46. INVEST IN YOUR TOOLS
● Without good tools to keep track
of incoming feedback, it’s
forgotten quickly
● Good tools are:
○ Shared by the team
○ Understood by the team
○ Used by the team
● Can be a spreadsheet, Google
Doc, JIRA ticket, or housed in a
dedicated tool like Pendo.io
(Screenshot of Pendo.io 👆 )

47. SIZE
OPPORTUNITIES
CONSISTENTLY
1. Use the same
rubric for sizing
new opportunities
2. Use this to review
upcoming work

48. SIZE OPPORTUNITIES
● Review potential work against a
consistent set of criteria that
matters to your team, your
product, your customers
● Impact and effort are commonly
used, but take it one level deeper
like evaluating against your goals
and long-term strategy

49. SIZE OPPORTUNITIES

50. SIZE OPPORTUNITIES

51. SIZE OPPORTUNITIES

52. SIZE OPPORTUNITIES
ICE Score = (Impact +Confidence +Ease) / 3

53. ACTION
FEEDBACK
(the how)

54. REVIEW &
ACTION
FEEDBACK
1. Once you figure out
what to tackle, you need
to figure out how to
tackle it
2. Use a customer-focused
framework to better
contextualize feedback
into actionable user
stories

55. REVIEW & ACTION FEEDBACK
Customer-Oriented User Story Frameworks
Outcomes over outputs
Align on ideal customer outcomes (or customer behaviors)
instead of outputs (or high-level goals).
Jobs to be done
Understand what customers are unable to do, and build out
specific “jobs” that help them accomplish their tasks.
“I wish I could ___, so I can ____.”
Frame user story as an action and a goal to help frame both
“the what” and “the why.”
How might we
A great brainstorm format that leaves solutions more open
ended.

56. SHARE
FEEDBACK
(the when)

57. CLOSE THE
FEEDBACK LOOP
1. Share how you
acted on feedback
with your
customers
2. Builds trust, and
encourages more
feedback

58. CLOSE THE FEEDBACK LOOP
● At Etsy, we personally
messaged sellers who helped
provide feedback
● And posted across all
communication channels to
say thanks and give heads up
that changes were coming

59. CLOSE THE FEEDBACK LOOP

60. nickey.substack.com

64. CUSTOMER-CENTRIC
FEEDBACK LOOPS:
WHY & HOW TO BUILD ‘EM
Product Management Today - November 2021
Nickey Skarstad, Director of Product, Duolingo

65. SIZE OPPORTUNITIES