Digital product teams are moving faster and the desire to learn directly from users is increasing. When research headcount hasn’t caught up, others in the organization are taking matters into their own hands. How can we shift our role from the research bottleneck in the product development process to the coaches who empower more decision-makers to speak to users directly? We’ll share the strategy we took to successfully scale the amount and impact of research happening at a fast-growing medium-sized startup as a team of three researchers. You’ll learn how we: Empowered stakeholders to answer their own tactical questions through research training workshops and self-serve resources. Initiated a knowledge base that allows stakeholders to synthesize fresh insights from across past studies, increasing the lifespan of past and new research findings. Slowed down and invested in research operations in order to move faster later.

1. SCALING RESEARCH
WITHOUT SCALING
THE TEAM
Cordelia Hyland | Jordan Berry | Erik Olesund
Thumbtack
UXPA 2019

2. Why did you choose this talk?

3. Let’s begin with a story.

4. Research team
Thumbtack

17. SOLO SMALL BIG

18. Erik Olesund
Leads research on how
customers search and discover
the right pros to work with
Cordelia Hyland
Leads research that helps pros
manage the services they offer and
their conversations with customers
Experience Researchers at Thumbtack
Jordan Berry
Leads research that helps pros match
with the right customers

19. SCALING RESEARCH WITHOUT SCALING THE TEAM
1 2 3
Help each
researcher
achieve more by
building scalable
processes for
repeat activities
Increase the value
and relevance of
existing research
by making past
insights easy to
discover
Make our
colleagues
stronger research
partners through
trainings and
coaching

22. Our
Pros
Hundreds of thousands
of Pros available in
every county in the U.S.
Top categories
1. Handyman
2. General Contractor
3. Cleaner
4. Painter
5. Photographer
Our
Customers
Millions of customers
each year, starting
projects across nearly
500 categories.
Largest regional markets
Atlanta, Washington DC,
Denver, New York, Virginia
Beach

25. Scale the researchers
How we built research processes that scale the
reach and impact of ourselves
1

27. Ways we’ve scaled the researchers
● Team branding
● Templates
● The pro online community
● Partnered with customer support for phone
surveys

28. ● Stickers!
● Slack emojis
● Custom report templates
Team branding

31. Templates
● Study plans
● Reports
● Observer note taking doc
● Group debrief framework

32. Framework for facilitating
group synthesis

33. Thumbtack Community

34. In review
● Build a team brand
● Create templates
● Leverage your customer forums

35. Partnered with
customer support for
phone surveys

36. The tiger team

37. Who are they?

38. What do they do?

39. Why did a
professional
recently churn?
Why did a
professional decide
that a lead wasn’t
the right fit?
What are the types of questions we ask?
1 2

40. How do we
analyze the data?
1
2
Use a form that does
double duty—as a
script and call notes
Analyze the calls as
you would a normal
study

41. How does this help scale our research?
● Research happens in the background,
concurrent with other studies
● Scales non-researchers, too

42. If you choose to do this
● Be sensitive towards the team’s time
● Provide real-time feedback to agents
● Clarify the purpose of the call for the user

45. What are existing resources you
can leverage?

46. SCALING RESEARCH WITHOUT SCALING THE TEAM
1 2 3
Help each
researcher
achieve more by
building scalable
processes for
repeat activities
Increase the value
and relevance of
existing research
by making past
insights easy to
discover
Make our
colleagues
stronger research
partners through
trainings and
coaching

47. Scale existing research
How to make past insights discoverable
and relevant today
2

49. More research. More data.

50. “Where is that video from the
customer we met in New York?”

51. “Didn’t we ask a similar question in
your study two months ago?”

52. “Can you send me any research
reports we have about X?”

54. How do we make past
insights discoverable and
relevant today?

55. What we did.

57. Is this the right approach for us?
Who is going to populate the database?
How do we develop a consistent yet flexible
tagging taxonomy?
How do we determine what qualifies as an
insight and what doesn’t?
Will our research partners be able to find the
relevant “nuggets”?
What about participant privacy?

58. Make insights discoverable.

59. Make insights discoverable.
For ourselves.

60. Make insights discoverable.
For ourselves.
And for our teams.

61. For ourselves
For our teams
+

62. Data repository
● Store recordings of
qualitative research
● Automatic
transcription
● Tag and search

63. “What do we know about [x]?”
How are pros managing their budgets?
How do customers use reviews?
What features do pros value most?
How are customers searching?
How is [x] feature perceived?

64. The Research
Archive makes
past insights
discoverable to
the rest of
Thumbtack

65. Overview of
completed
research projects
Each entry has
● Link to report
● Keywords
● Metadata
Anyone can
filter/search the
archive

68. Impact
Just in time answers.

69. One tool
doesn’t rule
them all
Organizing
your assets
are worth the
hassle
Make it easy
to search
1 2 3
What we learned from making past
insights easy to discover

70. What are you doing to keep
past research insights alive?

71. SCALING RESEARCH WITHOUT SCALING THE TEAM
1 2 3
Help each
researcher
achieve more by
building scalable
processes for
repeat activities
Increase the value
and relevance of
existing research
by making past
insights easy to
discover
Make our
colleagues
stronger research
partners through
trainings and
coaching

72. Scaling the contribution of
others
Making our colleagues stronger research partners
through trainings and coaching
3

73. The challenge.

75. Execute ongoing research to
support product teams
Conduct proactive, strategic
research
Research operations Scale research
What does the experience research team do?

76. Execute ongoing research to
support product teams
Conduct proactive, strategic
research
Research operations Scale research
What does the experience research team do?
Where
we
wanted
to be

77. Execute ongoing research to
support product teams
Conduct proactive, strategic
research
Research operations Scale research
What does the experience research team do?
Where
we
wanted
to be
What
we
needed
to do

79. How do we raise the
quality bar of all research
that happens at
Thumbtack?
1

80. How do we help non-
researchers know when to
do their own research and
when to partner with
research?
2

81. TRAINING PROGRAM

82. workshops
TRAINING PROGRAM

83. workshops 1:1 coaching
TRAINING PROGRAM

84. Workshop
content
● What research is (and isn’t)
● How to work with us
● Moderating practice
● Cognitive biases

86. Cognitive
biases 101

87. Biases that impact
our interpretation of
what we heard
Confirmation bias
Clustering illusion
Recency bias
Biases that impact
how the participant
responds
Observer-expectancy
Social desirability
False recall

88. Moderating
practice
research question
≠
interview question[ ]

91. Interview a participant
to better understand
behavior around a
specific action
1 moderator
Be yourself and
answer truthfully
1 participant
Use the checklist
to analyze how well
the moderator is
doing and provide
feedback
1 observer
ACTIVITY OVERVIEW

92. Interview your participant to understand
more about their sleep.
You want to understand their habits, values, and
how they fall asleep in order to inform the design
and messaging of a new sleep assistant app.
PROMPT

94. What we learned.

95. Break them
down to build
them up

96. Biases are
sticky, really
sticky

97. “I noticed that I often ask
leading questions and fall into the
trap of confirmation bias. So this
was extremely helpful for me
to be more aware of!
– workshop participant

98. Cater the
content to
the audience

99. Some training
is best left for
1:1 coaching

100. The impact.

101. 180 participants
4 teams
14 workshops

102. 67 studies across Thumbtack
In the last eight months
33 researcher-led
34 researcher-advised
21 individual non-researchers
with a tiny team of researchers
plus guerrilla research, product
feedback at monthly events, customer
advisory council calls, community
forum posts, tiger team calls

104. “User research is key to making a
great product and I love that the UXR
team is empowering teams to
conduct their own research and
welcoming opportunities to partner
on studies.
– workshop participant

105. SCALING RESEARCH WITHOUT SCALING THE TEAM
1 2 3
Help each
researcher
achieve more by
building scalable
processes for
repeat activities
Increase the value
and relevance of
existing research
by making past
insights easy to
discover
Make our
colleagues
stronger research
partners through
trainings and
coaching

106. THANK YOU

107. What’s working for you
to scale research?

108. Q&A
Jordan Berry
jordanb@thumbtack.com
@jordanberry
Cordelia Hyland
cordelia@thumbtack.com
@cordelia_hyland
Erik Olesund
erikolesund@thumbtack.com
@olesund

109. You?
+
+

110. We need help
feeding the
Thumbtack
monster!

111. YES, WE’RE SCALING THE TEAM
Interested in joining us?
bit.ly/thumbtackresearchishiring
or scan the QR code
Jordan Berry
jordanb@thumbtack.com
@jordanberry
Cordelia Hyland
cordelia@thumbtack.com
@cordelia_hyland
Erik Olesund
erikolesund@thumbtack.com
@olesund

113. EXAMPLE
SLIDES
BELOW

114. SOCIALIZING

115. 67 studies across Thumbtack
In the last eight months,

116. 67 studies across Thumbtack
In the last eight months,
33 researcher-led

117. 67 studies across Thumbtack
In the last eight months,
33 researcher-led
34 researcher-advised

118. 67 studies across Thumbtack
In the last eight months,
33 researcher-led
34 researcher-advised
21 individual non-researchers

119. 67 studies across Thumbtack
In the last eight months,
33 researcher-led
34 researcher-advised
21 individual non-researchers
plus guerrilla research, product
feedback at monthly events, customer
advisory council calls, community
forum posts, tiger team calls

121. +

122. How do we scale the
impact of research without
scaling the team?

123. 1. Summarizing a program of research
2. Documenting State of X
3. Reframing to answer specific questions

124. Take the time to compare
findings to past studies.
1

125. IF A BOTTLE FELL ON THE
FLOOR AND NO ONE WAS
AROUND TO HEAR IT, DID IT
MAKE A SOUND?
Image source: Time Magazine

126. SCALING RESEARCH
WITHOUT SCALING THE
TEAM
Cordelia Hyland | Jordan Berry | Erik Olesund
Thumbtack
UXPA 2019

127. ● Org/leadership
○ Promise to move faster
○ Empower their team members to answer their own
questions
○ Trainings curated by team and their needs
● IC teammates
○ Referencing current questions their team is asking
○ Show buy-in from leadership
How did we get buy-in for the
trainings?

130. FAST SLOW

131. PACE
IMPACT