Slides from my Remote Research workshop given on 8/22/2012 at UX Week 2012.

1. Remote Research
Workshop

2. Photo Credit: kaboodle.com

3. Remote Design Research
In person
Remote (2008)
Remote (2012)
Definition:
Research where the
participant is not sharing
physical space with the
researcher.
% of qualitative research done remotely*
* Source: My wild guess

4. But it is pretty useful
...in certain situations

5. THAT IS WHY WE GO REMOTE

6. #UXWEEK
#RR

7. Btw I’m Cyd Harrell
and assisting me is Sarah Spitler

8. 6 YEARS
187 REMOTE STUDIES
FAVORITE TOOLS: EMPATHY &
DUCT TAPE
(R.I.P.)

9. OXYMORON?
REMOTE EMPATHY

10. DUAL LENS
STREAMING
FIELD RESEARCH
• Livestream
• Flip Cam
• It’s the view
finder, silly
• Actual duct
tape

12. Since 2007

13. Can tools make us better
researchers?

14. You can
set up a
pretty
sweet
rig

15. But if
you’re not
asking the
right
questions
it doesn’t
matter.

16. This is a great book by my old boss.

17. Let’s jump right in

18. What’s New with UX Research Tools?

19. A DISTINCTION

20. Moderated versus Automated

21. Moderated versus Automated
10% 90%

22. Moderated versus Automated
Photo Credit: singularityhub.com

23. Moderated
GoToMeeting 5.2
Integrated Recording
That’s really awesome

24. Awesome compared to this!

25. Moderated
Hugging
At Mozilla
and MailChimp
Photo Credit: Diane Loviglio, Mozilla

26. Picture in picture
Mobile
Recording
Nice face,
earthling

27. Automated
Mobile App
GestureTracking

28. Automated
Usabilla
Mobile

29. Task-driven
Experience Sampling
iOS & Android
Pix or it didn’t
happen!

30. iOS
Storytelling

31. Beyond Usability

32. HOW TO BE
graceful
When All Your
TECHNOLOGY
BREAKS

33. Follow the buddy system

34. It’s laptop time

35. remoteresear.ch
or dev.ethn.io/18014

36. (BROWSER TIME)

37. IN CASE THE INTERNET FAILS
Research experience
Most important thing to get out of today?
Biggest fear or skepticism about remote research?

38. PRO TIP #1: MAKE A LANDING PAGE
bolt.ps

39. Thanks, Volunteer!

40. No sales

41. HOW TO BE
graceful
When All Your
TECHNOLOGY
BREAKS

42. WHAT WE’LL COVER TODAY Recruiting
Moderated
Automated
Results
Pitfalls

44. SOME THOUGHTS ON WEB ANALYTICS
What do helicopters do in action movies?
Explode

45. Now we do a little experiment

46. EXERCISE #1
“Giraffe”
Binder PDF Page 4
Table groups
Pick one moderator and one participant
(they need to be in different locations)
10 minutes of moderating

47. MODERATOR Instructions
Your participant has a sheet of paper and that’s it
Goal: find out what kind of physical task
instructions online
Hint: ask them to give you a play by play of what
they are doing
Try and follow along with your own paper
NO helping, even if you could.

48. It’s A RACE
You only have 10 minutes to do this
max.

49. Surprises?

50. Was there emotion?

51. OXYMORON?
REMOTE EMPATHY

52. Taking Notes
Second monitor if
possible
Excel Hack
(remoteresear.ch/samples)

53. Observer Involvement
IM
Campfire (!)
Integrated Chat
Passing Notes

54. WHEN DOES REMOTE WIN?

55. NATIVE ENVIRONMENT
CHEAP ETHNOGRAPHY, NOT CHEAP LAB.

56. TIME-AWARE
RESEARCH IS THE BEST

57. I’LL TRADE YOU TIME
FOR PHYSICAL PRESENCE

58. INVOLVED OBSERVERS
Easier to collaborate in real time.

59. GEOGRAPHY
TITLE TEXT

60. WHEN DOES REMOTE LOSE?

61. BANDWIDTH
Yes, it’s still an issue.

62. TRANSLATION
It’s expensive and difficult to do live.

63. MINORS
Consent is a bear.
Especially if they’re under 13.

64. FACES
Most users don’t have webcams.
Webcams are a little invasive.

65. SUPER DUPER SECURITY

66. AN EPIC FAIL.

68. We basically did
Bad Research

69. Kids will lie for coins
People mis-assessed their English levels
Bandwidth was really a problem
Kids don’t play when their parents are home

70. PRO TIP #2: HOW TO FIND ACTUAL SPEAKERS
OF YOUR LANGUAGE IN ANY COUNTRY:
“ What happens when you let go of a balloon? ”
“ Qu'est-ce qui se passe quand vous lâchez un
ballon? ”

71. DESIGNING
REMOTE STUDIES

72. Yo Marv, get me some
insight on WHY people are
bailing out of checkout.

73. Sure, Boss.
Let’s combine
methods.
Intercept &
observe 10
users live, and
do 10 self-
moderated with
usertesting.com

74. HEART & SOUL OF REMOTE RESEARCH
Not scheduling participants.

75. TIME-AWARE TASKS
Told to find purple pumps. Has to wear this for sister’s wedding this
weekend.
Doesn’t care.
Whole different interview.

76. SCRIPT FOR CUSTOMIZATION
“Please look for a pair of purple pumps in
your size.”
“Please look for a pair of shoes you’d like
to buy.”

77. REAL TIME OBSERVERS

79. WHAT IF IT’S NOT USABILITY?

80. EXPERIENCE SAMPLING

81. PROTOTYPES, WIREFRAMES,
SKETCHES?

82. HECK YES.

84. RECRUITING
FOR REMOTE

85. RECRUITING OPTIONS
Ethnio via Web or Twitter
Links or house ads to online forms
Recruiting agencies
Customer email lists
Panels
Craigslist/online ads

87. RECRUITING OPTIONS
realness
Grab someone in the hallway
Grab someone who isn’t directly involved
in your project
IM your friends for a quick look
Panels and recruiting agencies
Real customers from marketing lists
Site & offsite intercepts

88. GOLD STANDARD
Live recruiting
Get as close as you
can

89. USE ETHNIO WHEN
You have Website access
(or Twitter conversations)
Some budget
Quotas are relatively simple

90. TWITTER RECRUITING

91. ANATOMY OF A RECRUITING TWEET
@person we’re interviewing [job or characteristic] about
[topic] on [dates] (paid research) -interested? [LINK]
I’m looking for [job or characteristic(s)] for a [research
activity] on [dates] incentive $X [LINK] please RT
Pro tip #3: edit your profile to mention “UX Researcher”
and “Current Study: Martian Life”

92. USE FORMS WHEN
You can place a link but not code
Low budget
Quotas are relatively complex
Options:
Google Docs
Wufoo

93. Screener constraints
Keep it short
Always ask an open-ended
question
Different from surveys

94. USE AGENCIES WHEN
No website access
People whine about first-time
visitors
Higher budget
Online recruiting failed

95. CUSTOMER LISTS
Sub-segments
Ownership
requirements
No website access

96. PANEL

97. Would You
Use This?

99. CRAIGSLIST
When other methods don’t work
No dough
Takes a while
Embarrassing to admit to your friends

100. LIVE RECRUITING MATH
Target: 6 recruits/hour
Allow for: 1% response (low average
You Need: 10,000 uniques/day

101. TITLE TEXT

102. remoteresear.ch/samples
“Online Consent”

103. MODERATED must-haves
Consent
Tech set-up
The beef
De-installation
Thanks & incentive

104. AUTOMATED must-haves
Expectation setting
Instructions
The beef
Thanks & incentive

105. Recruiting
Same: Good “talkers”
Same: Demographic fits
Different: Relevant current
tasks
Different: Compatible tech

106. Logistics: What & How
Provide test
target
Access
Credentials

107. Logistics: Who & When
How to connect with the user
Managing time zones
Session length

108. Paying Incentives
Collect minimal
personal info
Deliver quickly
Appropriate for
location

109. STAGED Incentives
For Contextual Inquiry
Start with $75, then at the end offer
$25 more for environment photos
Self-made diary studies

111. EXERCISE #2
okay, draft a screener
Binder PDF Page 6
Use ethn.io or dev.ethn.io (beta) or wufoo
Under 8 Questions
Advanced Bonus: create an automated
and moderated screener

113. Marv-o - what tools should
we pick? I mean, tell me
what to use for pete’s sake

114. Sure, boss.
There is an
AVALANCHE
of UX TOOLS.

115. “At least a little
behavioral”
No surveys.
No standard web
analytics.

116. 5
CATEGORIES

117. Moderated
Self-Moderated
For A Long Time
Automated Live
Conceptual
Automated Static
Web Analytics

118. Talk To People
People Talk To A Machine
For A Long Time
Tasks On A Live Site
Tasks On Concepts
Tasks On Images
Fancy-Ass Analytics

119. Insight
Ammunition
For Ad Agencies
More Ammunition
I.A. & Navigation
Still More Ammunition
Just Better Analytics

120. Moderated
Self-Moderated
For A Long Time
Automated Live
Conceptual
Automated Static Deep
Behavioral UX
Web Analytics Insight

121. Moderated
Opinions
Self-Moderated
For A Long Time
Automated Live
Conceptual
Automated Static
Web Analytics

122. Moderated
}
Self-Moderated
For A Long Time
Mobile
Automated Live
Conceptual
Automated Static

123. Remote Research Toolkit
UNMODERATED (AKA AUTOMATED)

124. 3 parts
Provide test target
Record user behavior
Capture comments (optional)

125. USABILLA
Clicks & Questions
Heatmaps
You do your own recruiting
$49 - $199

128. LOOP11 (UN-MODERATED)
TITLE TEXT

129. TITLE TEXT

130. GOTOMEETING - $49/MO

133. TITLE TEXT

140. WHAT WE USE. A LOT.
• Ethnio + GTM
• Usabilla + Usertesting.com
• Silverback (Mac) & Camtasia (PC)
• Guerrilla mobile

141. EXERCISE #3
Rolling a design for your next study
Page 8 in your binder

142. Talk To People
People Talk To A Machine
For A Long Time
Tasks On A Live Site
Tasks On Concepts
Tasks On Images
Fancy-Ass Analytics

144. EXERCISE #4
Either “unmoderated” or “practice interviewing”
usertesting.com
usabilla - 2 months free (Sign up for
an account, but cancel on the payment page. Send an
email to support@usabilla.com and mention Nate &
Webvisions)
ethnio - 1 month free or trial.
CODE: WORKSHOPLOVE

145. RESULTS

146. REMEMBER, BE
graceful
When All Your
TECHNOLOGY
BREAKS

147. WHAT USERS WORRY ABOUT

148. WHO ARE YOU AND CAN
I TRUST YOU?

149. IF YOU CAN SEE MY
SCREEN,
CAN YOU SEE MY
FACE?

150. WILL YOU BE ABLE
TO TAKE
CONTROL OF MY
COMPUTER?

151. WILL YOU HELP ME GET
THE PLUG-IN
OFF MY COMPUTER AT
THE END?

152. WILL THIS WORK IF I HAVE A
FIREWALL/PROXY/STRICT IT
DEPARTMENT?

153. IS THIS GOING TO END
UP ON
YOUTUBE?

154. So, what could go wrong?

155. 3 points of failure
Technology
yours
participant’s
observer’s
Recruiting
Test Design

156. Control what you can control
Which is, your set up.
You wouldn’t go backpacking
without checking your gear.
Do it 24 hours in advance.
OR ELSE!!

157. Prepare for what
you can’t control
User’s dog trips over power
cord
Internet affected by local
weather
User can’t install
Doesn’t know how to install

158. Live Recruiting
Problems
What if they gave a study and nobody
came?

159. Increase exposure or try social media
When you’re not getting
enough views
When the wrong type of
users are responding

160. Increase incentives
When it’s cheaper
When you’re almost getting
enough
When segments are tough

161. Switch methods
When base traffic level isn’t high enough

162. Mid-Session
Tech Problems
mar tian?
act ual

163. There’s always a way

164. If NASA can do THIS from 153 million miles away, we
can meet our users where they are, even if we’re a
couple thousand miles away.

165. cyd@cydharrell.com
@cydharrell
Ethn.io promo code: WORKSHOPLOVE

166. Thanks for coming.
Go run some awesome research.