Remote User Testing - MSU 12 Nov2021

Remote User Testing
Chris Farnum
M S U X A P 2 W D a y o f P r o f e s s i o n a l i z a t i o n | N o v 1 2 , 2 0 2 1

Hello!
O h w a i t , i t l o o k s l i k e w e ’ r e d o i n g a r e m o t e u s e r t e s t …
Part 1 – Presentation: An intro to
remote user testing and research
• Why you should
• When you shouldn’t
• Types of testing
• Tools
• Tips
• Q&A
Part 2 – Activity: Optimal Workshop
@crfarnum

@crfarnum
Q - How many of you have
done remote research?

@crfarnum
Introductions
C h r i s F a r n u m , L e a d U s e r E x p e r i e n c e R e s e a r c h e r – G e n e r a l M o t o r s
20+ years as an Information
Architect / UX Designer
3+ years as a User Experience
Researcher
Embedded in a UX design team
focused on mobile apps and
web apps.
Librarian in a former career

@crfarnum
Argus Associates
ProQuest
Compuware
West Bloomfield Township Public Library
Enlighten
Blue Cross Blue Shield of Michigan
General Motors
x2

@crfarnum
Why Remote?
COVID 19
Lab-less
Logistics / cost
Recruiting without geo limits
Scheduling
Flexibility

@crfarnum
https://measuringu.com/build-usability-lab/
my crazy desk at home

@crfarnum
When NOT to use remote testing?
Face to face
Observe non-digital interactions
Contextual research
Need precise measurements
and/or recording
Proprietary or secret content

@crfarnum
Photo by Alejandro Escamilla on Unsplash.
More about contextual research via UX Collective
When physical setups, camera angles matter...
https://www.lextant.com/remote-research/

@crfarnum
Tools
Testing Platforms
• Conferencing software + screenshare
• UserTesting.com
• UserZoom
• Usability Hub
• Qualtrics
• DScout
Prototype / Stimulus
• AxShare
• Proto.io
• InVision
• Optimal Workshop…

@crfarnum
Types of Remote Testing
• Prototype testing
• First click
• Card Sorting (open / closed)
• Tree / Nav testing
• A/B testing
• Competitive analysis
• System Usability Scale –
benchmarking
• Visual aesthetic ratings
• New concept / design thinking testing
• User needs research
• Diary contextual studies

@crfarnum
Chalkmark – Optimal Workshop

@crfarnum
Moderated v Unmoderated
Moderated
• live/real-time testing
• like a Zoom call
Unmoderated
• Set up tasks/questions and launch
• Participants complete on their own

Moderated
- Flexible
- Ask follow-up questions
- Ability to explain stimulus / proto
- Generates more empathy
- Team may be more engaged
- …
@crfarnum
Unmoderated
- Ability to ask targeted questions
- Consistent questions / less variables
- Less time needed to run sessions
- Efficient for A/B, card sort, surveys,
etc.
- Easier to get quant-esque, e.g. SUS
- ….
Hybrid?
Q – Have you tried both? What do you prefer?

@crfarnum
Recruiting
Yes! It matters who you test with.
AND it depends on…
• Amount of background context,
experience needed
• Usability or desirability?
• Time / budget / tools available

On a budget
- Tear off flyer or Social media / forum
post
- Email address or screener survey link
- Schedule with online calendar, Doodle
@crfarnum
All-inclusive
- Vendor panel
- Custom screen questions
- Site intercept
- Set schedule availability
https://templates.office.com/en-us/flyer-with-tear-off-tabs-and-art-tm16392565 https://www.userzoom.com/participant-recruiting/what-are-screeners-and-why-are-they-important/

@crfarnum
Testing for Mobile Designs
Can be a challenge…
• MANY device screen sizes
• Extra work to build proto
• Can you share a URL? (info security)
• Host solution viewer code
It’s worth it when…
• Touch interactions matter (i.e., side-scroll)
• Testing new patterns, navigation
• Testing live apps

Testing Mobile on Mobile
- Participant screen shares from own
device
- Hosted proto
- Live app, downloaded?
Tip: make sure to test proto while
building it to prevent rework and
surprises.
@crfarnum
Testing Mobile on Desktop
- Participant views on desktop
- Web based proto viewer displays
screen frame
- Screen cast from moderator
device, share control
https://www.flaticon.com/free-icon/responsive_896405?term=mobile%20devices&related_id=896405
Q – Have you ever tried testing a mobile design?

@crfarnum
UXRs and the Team
If you are a UXR…
• Be part of the process – Agile or other
• Communicate that you can support both user
research and testing
• Get involved with designers and stakeholders early
• Set goals
• Understand audience
• Plan scenarios and protos
• Ask them to observe, take notes, ask
questions
• Be visible – present your work, attend critiques

Moderated
ask t eam t o observe in real t ime
@crfarnum
https://www.usertesting.com/blog/invite-observers-live-conversation
Unmoderated
share videos, creat e highlight reels

@crfarnum
What should designers know about test prep?
If you are a Designer working with a
UXR…
• Build protos that demonstrate just-enough
• Key use cases (usually not edge cases)
• Need to be “tight” so that participants can
navigate without a moderator
• Consistent details matter
• Collaborate – build the script and proto in tandem
• Mobile design? Test it on your own device(s).

design like you’re right and
test like you’re wrong
@crfarnum
Expect explosions
leave t ime and space
f or it erat ions
Images via https://www.flaticon.com/
Jeff Patton OnAgile2017 Q&A
via Agile Alliance
Q – Have you ever been asked to “validate” a design?

@crfarnum
What makes a good test plan/script?
Ingredients for a successful remote test:
• Make sure to ask participants to “think aloud” and
to turn on their video camera
• Give participants a scenario, goal, situation, and
even an identity for task-based testing
• Use visuals to explain complex background and
scenarios
• Likert scales for ratings, multiple choice – great for
generating comments!
• Ask clear, straightforward questions – especially
for unmoderated scripts

T h i s i s M o r r i s F a m i l y. F r a n k
a n d D e n i s e a r e t h e p a r e n t s ,
a n d S a m a n d D i a n n a a r e t h e i r
t e e n a g e k i d s . F r a n k , a n
O n S t a r m e m b e r w i t h a p a i d
p l a n , r e c e n t l y l e a r n e d a b o u t
i t s m o b i l e a p p a n d d e c i d e d t o
c h e c k i t o u t f r o m t h e a p p
s t o r e …
@crfarnum
Prototype credit – Claire Koo
S u p p o s e D i a n n a i s g o i n g t o t h e
m a l l w i t h h e r f r i e n d s . H e r f r i e n d
c a m e t o p i c k h e r u p . O n t h e i r
w a y t o t h e m a l l , a c a r v e e r e d
s u d d e n l y a n d c o l l i d e d w i t h
t h e m …

@crfarnum
Analysis and Deliverables
Typical Artifacts
• Test plan
• Script
• Excel analysis
• Debrief session notes
• Report – often slides
• Executive summary
• Video highlight reels

Reporting Tips for Remote Testing
- Use test platform analysis
tools to accelerate analysis
and reporting
- If your study is qualitative,
make that obvious
- Use video clips when it
makes sense
@crfarnum
86.9% 5 - Very Easy
10.1% 4 – Easy
3% 3 - Neutral
Bar chart via UserTesting.com
UserTesting.com auto-creates
metrics that emphasize the
number of participants .

@crfarnum
Challenges & Frustrations
• Prototype glitches and
incompatibilities
• Hosting for video files
• Bad egg participants (especially
unmoderated)
• Account limits
• Reports ;-)

@crfarnum
Triumphs
• Remote testing resulted in 2x-3x more
studies
• Able to reach narrow segments – EV
drivers
• Well suited to pace of sprints as
design group transitioned to Agile
• Concept testing for new products and
a large-scale redesign

@crfarnum
Likely to Recommend?
5
It’s a game-changer!
Start small, take advantage of trials and
existing solutions and tools.

@crfarnum
Your Questions?
Thanks for listening!
Chris Farnum
crfarnum@gmail.com
Christopher.Farnum@gm.com
@crfarnum

@crfarnum
Break / Coffee Time! – 5 mins

@crfarnum
Imagine you have a research project f or Technology A t MSU …
https://tech.msu.edu/

@crfarnum
Background ( 1 0 0 % F I C T I O N A L – C h r i s m a d e t h i s u p . ) https://tech.msu.edu/
MSU IT wants to evaluate the tech.msu.edu to plan priorities for future enhancements.
You’ve done stakeholder interviews and have learned:
• Key target audiences: students, faculty, staff, visitors. In the last redesign, they organized the
site/home page according to these audiences. Now they want to know if this structure is working well.
• Call volume for the help desk is very high at move-in times and during peak weeks during the
semester. IT would like students and faculty to use more self-help to reduce call wait times.
• Support for distance learning and remote group projects is a continuing challenge. The
department is rolling out Microsoft Teams for class group projects. They want to get more people on
campus to adopt it. They have started training session events, but sign ups are slow.
• Analytics show that tip sheets and documentation are nearly always accessed via external (Google)
search or site search rather than browsing traffic from the Tech site.

@crfarnum
Imagine that you have Optimal Workshop available as a
toolkit for this project…
https://www.optimalworkshop.com/
Card Sort
(open/closed/hybrid)
Tree Test First Click Survey

@crfarnum
Instructions – 30 min exercise https://tech.msu.edu/
Step 1: In your group discuss and decide the following (put your answers on a shared slide or doc)
• Take 5 mins to review the site.
• Decide your test goals, audience, and the type of insights that you need to help move the project forward.
• Stuck? Try filling in the blanks: In order to know ___<main test goals>___ we need to learn __<missing
insights/knowledge gap>___ by getting input from __target audience___.
• Example: In order to design the best way to teach people how to start an e-scooter we need to learn what’s
difficult about it by getting input from people in an urban environment that haven’t tried an e-scooter yet.
Step 2: Choose a test method from Optimal Workshop.
• Explain why you chose this method.
• Will this study be moderated or unmoderated?
• What questions/stimuli/assets will you need for your test?
• (If time allows) Try getting started with a draft of your test. Bonus question – how will you recruit?
Step 3: We’ll regroup and discuss
https://www.optimalworkshop.com/

@crfarnum
Discussion / Wrap up
Go Green!
What was the goal of your study?
What method did you choose? Why?
What was difficult or easy about the activity?
What did you learn?
Thanks again for inviting me!

Remote User Testing - MSU 12 Nov2021

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Remote User Testing - MSU 12 Nov2021

Similar to Remote User Testing - MSU 12 Nov2021 (20)

More from Chris Farnum

More from Chris Farnum (14)

Recently uploaded

Recently uploaded (20)

Remote User Testing - MSU 12 Nov2021