1. Webinar:
How to Conduct Unmoderated Remote
Usability Testing

2. Intros
Alfonso de la Nuez
Co-Founder & Co-CEO
UserZoom
Nate Colker
UX Researcher
UserZoom
John Romadka
UX Researcher
Optum Insight
(Division of UnitedHealth Group)

3. Overview
Part 1 (20 min)
 Introduction to unmoderated remote usability testing
 A real life case study featuring a company in the insurance industry
Part 2 (30 min)
 How to plan for an unmoderated remote usability study
 How to design a study
 What metrics you need to collect
 How to recruit participants
 How to analyze the data
 Q&A

4. User testing, feedback & analytics tools mania!

5. Let’s try to make some sense out of this…
Many ways of organizing remote testing tools:
 Moderated vs. Unmoderated
 Qualitative vs. Quantitative-based
 User engaging vs. Non-user engaging (analytics)
 Usability focused vs. ‘quick user feedback’ (VOC)
 Video vs. Stats/metrics
 Etc.

6. Remote, unmoderated - A growing research method

7. Why this testing tool fever?
 Consumers more empowered than ever, thx in part to Social Media
 Ecommerce maturing, more competition
 Web analytics, usability test in the lab, mkt research… no longer
enough or are very costly
 Agile dev is all about speed, iteration
1. Designing & Testing UX quickly & cost-effectively is KEY
2. People want better!

8. Why Should You Care about Remote Testing?
 Cost-effective, save time and money on research
 You can quantify usability, obtain statistically significant UX data
 Run competitive UX Benchmarking
 Iterative testing, validate design/prototypes
 Test with users from different geographic locations
 Objectivity + accountability
 Improves communication
 Mitigate risks

9. What can be tested with URUT?
 Static mockups
 Wireframes
 Prototypes
 Multimedia
 Live websites
 Intranets
 …any web-based UI
 …anything you can
host on a server
and access through
a web browser

10. Pros & Cons
Pros:
 Cost-effectiveness
 Quantify UX
 Geographic representation
 Honest feedback
 Combination of results and metrics
Cons:
 Obviously… no face to face

11. Case Study:
Remote UX Studies - Tools & Methods
John Romadka, UX Researcher
OptumInsight (Division of UnitedHealth Group)

12. Challenges
New project
We’d depleted our pool of internal (agent) participants
locally
Needed to get a broader perspective from agents
across country
Network security issues
Limited number of dedicated UX Researchers

13. Challenges
Couldn’t convince business partners to test iteratively
Lab studies were costly
and often resulted in
large margin of error
90% confidence interval

14. Making the Case
Made the unmoderated testing business case
More cost efficient
• Unmoderated testing can cost 50% less than
traditional moderated testing
• Primarily due to reduction in effort hours spent in lab
Timeline
• Reduced the 3 week lead time for recruiting to
less than 1 week.

15. Making the Case
Validation Testing
Retest within days, instead of week
Moderated test: 180 hours vs. Unmoderated test: 40 hours
Remote Unmoderated Testing performed for ~20% the cost of benchmark
Moderated test
Test 1- (Moderated) = $10,000
Test 2- (Unmoderated) = $2000)
Improved Confidence Intervals

16. Results
Successful company-wide adoption of UserZoom
Obtained a pilot group of “Field” participants
Broad range of demographics, locations
Bi-weekly expectation of participating in whatever UX
research was available and ready to study
• Required less context, setup

17. Results
UserZoom allowed us to quickly setup/run/analyze
Online Surveys
Task-based studies
Card Sort exercises
Tree Testing studies
Task-based studies:
Moderated Remote
Unmoderated Remote
Moderated Lab

18. Results
Task Based Studies -
Remote Moderated: combine 3 tools
UserZoom (metrics)
Lead users through study
Central point for metrics
Quick and easy to re-test
Webex (screen sharing, audio)
Deployed company-wide
Giving participant control of web app on dev
Morae (note taking, recording)

19. Results
Task Based Studies -
Combined Unmoderated/Moderated
Users were onsite occasionally
Validate our unmoderated results
UserZoom: Used same setup for both
Consistent test method and process
Freed up the Facilitator to observe, take notes.
Knowing UserZoom was gathering metrics

20. Thank You!
@jaromadGplus.to/jromadka

21. How to Conduct Unmoderated Remote Usability Testing
• How to plan for an unmoderated remote usability study
• What metrics you need to collect
• How to design a study
• How to recruit participants
• How to analyze the data
Planning Design Recruitment Analysis

22. What are the goals of the study?
A Few Examples….
Evaluate and Quantify the User Experience
Comparing Multiple Designs (A/B)
Competitive Benchmark
Identify Visitors’ Core Tasks
Redesign Navigation Structure
Improve the Navigation Experience
Planning Design Recruitment Analysis

23. What are the goals of the study?
A Few Examples….
Remote Task-Based
True Intent
Card Sort / Tree Test
Evaluate and Quantify the User Experience
Comparing Multiple Designs (A/B)
Competitive Benchmark
Identify Visitors’ Core Tasks
Redesign Navigation Structure
Improve the Navigation Experience
Planning Design Recruitment Analysis
Method

24. A Few Examples….
Remote Task-Based
True Intent
Card Sort / Tree Test
Evaluate and Quantify the User Experience
Comparing Multiple Designs (A/B)
Competitive Benchmark
Identify Visitors’ Core Tasks
Redesign Navigation Structure
Improve the Navigation Experience
Planning Design Recruitment Analysis
Method
What are the goals of the study?

25. A Hulu Demo Study
Planning Design Recruitment Analysis

26. What Metrics do you Need to Collect?
 Jeff Sauro,
Measuringusability.com
Planning Design Recruitment Analysis

27. Create A Study Script
Recruitment AnalysisPlanning Design
• Study Goals & Participant Profile
• Welcome Page
• Initial Questions
• Task 1
• Post-Task Questions
• Task 2
• Post-Task Questions
• Final Questionnaire

28. Welcome Page Design
Recruitment AnalysisPlanning Design
Set Expectations for the participants
• Introduce study
• Time estimate and incentive
• Keep short and to the point
Purpose of
the study
Time
estimate
Incentive
Technical
requirements
Instructions

29. Example Welcome Page
Recruitment AnalysisPlanning Design

30. Initial Questions - Tips
Recruitment AnalysisPlanning Design
You may want to
collect what type of
experience users
have with the product.
You can use branching logic to
give certain tasks to specific
participants based on their
expertise
Avoid asking hypothetical
questions. Participants
are not good at knowing
how they’ll behave in a
certain situation.
“Would you book flights on
this website if we added this
option?” is a bad example
Be specific. “Daily”
and “Weekly” is
more specific than
“Always” and
“Sometimes”.
Avoid “double
questions” that have
two questions in one.
“Please indicate how easy
and fast it was for you to
complete the task?”
Avoid asking yes/no
questions. It can lead
participants into an
extreme statement.
“Do you use this website all
the time?” is not a good
example.
Don’t reveal too
much information
to the participant
upfront
Brand recognition can
have a large impact on
people’s reactions to
products.

31. Initial Question Examples
Recruitment AnalysisPlanning Design

32. Initial Question – Logic & Conditions
Recruitment AnalysisPlanning Design
• Skip participants to a specific task or question
• Hide or show questions based on answers (example above)
• Disqualify participants

33. Selecting the Right Tasks
Recruitment AnalysisPlanning Design
Balance of User Data and Business Goals
• What are users coming to your site to do (core tasks) ?
• Are there any tasks that are required to use your site?
• What drives revenue on your site?

34. Selecting the Right Tasks
Recruitment AnalysisPlanning Design
Balance of User Data and Business Goals
• What are users coming to your site to do (core tasks) ?
- Find and watch a TV show
• Are there any tasks that are required to use your site?
- Create an account?
• What drives revenue on your site?
- Sign up for a premium account

35. Task Construction - Tips
Recruitment AnalysisPlanning Design
Tasks should be
natural and easy to
understand, but not
easy to guess
Randomize task order
to minimize the
learning effect, if tasks
are not dependent on
each other.
Create tasks that
are easy to
validate (you need
to be able to
confirm that the
participant actually
found an answer).
Tasks should be
worded in a way that
doesn’t lead the
participant
When using Validation
word tasks so that
there can be a
discrete answer
(price, date etc.)
Include “none of
the above” option
(it will discourage
participants from
using deduction to
figure out the
answer).

36. Building Multiple Tasks - Randomizing Order
Recruitment AnalysisPlanning Design
Screener Pre-task questions
Hulu Task 1
Hulu Task 2
Control for learning effects
• Participants become more proficient as they use the site
There are exceptions, sometimes tasks must be completed in order
Final QuestionsRandomized

37. Task Validation (3 options)
Recruitment AnalysisPlanning Design
• Validate by question
• “Quiz question” used to verify task completion
• Validate by URL
• Successful if participant reach a specific URL
• Think carefully about task wording and defining success
• “Find episode 5 season 5 of Family Guy” (Is someone successful if they get to search results page vs.
the specific episode page)
• Self-reported

38. Task Description Example
Recruitment AnalysisPlanning Design

39. Post Task Questions
Recruitment AnalysisPlanning Design
Self-Reported Data
• Are user satisfied with the experience?
• Do users believe they completed the task successfully?
• Why wasn’t a participant successful ?
Task
Success
Button
Validation
Abandon Button Abandon Questions
Error Questions
Success Questions

40. Post Task Question Examples
Recruitment AnalysisPlanning Design
Validation
Question
Error
Question
Success

41. Final Questionnaire - Tips
Recruitment AnalysisPlanning Design
Overall, how easy or
difficult was it to use
this website?
How likely is it that
you would recommend
this website to a friend
or colleague?
After your experience
with the website
today, which of the
following words do
you associate with the
brand?
When using Validation
word tasks so that
there can be a
discrete answer
(price, date etc.)
OVERALL RATING
SCALES
NET PROMOTER
SCORE
CHANGE IN BRAND
PERCEPTION
OPEN-ENDED QUESTIONS

42. Final Questionnaire Example Questions
Recruitment AnalysisPlanning Design

43. What about recruiting participants?
AnalysisPlanning Design Recruitment
Define a screener
Hulu Demo Study
• Representative users
• Demographics (gender, age etc.)
• Control for experience
• Larger sample size -> more precise predictions
• Set profiles and define quota limits
• Set quality controls

44. What about recruiting participants?
AnalysisPlanning Design Recruitment

45. Quality Control Settings
AnalysisPlanning Design Recruitment
Quality Controls
• Be careful not to exclude fast users
• Cheaters often take the most efficient path (look out for “0 clicks”)

46. What about recruiting participants?
 3 choices to recruit participants:
1. Panel vendor (SSI, Research Now, etc)
2. Private Mailing List
3. Intercept visitors
AnalysisPlanning Design Recruitment

47. Hulu Demo Walkthrough
Planning Design Recruitment Analysis

48. Analyzing the Results
Planning Design Recruitment Analysis
1. Task Success
2. Efficiency (time, clicks)
3. Self-Reported Satisfaction
4. Open prompts
5. Behavioral Data

49. Analyzing the Results – Clean Data
Planning Design Recruitment Analysis
Review your data for outliers / “bad data”
• Very High / Very Low task times
• 0 Clicks
• Nonsense Answers

50. Make assumptions about user population (Confidence Intervals)
Hulu baselines can be compared with data from future study (Significance Testing)
Analyzing Metrics – Beyond the Tool
Planning Design Recruitment Analysis
Jeff Sauro,
StatUsabilityPak

51. Analyzing Metrics – Open Ended
Planning Design Recruitment Analysis
Use Text Clouds & Search for Patterns
Use Comment Coding for more Detailed Analysis

52. Analyzing Metrics – Behavioral
Planning Design Recruitment Analysis
What strategies are being used (how successful are they?)
What are “Non-Success” participants doing? (filter data)

53. Q&A

54. Thanks so much for your time!
Check out our upcoming news and events at www.userzoom.com
Sunnyvale (USA)
440 N. Wolfe Rd. Sunnyvale,
CA 94085
Phone: +1 (408) 524 7445
Contact: Alfonso de la Nuez
Barcelona (Spain)
Av. Diagonal 419 3º 2ª 08008
Barcelona Phone: +34 93 414
7554 Fax: +34 93 209 8380
Contact: Xavier Mestres
Cheshire (UK)
Booths Park. Chelford Road, Knutsford,
Cheshire WA16 8GS
Phone: + 44 (0)1565 759890
Cell: + 44 (0)7900 472 920
Contact: Arthur Moan