Unmoderated remote usability testing

1. www.userzoom.com
Tips and Tricks for
Remote Unmoderated Usability Testing
Alfonso de la Nuez
Founder at UserZoom
May 2009
1

2. www.userzoom.com
Overview
• Introduction to Remote Unmoderating User Testing (RUUT)
• What is the value and how does it work?
• When and how to use it as a research method
• Best practices and examples

3. www.userzoom.com
What does the market say about web research & analytics?
Usability testing in the lab is great, but it has its limitations
Web analytics tells you a lot about the WHAT, not the WHY
Conducting international testing is expensive
Benchmarking and measuring UX is more important than ever

4. www.userzoom.com
Overall conclusion:
Today’s Web and E-business Managers seek more
sophisticated and innovative research and analytics products
to help them compete

5. www.userzoom.com
Comparing various research methods for usability research
Test REAL Large Behavioural Natural
tasks Samples Reasons WHY Data Setting
Focus Groups
Usability Labs
Online surveys
Web analytics
RUUT

6. www.userzoom.com
RUUT: What is the value and how does it work?
1. No face-to-face, human moderation (moderation through software)
2. Hundreds of users participate simultaneously…
3. …from geographically spread locations (local & int’l) and…
4. …in their natural context (using their own PCs)
5. Software captures both usability data and users’ behavior
Very cost-effective and faster

7. www.userzoom.com
Define your study Launch study, collect data Analyze results &
& monitor progress generate a report
Choose the type of study Invite users to participate Perform advanced data
you want to conduct (or ask your visitors to do it) analysis
Write your study script Collect data automatically Use filtering capabilities
Choose the recruitment Monitor the progress real- Copy & paste results
method & audience time Export automatically into
Pre-test before launch Word or Excel

8. www.userzoom.com
What about user recruitment?
Option 1: Panel companies*
• Million users, spread across the world
• Wide variety of profiles, great segmentation capabilities
• Strict, professional control and panel mgmt.
Option 2: Intercept method
• Ask users who visit the website to participate in a study.
• Insert a javascript in the page
* Use a panel company
Option 3: Private panels, mailing lists
• We can also work with private databases

9. www.userzoom.com
When to use this research method
Whenever you need to quantify usability problems, not just identify them
If you need to test users in various locations (local or international)
For benchmarking and UX measurement projects
To obtain user behavior data (search strategy, click-streams and click-maps)
When you want to combine usability and market research
For quick pre-testing during the development cycle (prototype testing)

10. www.userzoom.com
Task based studies: Real users attempting real tasks
Websites

11. www.userzoom.com
Task based studies: Test any web-based interface
Prototypes

12. www.userzoom.com
Task based studies: Test any web-based interface
Mobile interfaces

13. www.userzoom.com
Also, you can conduct…
• Online Card sorting exercises
• True Intent studies
• Free search studies
Let’s take a look…
(all videos are taken from pilot studies)

14. www.userzoom.com
Card sorting: Open and closed categories

15. www.userzoom.com
True intent studies (through intercept method)

16. www.userzoom.com
Online user behavior research (through free search studies)

17. www.userzoom.com
Important! Remote does not mean you’re not in control
About scripting capabilities:
• High-level of flexibility
• Advanced task validation criteria (by question, by page interception, by time...)
• Advanced questionnaires tool (all type of questions: Choice, scales, open...)
• Insert multimedia (images, videos, flash, applets, sounds...)
• Manage advanced logic/branching & conditions
• Behavioral questionnaires
• Task randomization
• Time-out options
• Sample quality control

18. www.userzoom.com
Ok… What about the results you get?

19. www.userzoom.com
Effectiveness & efficiency ratios, plus answers to the questionnaire

20. www.userzoom.com
You have filtering capabilities

21. www.userzoom.com

22. www.userzoom.com

23. www.userzoom.com
Text clouds to analyze open-ended questions

24. www.userzoom.com
Clickstreams per task

25. www.userzoom.com
Clickmaps/heatmaps per page

26. www.userzoom.com
Finally, about reporting & exporting capabilities
Generate automatic reports (.doc, .excel, .ppt)
Copy & paste capabilities
Export raw-data for further advanced analysis
Share results online

27. www.userzoom.com
Any examples?

28. www.userzoom.com
Testing users in various locations (local or international)
Monster’s UX Director:
“Need to test various local sites, in 5 countries, but
I have a limited budget and time plan”
They tested 200 users online, in 5 markets
Data was collected in approximately 1 month
Visitbritain.co.uk
A year later the study was repeated

29. www.userzoom.com
Need to quantify usability issues, not just identify them
Spanair’s Marketing Director:
“Loved the results from the lab, but I simply can’t Spanair is Spain’s 2nd
largest airline
make a decision based on the feedback from 10 users”
The airline ended up testing 15 users in the lab (included
eye-tracking), then added 229 users online
Tasks were identical, results were not…

30. www.userzoom.com
Nº Nº Success Success
of users of users rate rate
Task (Lab) (remote) (Labs) (UZ)
Find a flight of a special class 3/15 185/229 20 % 71 %
Register on the web 4/5 5/12 70% 42 %
Check your current balance (loyalty
program) 8/15 65/73 53.3% 88 %
Spanair is Spain’s 2nd
largest airline

31. www.userzoom.com
For UX benchmarking & brand proposition research
Arsys’s Marketing Director:
Arsys is Spain’s
“We’d like to measure both user experience and leading ISP
branding metrics over a period of 3 years”
They test 200 users each semester
UK’s Training &
Upload results in a UX Dashboard Development Agency

32. www.userzoom.com
When to use unmoderated RUT Examples
Testing users in their natural context of use
Badoo’s Management had 3 goals and issues in mind:
Badoo is a UK-
1. Were concerned about testing in the lab based, fast-growing
online dating and
2. Wanted to check differences between cultures social network site
3. Wanted to combine usability testing with brand-value
proposition testing
RUUT was the perfect solution for them:
Were able to ask private matters openly
Were able to test in 5 markets at once and
compare results
Asked users about their brand-value perceptions

33. Thanks and have a great day!
Would you like a demo?
Contact us and we’ll offer you a chance to see and try out our technology
Do you have any questions?
We’re sure you have many questions. Don’t hesitate to contact us and we’ll be glad to answer
them.
Contact information
Alfonso de la Nuez Virginie Glaenzer Xavier Mestres
Founder & Int’l Business Dev Sales Manager, USA Founder & CTO / R&D
alfonso@userzoom.com vglaenzer@userzoom.com xmestres@userzoom.com
USA Office
Members of 440 N. Wolfe Rd. - Sunnyvale, CA 94085 (USA) · [t] +1 (408) 524 7445
Spain offices
Av. Diagonal 419, 3º 2ª - 08008 Barcelona - T. +34 93 414 75 54 - F. +34 93 368 42 95
Gran Vía 62, 7º Izda. - 28013 Madrid - T. +34 91 542 62 52 - F. +34 91 298 08 92
www.userzoom.com

34. www.userzoom.com
Introduction: Our ‘elevator pitch’
1. What is UserZoom and what do we do?
• International market research firm, specializing in online
user experience and usability research.
• We help brands boost website performance and improve
online business revenues.
2. How do we do it?
• We create a unique web-based software application
that empowers researchers to conduct highly
sophisticated online research.
• We offer both a self-service solution and professional
services.
3. Bottom line… Why should you work with us?
• You’ll be able to capture & measure the critical and
actionable data about your users’ online experience
that you need to manage your business, and no other
tool can provide!