How To Run Your Own Guerrilla Usability Testing

How to Run Your Own

Guerrilla Usability
Testing
A Guide By
John Chan & Jonathan Kift

April 18, 2012

Friday, 20 April, 12

So What Is Usability
Testing Anyway?

@jtcchan & @daftkifty


So what is
USABILITY TESTING anyway?

“ Usability really just means making sure that something
works well — whether it’s a Web site, a fighter jet or a
revolving door.
Steve Krug
Author of “Don’t Make Me Think”



So what is
USABILITY TESTING anyway?

Usability testing usually involves

1. Making a series of tasks for people to complete and
2. Noting any problems they come across



Here’s an example:



Why Should We
Test?



Why should we test?

Cultivate Surprises
It is an environment where we can find out how we intend
something will be used vs. how it actually is being used.



Why should we test?

Testing Works 99.99%* Of The Time
Even a ‘bad’ test with the wrong users can show you important
things you can do to improve your site.

* Can someone really make a design worse from testing?
Possibly but I honestly believe you can’t do wrong.



Why should we test?

Buy-In From Team & Stakeholders
User testing puts stakeholder and teams on common goals
with data. Also, great for informing decisions with greater
confidence. (e.g. picking “a” vs “b”)



What Should We
Test and When?



What should we test and when?

When

Research

What
Current site
Competitors site
New Ideas




When

Research Design

What
Current site Paper Prototypes
Competitors site Wireframes
New Ideas Mockups




When

Research Design Develop

What
Current site Paper Prototypes HTML Prototypes
Competitors site Wireframes Navigation
New Ideas Mockups Interactions




When

Research Design Develop Evaluate

What
Current site Paper Prototypes HTML Prototypes Live Site
Competitors site Wireframes Navigation A/B Testing
New Ideas Mockups Interactions Multivariate Testing



How to Run Your
Own Test



How to Run Your Own Test

What you’ll need:
1 x Facilitator : someone that explains and goes through the tasks




What you’ll need:
1 x Observer/Recorder : someone that notes key observations




What you’ll need:
1 x Observer/Recorder : someone that notes key observations
1 x Laptop : for the participant to perform their test




Nice to haves:

Some Money
(for recruiting!)




Nice to haves:

Some Money Screen Recorder
(for recruiting!) (or a video camera)




Planning

Before During After




Planning Facilitating

Before During After




Planning Facilitating Analyzing

Before During After



Planning Your Test



Planning Your Test

1 Preparing Tasks



Planning Your Test

1 Preparing Tasks

2 Recruiting Participants



Planning Your Test

1 Preparing Tasks


3 Finding a Location



1 On Preparing Tasks

Pick Tasks Based On Your Site/User Goals
What is your site trying to accomplish? What are your users
trying to accomplish?




Keep It Under 15 Tasks
This takes about 45 minutes to complete, which is perfect if
you require participants an hour of their time. It also reduces
the amount of data analysis you’ll have to do, which increases
significantly with each new task you introduce.




Create Tasks That Are Measurable
Pick tasks that are easy to determine success or completion.
This helps when you evaluate the data.



For Example

Bad Tasks Good Tasks

Search for a book. Please show me where would you look to see if
the library has a copy of the book titled The
Tipping Point by Malcolm Gladwell.




A few places that subjects can be found:
Coffee Shops, Bars, Malls, Libraries
Client or Colleague Contacts
Facebook/Twitter
Bulletin Boards
Classified Ads (e.g. Craigslist)
Mailing Lists
Amazon’s Mechanical Turk




Don’t Worry About Finding The Right Users
It’s good to test with the right people but it’s much more
important to test often and test early.




Use Incentives If You Need Their Time
It’s important to properly incentivize your volunteers and
participants. This makes recruitment much easier. Anything
under about 3 minutes can be free.




Some incentives you can use:

Cash
Gift Cards
Food and Drinks (Coffee, Pizza, Beer, Chocolate)
Movie Tickets




It’s Better to Have More Participants and
Fewer Tasks Per Session
In our experience, we’ve been much more effective in
recruiting for shorter tests and smaller incentives because
there’s less commitment involved.




You’ll Need 3-5 Participants Per Task
Most of the major findings will be revealed with the first 3
participants and the value of each new participant
significantly decreases over time.



3 On Finding a Location

Pick An Open Area
This works better than a closed room as it is a more
comfortable environment to reduce anxiety.




Ask beforehand
If you’re using a space like a mall, a school or coffee shop,
remember to reach out to the appropriate contacts to let them
know what you’re doing in advance. And remember to buy
coffee!




Their Office is Too Far?
Visiting in person would be great but consider doing a remote
user testing by doing a video call (e.g. Skype) to conduct your
test.



Facilitating A Test



Facilitating a Test



Facilitating a Test

1 Preparation and Setup



Facilitating a Test


2 Moderating the Test



Facilitating a Test



3 Finishing Up a Session



Preparation and Setup

Here’s an example of our set up.




Once Your Participant Arrives




• Welcome them and thank them for joining you




• Explain what you’re doing and how the test works




• Let them know you’re recording for research purposes and
ask them to sign a waiver




• Remind them you’re testing the site, not them - no pressure!




• Remind them you’re testing the site, not them - no pressure!
• Ask them to pretend they’re using the site “by themselves”
but encourage them to think aloud




Encourage Them To Think Aloud
This bears repeating. This is HOW you find out what they’re
thinking as they go through your site. Important!




Start With a Simple Task
This helps them build their confidence and puts them at ease.
You’ll want them to think “hey, this is easy - I can do this!”




Resist Giving Hints Or Ask Leading
Questions
Watch out for using “keywords” from your tasks as that may
not be a sign that your site is easy to use. If appropriate, try to
use synonyms for the answers they’re looking for.




Know When To Stop
If your subject is clearly having troubles, it’s okay to ask
“would you keep looking or give up”?




Good Questions for Moderating
“What are you currently thinking?”
“What would are you looking for?
“Is that what you were expecting?”
“What do you think would happen next?”



3 Finishing Up Your Test

Once You’re Done With Tasks




• Answer any questions they may have




• Start the post-test survey




• Thank them for their help and give them their incentive




• Record any key takeaways immediately




• Record any key takeaways immediately
• Prepare for the next test



Analyzing Test
Results



Analyzing Test Results

Debrief With Your Team Immediately




• Discuss key takeaways and note problems




• Determine what is easy “easy to fix”




• Schedule the changes in for the next iteration




• Schedule the changes in for the next iteration
• Prepare for the next test




Ignore “Kayak” Problems




Ignore “Kayak” Problems
Unless you have none, look for “low hanging fruits”. If you find
your participants are getting lost but end up finding the
correct answer with the “wrong path”, the problem is probably
safe to ignore.



How We Do Our
Testing



“5-Second Test”



“5-Second Test”

Goals
Helps us understand if our users “get it” (e.g. if this page is
trying to help them find and research stuff, can they figure
this out on their own?).



“5-Second Test”

Requirements
A sketch, wireframe, mock-up or complete design you can
show people. No interactivity is required.



“5-Second Test”

Method
Done in person. Approach strangers that “seem idle” and ask
them if they have 2 minutes? No incentives required. If they’re
on Facebook, they usually have to say yes!



“5-Second Test”

Procedure
Let them know you’re designing a site and you need their
opinion. Show them the design for 5 seconds and ask them:
1. What they saw
2. What they they think the site is trying to do
3. If they were to do something on the site, what would they
click?



Task Analysis



Task Analysis

Goals
Finds out if our visitors can use our site and elicit key design
problems.



Task Analysis

Requirements
A prototype or complete design you can have people play with.
You’ll need to go through the standard preparations (create
tasks, find locations and a small budget for incentives).



Task Analysis

Setup
A table, 2 computers and 3 team
members (one for facilitating, one for
observing and one for dealing with sign-
ups).

And a BIG sign that says

“FREE CHOCOLATE BARS”


Task Analysis

Record And Review Our Findings

We enter all our data into a spreadsheet to look for trends, key findings, and patterns.


Thank you!
Questions?
Email us at
john.chan@ubc.ca / jonathan.kift@ubc.ca

or on Twitter
@jtcchan / @daftkifty


How To Run Your Own Guerrilla Usability Testing

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (19)

Recently uploaded

Recently uploaded (20)

How To Run Your Own Guerrilla Usability Testing