3. 340 TESTS IN LAST 18 MONTHS
TEST 2-3 DAYS A WEEK
EVERYTHING USER-FACING GETS TESTED
ON PACE TO DO 400+ SESSIONS THIS YEAR
SUPPORTING 6-8 PROJECT TEAMS
WILL COST ABOUT AS MUCH AS DOING 4
OUTSOURCED USABILITY PROJECTS
11. DIY EXPERIMENT
â˘âŻ RECRUITED OFF OF CRAIGSLIST
â˘âŻ 1 PRODUCT PERSON ENGAGED USERS IN A FEW DIRECTED TASKS
THEN SYNTHESIZED FINDINGS INTO A PRESENTATION
VERDICT
â˘âŻ LOW COST
â˘âŻ LEARNED INTERESTING FINDINGS
â˘âŻ HARD TO REALLY CONVEY USER BEHAVIOR TO REST OF TEAM
â˘âŻ WE DID NOT MAKE INTO ONGOING ROUTINE
13. LOOKED AT FORMAL USABILITY TESTS
â˘âŻ RECEIVED PROPOSALS FROM SEVERAL USABILITY TESTING FIRMS
â˘âŻ $32K - $53K FOR 2 DAYS (16 TESTS)
â˘âŻ INVOLVED USE OF TESTING FACILITY
â˘âŻ PROPOSED DELIVERABLE WAS A REPORT/RECOMMENDATION
VERDICT
â˘âŻ HELD OFF
15. âIâLL LET YOU INTO THE BETA PROGRAM IF YOU
AGREE TO LET MY TEAM WATCH YOU SET IT UPâ
ââŠCHANCEâŠCONVERSATIONâŠTHATâŠSETâŠFOUNDATIONâŠFORâŠCURRENTâŠPROGRAMâŠ
19. ACCIDENTAL EXPERIMENT
â˘âŻ USER GOT TO JUMP TO FRONT OF LINE IN BETA PROGRAM IN RETURN
FOR PARTICIPATION
â˘âŻ HELD TEST IN CONFERENCE ROOM WITH PROJECT TEAM IN THE ROOM
â˘âŻ PROJECTED USERâS SCREEN ON WALL
â˘âŻ INFORMALLY MODERATED USER TO VERBALIZE THOUGHT PROCESS
â˘âŻ TEAM DISCUSSION AFTER THE SESSION
VERDICT
â˘âŻ LOW COST
â˘âŻ USABILITY ISSUES WERE OBVIOUS TO ENTIRE TEAM
â˘âŻ REALIZED THIS APPROACH COULD BE POWERFUL
25. ~ FIFTH ITERATION ~
COMMITTED TO USABILITY TESTING
& MADE IT ROUTINE
26. â˘âŻ CTO MANDATED THAT ALL USER-FACING PRODUCT BE TESTED
AS PART OF DEVELOPMENT PROCESS
â˘âŻ PURCHASED SOFTWARE TO HELP SHARE/RECORD SESSIONS
â˘âŻ TEAMS VIEWED ALL SESSIONS LIVE
â˘âŻ IMMEDIATELY FOLLOWED UP WITH A DISCUSSION
â˘âŻ NO REPORTS
â˘âŻ DID OUR OWN RECRUITING OUT OF THE COMMUNITY TO TEST
MEMBER EXPERIENCE
â˘âŻ USED OUTSIDE RECRUITERS TO TEST EXPERIENCE FOR
PEOPLE UNFAMILIAR WITH MEETUP
27. PURCHASED SOFTWARE TO HELP
â˘âŻ USE MORAE ($1500)
â˘âŻ PC BASED
â˘âŻ ALLOWED US TO BROADCAST PICTURE
IN PICTURE
â˘âŻ ALLOWED FOR SESSIONS TO BE
RECORDED AND LIVE BROADCASTED
TO OTHER ROOM
â˘âŻ ALLOWED US TO MAKE HIGHLIGHT
VIDEOS
â˘âŻ LOTS OF FUNCTIONALITY (BUT WE
ONLY USE A FRACTION OF IT)
28. ProjectedâŠuserâsâŠscreenâŠonâŠwall⊠DESKâŠ
computerâŠ
MOD⊠USERâŠ
deve
loper⊠deve
loperâŠ
deve Table⊠deve SMALLâŠCONFERENCEâŠ
loper⊠loper⊠ROOMâŠ
deve
deve
loperâŠ
loperâŠ
MAINâŠCONFERENCEâŠ
ROOMâŠ
30. EXPERIMENT WITH COMMANDO TESTING
â˘âŻ TAKE A LAPTOP WITH SILVERBACK SOFTWARE TO A COFFEE SHOP
â˘âŻ GET PERMISSION FROM THE MANAGER
â˘âŻ OFFER FANCY CUP OF COFFEE FOR A TEN MINUTE TEST
VERDICT
â˘âŻ CHEAP
â˘âŻ DO NOT NEED TO PLAN AHEAD
â˘âŻ FEEL LIKE A BAD ASS
âŚUNTIL YOU GET TO THE COFFEE SHOP
â˘âŻ WASTE A LOT OF TIME WRANGLING PEOPLE
31. EXPERIMENT WITH SILVERBACK SOFTWARE
â˘âŻ $50 (FREE 30-DAY TRIAL)
â˘âŻ MAC ONLY
â˘âŻ LIGHTWEIGHT
â˘âŻ EDIT VIDEO USING IMOVIE
â˘âŻ WE CURRENTLY USE IT FOR COMMANDO
TESTING IN THE FIELD AND IPHONE TESTING
VERDICT
â˘âŻ HIGHLY RECOMMEND AS A STARTING POINT,
BUT ONLY TESTS ON MAC OS
32. OTHER EXPERIMENTS
â˘âŻ LOW COST RECRUITING
(FROM THE BUILDING)
â˘âŻ SCREENCASTING WITH CHAT
(USING DIMDIM)
â˘âŻ TIVO-IZING / TIME SHIFTING
TESTS TO SAVE DEVELOPERâS
TIME
34. 9 LEAN USABILITY PRINCIPLES
â˘âŻ LEARN TO LOVE ERROR
â˘âŻ LOOK FOR BOULDERS IN THE ROAD
â˘âŻ SUBSTITUTE FREQUENCY FOR PRECISION
â˘âŻ STRIP OUT COSTS WHEREVER POSSIBLE
â˘âŻ THINK MINIMUM VIABLE PRODUCT PROCESS
â˘âŻ IN RECRUITING, BE PREPARED TO TRADE MONEY FOR TIME
â˘âŻ BASIC MODERATION TECHNIQUES CAN GO A LONG WAY
â˘âŻ EXPOSE TEAM TO USERS
â˘âŻ TAKE NOTES & HAVE DISCUSSIONS (DONâT WRITE REPORTS)
35. Recruiting
DO DONâT
1.⯠Practice with your friends and family 1.⯠Rely on the kindness of friends to
2.⯠Have 1 person own usability testing volunteer every time you need testing
for the company 2.⯠Use the same tester(s) for different
3.⯠Create a screener listing minimum iterations of the same project
user requirements â completed over 3.⯠Explain too much about the purpose
phone or online (guidelines available of the test when recruiting
online) 4.⯠Bother the participant with too many
4.⯠Consider using a vetted recruiter if calls or emails
you can afford it or for specialized 5.⯠Be afraid to turn them down if they
audiences seem âweirdâ
5.⯠Allow a week to schedule in-house 6.⯠Forget to thank them! (They may
sessions become evangelists)
6.⯠Follow up with testers the day before
7.⯠Offer incentive
36. Moderating
DO DONâT
1.⯠Practice the test ahead of time 1.⯠Start the test before they sign a
and write down questions consent/release form (example
2.⯠Create a scenario to test an available online)
experience, not just a feature 2.⯠Forget to remind participant that
3.⯠Be aware of your assumptions you are testing the product, not
beforehand as not to project them the participant
on the participant
3.⯠Answer their questions or explain
4.⯠Spend a few minutes getting to
know the participant things about the project to help
5.⯠Ask general, open-ended them along
questions first and then probe for 4.⯠Use terms from your website or
details industry jargon when asking
6.⯠Let participant explain things in questions
their own words and avoid 5.⯠Accept just âyesâ or ânoâ for an
correcting them answer
37. A few great moderating questions
When first looking at project
â˘âŻ What are you looking at here?
â˘âŻ What can you do?
As test progresses
â˘âŻ Show me how you would ____________
â˘âŻ What do you expect will happen when you _____?
Wrapping up
â˘âŻ What did you find confusing?
â˘âŻ How would you describe what you did here today?
â˘âŻ What, if anything you saw today, could you imagine using at home? How?
38. Sharing
DO DONâT
1.⯠Record the sessions 1.⯠Assume everyone saw the
2.⯠Take notes for immediate same things you did in a
test
discussion afterward
2.⯠Interpret user reactions if
3.⯠Debrief with project teams you donât have the data to
the same day support it
4.⯠Provide notes and video for 3.⯠React too strongly to any
those who could not be single test
there 4.⯠Just cherry-pick your pet
5.⯠Protect the privacy and issues that might have come
dignity of the participant up
39. Testing day checklist (office test)
1.⯠Set up item to be tested
2.⯠Start recording and sharing processes
3.⯠Greet participant and bring them to the private testing room
4.⯠Collect release form, explain test and build rapport
5.⯠Conduct the test, reminding them to think out loud and try to do things as
if they were using their own computer at home (or work)
6.⯠Have a channel of communication between tester and remote observers
(chat or text messaging) for emerging questions
7.⯠If testing multiple projects/scenarios, introduce each one separately, one
after the other
8.⯠Reserve time at the end to answer questions and explain your product, if
they ask
9.⯠Thank the participants and provide incentive
10.⯠Stop recording and sharing processes
11.⯠Meet with team to discuss what you saw
12.⯠Adjust test if necessary for next session
40. Do it!
â˘âŻ Anyone can run a usability test with the right
preparation and attitude
â˘âŻ There is lots of advice available online
â˘âŻ Learn from mistakes and evolve your methods
â˘âŻ Ask people who have done it before to help you
out
â˘âŻ Team up & test each other