1. Testing
(AKA Validation)
“How do I know my prototype is any good?”
Rick Boardman
@uxrick

2. Cast
Product/Biz UX
User
Eng
Thanks to Braden Kowitz for smiley graphics

3. Think Usability?
Intro to Intro to
(Speaker/ How to Lean UX Task
generative Personas Intersect What are methods? Analysis?
Lecture) interviewing Distinguish
& POVs
Brainstorm
you building
btw needs w/ Biz & Analytics
Distro in next week
& insights Agile?
& next 3 mo?
User Customer/ Solution
Prototype Story/ Test
How do we
Needs Problem Uses Features
Empathy Ideation Release
Definition
test our
Make
hypotheses?
(Method/ Practice with
Make persona
of target user(s) Framing Mock
Wireframes
Activity) partner Brainstorm dashboard
POV madlib
Practice Ecosystem & Storyboard
Sketches
synthesizing Distribution Flows
empathy map map How to test
conversion
Refine survey Compare personas Prioritize How to events?
questions created by diff based on tech/ test? Test with real
team members biz/distro users &
Check Product
Design Guild
What are How to
(Method/ riskiest How to
test?
Activity) Show partner test?
hypotheses?
map
Interview
real users?
Thurs Fri Sat

4. http://www.all-tribes.com/eng/catalog/rtfm-vol2-girls-mastic-p-51.html

5. How do I get MTV
in MS-Office?
http://www.all-tribes.com/eng/catalog/rtfm-vol2-girls-mastic-p-51.html

6. How do I get RTFM!
CNN in MS-Office?
Read The Fantastic(?) Manual
http://www.all-tribes.com/eng/catalog/rtfm-vol2-girls-mastic-p-51.html

7. Pic: http://www.flickr.com/photos/victoriapeckham/164175205

8. How do we
test our
hypotheses?
Pic: http://www.flickr.com/photos/victoriapeckham/164175205

9. How do we TTYFU!
test our
hypotheses?
Talk To Your Fantastic(?) Users
Pic: http://www.flickr.com/photos/victoriapeckham/164175205

10. Which method?
Demo at “Launch and
cognitive walkthrough see”
SXSW Card sorting
In-person Analytics
expert review Usability testing Friends n’ Family
A/B testing Dogfooding Funnel analysis
remote interviews Contextual interviews
Beta test Show to
click/5sec tests
Focus groups Surveys David Pogue

11. Which method?
dir ty“Launch and
Demo at
SXSW
H ere’s a
cognitive walkthrough
Card sorting
see”
In-person
secUsability
expert review
...
ret .testing Friends n’ Family
Analytics
s shhhh Funnel analysis
sssDogfooding
A/B testing
remote interviews Contextual interviews
Beta test Show to
click/5sec tests
Focus groups Surveys David Pogue

12. Which method?
dir ty“Launch and
Demo at
H ere’s a
cognitive walkthrough
TheeUsability.testingsFriends kFamily
yeta.l.l. ucn’ *
SXSW see”
Card sorting
Analytics
s cr
In-person
expert review
s shhhh Funnel analysis
sssDogfooding
A/B testing
remote interviews Contextual interviews
Beta test Show to
click/5sec tests
Focus groups Surveys David Pogue
*More diplomatically: they have strengths and weaknesses

13. Which method?
dir ty“Launch and
Demo at
H ere’s a
cognitive walkthrough
TheeUsability.testingsFriends kFamily
yeta.l.l. ucn’ *
SXSW see”
Card sorting
Analytics
s cr
In-person
expert review
s shhhh Funnel analysis
sssDogfooding
A/B testing
remote interviews Contextual interviews
Beta test Show to
Focus groups Surveys Especiallytests one
click/5sec this
David Pogue
*More diplomatically: they have strengths and weaknesses

14. Triangulate
multiple methods
Quantitative
Qualitative

15. Triangulate
multiple methods
Quantitative
Analytics
Lea
n UX
Qualitative !
Usability tests

16. Analytics

17. Oooo
AAARRRR, got a
dashboard?
Link: Metrics
for Pirates

18. Example Conversion Dashboard
(note: *not* actuals… your mileage may vary)
Stage Conversion Status Conv. Est. Value
% (*not* cost)
Acquisition Visitors -> Site/Widget/Landing Page 60% $.05
(2+ pages, 10+ sec, 1+ clicks = don’t abandon)
Activation “Happy” 1st Visit; Usage/Signup 15% $.25
(clicks/time/pages, email/profile reg, feature usage)
Retention Users Come Back; Multiple Visits 5% $1
(1-3x visits/mo; email/feed open rate / CTR)
Referral Users Refer Others 1% $5
(cust sat >=8; viral K factor > 1; )
Revenue Users Pay / Generate $$$ 2% $50
(first txn, break-even, target profitability)
Link: Metrics for Pirates

19. Usability tests

20. Usability testing
• Watch your product in action with a
target user
• Follow some tricks to make as
objective as possible
nterview bonus: opportunity for
Live i
more gene rative research!

21. Usability Howto
1. Identify research goal -> Scenario
2. Recruit
3. Prepare interview outline, pilot test!
4. Test!
- Intro (welcome/confidential/reassure/thinkaloud)
- Record / take notes
- Debrief
5. Review learnings and GET CODING

22. Awesome
2 hour read
http://www.sensible.com/rocketsurgery/index.html

23. Lean questions
1. What gear do I need?
2. How many users?
3.Which users?

24. observation
test room room
1-way mirror
eyetracking
UI
usb mic
Eng
Moderator User
PM
**Amazing
revelatio24
n**

25. Eng (notetaker)
Moderator User
**Amazing
revelatio
25
n**

26. Lean questions
1. What gear do I need?
2. How many users?
3.Which users?

27. Golden Rule

28. Golden Rule
• Participant should do 90% of the talking
Shut up

29. Interview Practice

30. Interview Practice
I’m super
nervous ... am I
being tested?

31. Interview Practice
What do I do
next?
Should I sign up?
Is this OK?

32. Interview Practice
I HATE IT.
THIS IS &^**(}ED
UP, MAN.

33. Interview Practice
I LOVE THIS. ITS THE
BEST THING EVER

34. Interview Practice
<click>
<click> <done>
<silence>

35. Interview Practice
Weather
wafflewaffle waffle
Superbowl waffle
waffle iPhone

36. Things to look for
• Obvious usability issues
• Opportunities
• Bugs
• Delights!

37. GO FORTH & TTYFU!
In 30 mins - pilot test!
1. come up with scenario and interview
framework
2. Interview your neighbours
3. Get interviewed

38. Is anyone getting a beer?
1. CrowdRally: paper prototyping works! Navigation behaviour. Need for a photo
prompt. No
2. Spoondate: product value prop not clear from homepage, signup issues, hesitant
to use FB (privacy issues), some missed expectations. No
3. Networked Blogs: nameless product (mobile news reader), follow worked OK,
need option to hide, game mechanics went well,. No
4. InternMatch: testing evaluation flow, wanted resume up-front and in-line. No
5. Yongopal: companion product/mobile app for students to take notes, button/
section labels, provide better feedback to user, feature ideas. No