Overview of usability testing methods for mobile devices and apps. Includes information on usability, explanation of challenges introduced by the mobile context, and practical tools and techniques.

1. Mobile
Usability
Evaluation
Garrett Stettler
UX Researcher
The Advisory Board Company - Crimson
April 8, 2015
Presented to INF 385T: Mobile IxD
U.T. Austin School of Information

2. MSIS 2010@gstett
AT&T Labs Human Factors, UX Research
Garrett Stettler
Advisory Board Co. UX Research & Strategy
Austin UXPA
meetup.com

3. Preview
● Definition of “usability”
● Evaluating usability
● Mobile challenges
● Tools and techniques
● Examples

4. “The user can do what he or she wants to do the way he
or she expects to be able to do it, without hindrance,
hesitation, or questions.” 1
Defining Usability
1. Rubin, J. & Chisnell, D. Handbook of Usability Testing, 2nd ed.
Efficiency
Effectiveness
Satisfaction
Discoverability
Learnability
Memorability
ISO 9241
More: 5 Dimensions of Usability

5. UCD and Experience Design
Useful
Usable
Accessible
Desirable
Emotional
Valuable
Adoptable
Social
etc...
Usability Testing
User-Centered Design
Experience Design

6. Usability Inspection
Heuristic Evaluation
Cognitive Walkthrough
Pluralistic Walkthrough / Design Review

7. Usability Testing
● “Early and Often”
● Iterate
● Keep your goals in mind when
choosing methods & metrics

8. Formative vs. Summative
Formative
Problem-finding during development
Summative
Measure against specific goals when
development is complete
In practice there isn’t always a clear distinction, and you should use
quantitative metrics in both types.

9. Usability Metrics
Objective and Subjective
Task Completion
Task Time
Task Satisfaction
Errors
Perceived Difficulty
Overall Satisfaction
Expected Difficulty
Clicks
Conversion Rate
UI Problems (Severity, Frequency)
More: Making usability metrics count

10. Participants
● Many ways to find participants
● Magic Number 5 (J. Nielsen)... not magic
● Use more frequent tests (with smaller
sample sizes) when possible.
More: A Brief History of the Magic Number 5

11. Usability Testing Methods
Lab-based Remote
Moderated
Remote
Unmoderated
Art: Todd Zazelenchuk, Elizabeth Boling

12. The Challenges of Mobile

13. Photo: Marcelo Druck (horizontal flip)
Context

14. Photo: Kathy Cassidy

15. Photo: David van der Mark

16. Photo: Wesley Fryer

17. Photo: Garrett Stettler

18. Photo: Ed Yourdon

19. Photo: Daniel Hoherd

20. Test in the lab or in the field?
Disagreement among practitioners and in literature.
Kjeldskov, Skov, et al (2004)
Kakkonen, Kallio, et al (2005)
Nielsen, Overgaard, et al (2006)
Duh, Tan, et al (2006)
Same UI problems identified in both
Field study costly
More/different UI issues found in field
Field study costly
“A lab experiment is recommended when the testing focus is on the user
interface and application-oriented usability related issues. However, the
results suggest that a field experiment is more suitable for investigating a
wider range of factors affecting the overall acceptability of the designed
mobile service.” Sun & May (2013)

21. Test in the lab or in the field?
Field testing is getting easier and cheaper, but
depending on your goals and circumstances a lab
study may still be the best choice.
Given the particulars of your situation, it’
s up to YOU to be pragmatic and
determine the best test setting.

22. Photo: Jeremy Keith
Devices

23. Use participant’s device?
Using the participant’s device adds to realism, but
introduces extra variables.
Participants may be uncomfortable if software must
be installed or modified.
If not using the participant’s device, you may need to
include a short training session or warm-up tasks.
It depends...

24. Photo: Ikhlasul Amal
Connectivity

25. Control connection?
Regardless of whether or not it makes sense to reduce
connectivity variables during your usability testing, your design
MUST consider connection conditions in the field and you
should test it under varying conditions.
Not all cellular or WiFi connections are created equal, especially
in congested settings.
Location isn’t determined just by GPS, e.g., what happens if the
WiFi radio is turned off?
It depends...

26. Photo: City University Interaction Lab
Recording

27. Why Record?
Memory Aid - events happen quickly!
Powerful way to present UI issues
to stakeholders
Ability for detailed analysis later

28. Recording Mobile
Recording desktop PCs and laptops
is easy.
Smartphones, not so much.
Wearables and IoT...

29. Recording Mobile
Luckily, phones and video cameras are much
better than they used to be.
Photo: State Library and Archives of Florida

30. Usability Testing Methods
Lab-based Remote
Moderated
Remote
Unmoderated
Art: Todd Zazelenchuk, Elizabeth Boling

31. Recording - Screen Capture
Easily view and record screen
Some can capture gestures and participant’s face
Pros
Cons
Most are not cross-platform or have other limitations
Some have recurring costs
Recording time can be limited by space
Many don’t capture gestures or participant’s face

32. Recording - Screen Capture
UX Recorder
Records on-screen activity, gestures, front camera
iOS only, websites only $60 or $2 per session
AirPlay, Reflector app
Mirror iOS to Mac or PC
No gesture or front camera support
Android apps (various)
Mirror to Mac or PC
No gesture or front camera support
Magitest (acquired by TryMyUI?)
iOS only, can use native apps with SDK $50
Lookback
Records on-screen activity, gestures, front camera
iOS, Android 5.0 (iOS requires SDK or jailbreak) Currently in beta

33. Recording - Cameras
Flexible, device-agnostic
Can be relatively inexpensive over time
Better view of participant’s gestures and movement
Pros
Cons
Need to focus and position camera
User’s fingers can obscure view
Need TWO cameras to record user’s face

34. Recording - Cameras
Document Cameras and Desk Mounts

35. Recording - Cameras
Problems with Document Cameras
Fixed (participant cannot hold and move the device)
Angle must usually remain constant
Focus is usually fixed

36. Recording - Cameras
Sleds - many inexpensive / DIY options

37. Recording - Cameras
Problems with Sleds
Can be heavy or otherwise unwieldy (e.g., unbalanced)
Camera can get in participant’s way
Can be difficult to rotate device (portrait <-> landscape)
Do not always accommodate different device sizes

38. Recording - Cameras
Free-standing / mounted cameras

39. Recording - Cameras
Problems with Mounted Cameras
User may block camera
Lighting, glare, and focus can be a problem

40. Multi-Camera Human Factors Lab

41. DIY “Guerilla” Mobile Testing Lab
● Two laptops (1 with webcam)
● Device sled with camera
● Screen capture software (Camtasia, Silverback,
Morae)
● WebEx

42. You don’t always need
to record.
You don’t always need
to go high-tech.

43. Test Early! Test Often!
Draw screens on Post-It notes
Rubber-band screen shots to an actual phone
Cardboard and adding-machine tape
POP app (Prototyping on Paper)

44. Usability Testing Methods
Lab-based Remote
Moderated
Remote
Unmoderated
Art: Todd Zazelenchuk, Elizabeth Boling

45. Remote Moderated Testing
GoToAssist
GoToMeeting (iPad websites only)
Still quite challenging for mobile
MailChimp
“hug” technique

46. Usability Testing Methods
Lab-based Remote
Moderated
Remote
Unmoderated
Art: Todd Zazelenchuk, Elizabeth Boling

47. Remote UNmoderated
TryMyUI
AppSee
UXCam
(to name a few)
Lots of options
Unmoderated remote testing can be a
time- and cost-effective way to test a
mobile app or website.

48. Recording - Wearable
Someday we may be able to conduct true in-context
testing without a rig like this:
Delikostidis, 2007

49. Recording - Wearable
GoPro:
(Google Glass project, anyone?)
Head-mounted camera

50. Recording - Eye Tracking

51. Examples
(removed)

52. Now you try!
Task-Based Usability Study
Find some participants (food & beer are good incentives)
Setup a “lab”. Whatever you can manage.
Create a set of tasks.
Intro to users. (Consent. Recording. “We’re not testing you.” Survey?)
Run through tasks (randomize/counterbalance). Think-aloud protocol.
Capture issues, errors, comments, metrics (e.g., success), path
After all tasks, survey? Satisfaction (System Usability Scale) and/or
other questions.

53. Summary
● Definition of “usability”
● Testing for usability
● Mobile challenges
● Tools and techniques
● Examples