This document provides techniques for conducting low-cost, fast usability testing for mobile devices. It discusses common problems with mobile usability testing including issues related to the physical devices, capturing user behavior, and testing emotional engagement. The document then recommends incremental research methods using a variety of low-fidelity and higher-fidelity techniques at different stages. Specific techniques discussed include paper prototyping, emulator studies, early app builds, and diary studies. The document emphasizes the importance of involving the product team and addressing issues quickly through methods like RITE testing.

1. DISCOUNT MOBILE USABILITY
TECHNIQUES
Getting cheap, fast, reliable product
feedback for mobile devices
Chris Nodder
Chris Nodder Consulting LLC

2. Share!
• Issues you face
• Tips and tricks
• Best/worst practices

3. 1987
1994
The issues with testing
mobile devices aren’t new
1999

4. Problems doing mobile usability
• Physical
– How to see/record what’s going on
– Many device types – which to test?
• Behavioral
– Triggering/capturing the important moments
– Observing the interaction without changing it
– Usability labs aren’t very true-to-life
• Emotional
– Many features/apps are discretionary
– Emotional engagement is hard to test for

5. Cheap, fast, reliable: pick two
• How can you get feedback to the product
team quickly and cheaply, and still feel
confident about it?
• Incremental research
– Each piece is cheap and fast
– Each piece answers specific questions that are
preventing the team from moving on
– In aggregate, the observations back each other
up and provide the reliability you need

6. Time in project cycle Location Measurement Technique
Customer Dev’t Lead User studies
User needs Ethnography
Effectiveness RITE testing
Efficiency Metrics
Utility Metrics
Delight Observation

7. Physical issues
flickr/waltarrrrr

9. Little Springs Design Lokion Interactive
Contour
Noldus

10. Recruiting, location issues
• Recruiting
– Require at least 3 months familiarity with current device
– Remind users to bring their phone & charger
– Find a way to reimburse them for data/minutes used if
not on all-you-can-eat plan
– Make sure their provider has reception at your location (if
lab-based)
– Do they need glasses to read phone screen? (bring them)
• Testing tips
– Room without direct overhead lights (glare)
– Be prepared for higher failure rates doing tasks on mobile
devices (need to reassure users)

11. Capturing behavior
• Low-fidelity for concept validation
– Paper prototyping
• Higher fidelity for interaction validation
– Flash, DHTML either on phone or on PC
– Emulator studies
– Competitor studies
– Early builds
• On-phone (user’s phone) as soon as possible
– Must be stable enough
– Gather metrics
– OTA updates if possible (roll out bug fixes)
– Diary studies via twitter and e-mail

13. Emotional element (delight)
• How do we measure engagement?
– Amount of use (and use over time) is a proxy
– Desirability toolkit (Product Reaction Cards)
– Analysis of adjectives used in forums/blog
postings

14. User experience over time
Orientation Incorporation Identification
Learn about the Use the product in Differentiate self from
product everyday life others
Anticipation
1 week 4 weeks
before after
What “good product” Ease of use Usefulness Social impact
means at each phase Stimulation Fits daily rituals Stimulation
Karapanos et al, CHI 2009

15. Accessible Creative Fast Meaningful Slow
Advanced Customizable Flexible Motivating Sophisticated
Annoying Cutting edge Fragile Not Secure Stable
Appealing Dated Fresh Not Valuable Sterile
Approachable Desirable Friendly Novel Stimulating
Attractive Difficult Frustrating Old Straight Forward
Boring Disconnected Fun Optimistic Stressful
Business-like Disruptive Gets in the way Ordinary Time-consuming
Busy Distracting Hard to Use Organized Time-Saving
Calm Dull Helpful Overbearing Too Technical
Clean Easy to use High quality Overwhelming Trustworthy
Clear Effective Impersonal Patronizing Unapproachable
Collaborative Efficient Impressive Personal Unattractive
Comfortable Effortless Incomprehensible Poor quality Uncontrollable
Compatible Empowering Inconsistent Powerful Unconventional
Compelling Energetic Ineffective Predictable Understandable
Complex Engaging Innovative Professional Undesirable
Comprehensive Entertaining Inspiring Relevant Unpredictable
Confident Enthusiastic Integrated Reliable Unrefined
Confusing Essential Intimidating Responsive Usable
Connected Exceptional Intuitive Rigid Useful
Consistent Exciting Inviting Satisfying Valuable
Controllable Expected Irrelevant Secure
Convenient Familiar Low Maintenance Simplistic
www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc

16. Extra time…

17. Involving the team
• List of questions team has
– Write down how each will be answered
– Write down answers as they come in
…this way team has a stake in finding answers
• RITE testing: team must attend
• Metrics: team must code into product
• Field visits: encourages user empathy

18. RITE (Rapid Iterative Testing and
Evaluation)
• Ship an improved interface as rapidly and cheaply as
possible
– More important to find and fix big issues than to find
every issue
• Fix issues as they are found in a study, run only
enough users to ensure the fix worked
– Development team must agree what users should be able
to achieve with the system (helps define issue severity)
– Development team must attend, agree issue fix, be
prepared to code fixes “on the fly”
– Usability Engineer must be experienced in domain and in
typical user issues to calculate level of severity

19. RITE - fixing issues
• Categories of issues
1. Issues with obvious cause and solution, quick fix
 Fix and test with next participant
2. Issues with obvious cause and solution, big fix
 Start fix now, test with fixed prototype when stable
3. Issues with no obvious cause (or solution)
 Keep collecting data, upgrade issue to 1 or 2
4. Issues caused by other factors (test
script, participant)
 Keep collecting data, learn from mistakes
… allows you to test fixes in the same study
… not an excuse for sloppy coding, UX work

20. RITE - Age of Empires II example
Vertical lines
are revisions
to test code
“Blip” = more
errors seen after
blocking issues
removed
Extra users
tested to see
fixes worked

21. Forrester mobile app model
• Handy structure for thinking about mobile user testing
• The five contexts which are amplified by mobile are:
location, locomotion, immediacy, intimacy and device.
– Location: People use apps in a wide variety of locations, which can be determined
through the use of GPS.
– Locomotion: Mobile users access their devices while on the move - walking, running
and even (unfortunately), driving. If a phone has an accelerometer, the app can
detect the motion, speed and direction of the device.
– Immediacy: Mobile users are not stationary - they need a mobile app to immediately
react to find a price, transfer funds or update their status, for example. They'll be
even more pleased when the app combines immediacy with location and locomotion
info to anticipate their needs.
– Intimacy: Mobile users identify with their device, but designing for intimacy means
you have to understand each person's relationship with their device. For example, a
bargain shopper may love getting in-store coupons via push notifications, but
another user may hate it.
– Device: Finally, developers should take into consideration the features specific to
the device, including the varying form factors, plus the device's touch, voice
recognition and image recognition capabilities.
Mobile app design best practices
Mike Gualtieri, Forrester Research

22. chris@nodder.com
@uxgrump
Taming evil interfaces
Creating heavenly experiences