8. usability testing
a process that employs people as testing
participants who are representative of the
target audience to evaluate the degree to
which a product meets speciïŹc usability
criteria.
Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell
23. ïŹeld vs. lab
Itâs Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
24. experts disagree
according to our study there was no
difference in the number of problems that
occurred in the two test settings. Our
hypothesis that more problems would be
found in the ïŹeld was not supported
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. KekÀlÀinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
25. experts disagree
evaluations conducted in ïŹeld settings can
reveal problems not otherwise identiïŹed
in laboratory evaluations
Itâs Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
26. but they all agree
evaluations in the ïŹeld (are) more
complex and time-consuming
Itâs Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
testing in the ïŹeld requires double the
time in comparison to the laboratory
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. KekÀlÀinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
28. remember...
for most software, lab testing is ïŹne
if you must do ïŹeld testing...
do it late
plan and run pilot tests
be prepared (like the Scouts)
31. state of coverage: UK
mobile coverage for 2G is extensive at
98%. The picture for 3G coverage is
different. Figures are high overall (87%),
but vary markedly in the nations, with
only 43% availability in Northern Ireland
Mobile Evolution - Ofcomâs mobile sector assessment - 17 December 2009 (p35)
http://stakeholders.ofcom.org.uk/binaries/consultations/msa/statement/MSA_statement.pdf
32. Map of UK 3G geographic coverage by number of operators
Ofcom / GSM Association / Europa Technologies, Q1 2009
35. tsdohoeaygtrtwt
dut = (mut + afec)
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges
tsdohoeaygtrtwt = the small
detail of how on earth are you
going to record the whole thing
37. 4 approaches to the
small detail of how on earth
are you going to record the whole thing
38. 1. wearable equipment
Methods and techniques for ïŹeld-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)
47. Google
Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)
48. Nielsen Norman
Elmo TT-02RX Teachers Tool
Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)
49. Nielsen Norman
document camera with autofocus
controlled remotely from a laptop
records screen and ïŹngers
webcam records both video streams
participantâs face fed onto a laptop and
recorded with Morae
Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)
54. phone must lay
on a desk or be
hold at a ïŹat angle
http://www.ïŹickr.com/photos/zabriskiepoint/2806511301/sizes/m/in/photostream/
55. 4. mounted devices (a) ready made
Mobile Device Camera by Tracksys
http://www.tracksys.co.uk/product-
details.php?id=9
Mobile Device Camera by Noldus
http://www.noldus.com/human-behavior-
research/accessories/mobile-device-camera-
mdc#
56. 4. mounted devices (b) DIY
Nick Bowmast http://www.bowmast.com/mob-device-cam/
Little Springs Design
http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-
for-mobile-devices-2/
57. 4. mounted devices (b) DIY
Google
Towards the Perfect Infrastructure
Usability Sciences
for Usability Testing on Mobile
http://www.usabilitysciences.com/services/lab-
Devices, R. Schusteritsch, C.Y. Wei,
based-usability-testing/mobile-usability-testing
M. LaRosa - Google (CHI 2007)
61. and...
if bulky they can prevent single-
hand use
if heavy they can become tiring
during long tests
62. easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
63. 1 2 3 4a 4b
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
64. 1 2 3 4a 4b
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
65. 1 2 3 4a 4b
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
85. a fake (but realistic)
impromptu usability test
86. task 1
Your friend from Dublin is ïŹying to
London tonight. He is a cheap dude, so he
must be ïŹying Ryanair. He hasnât told you
which airport he is ïŹying to, but you know
he should be landing at 12:45 in the
afternoon. Go to www.ryanair.com and
ïŹnd out if his ïŹight is on time.
87. task 2
Your friend from Dublin is ïŹying to
London tonight. He is a posh dude, so he
must be ïŹying BA. He hasnât told you
which airport he is ïŹying to, but you know
he should be landing at 11:10 in the
morning. Go to www.ba.com and ïŹnd out
if his ïŹight is on time.
90. b&bâs
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
91. b&bâs
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
92. 15-model meccano set 18.33
Hue HD webcam 28.94
Philips webcam 52.08
additional meccano parts 4.90
blu tack 0.98
jubilee clips (x2) 1.55
USB cable 8.99
screwdriver 5.99
CamStudio 0.00
total (in GBP) 121.76
94. b&bâs
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
95.
96.
97.
98.
99.
100. b&bâs
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
101. weight: 125 grams
an iPhone weighs 137 grams
an iPad weighs 680 grams
I weigh 55,000 grams
a blue whale weighs 136,400,000 grams
102. b&bâs
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
106. Nokia N900
slider
touchscreen
with qwerty
keyboard
107. b&bâs
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
108.
109.
110. b&bâs
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
111. b&bâs
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
112.
113. b&bâs
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & ïŹngers
runs tests with participantsâ phones
gives enough video quality
114. a call to arms
an Irish unit of the International Brigade http://www.communistpartyoïŹreland.ie/spainn.jpg