This is our DIY Mobile Usability Testing presentation in its SXSW Interactive 2012 incarnation.

1. Thanks for coming!

2. Bernard, packet core engineer at NSN

3. Belén, interaction designer at Intel’s OTC

4. #SXdiymut

5. usability testing
a process that employs people as testing
participants who are representative of the
target audience to evaluate the degree to
which a product meets specific usability
criteria.
Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell

6. usability testing
a process that employs people as testing
participants who are representative of the
target audience to evaluate the degree to
which a product meets specific usability
criteria.
Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell

7. please, stand up

8. take out your
cellphone

9. sit down if you don’t
have a US cellphone
with a data plan

10. sit down if you don’t
like beer

11. sit down if you are
absolutely terrified by
the idea of being our
test subject

13. why recording?
memory aid
powerful communication tool

14. actions
reactions

15. dut = mut

16. dut = mut
where:
dut = desktop usability testing
mut = mobile usability testing

17. dut = mut + afec
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges

18. which phone?
which context?
which connection?

19. which phone?
which context?
which connection?

20. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

21. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

22. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

23. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

24. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

25. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

26. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

27. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

28. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

29. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

30. handset usability
affects test results

31. remember ...
test with participants’ own phones
if not possible, include training and
warm-up tasks

32. which phone?
which context?
which connection?

33. field vs. lab
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

34. field vs. lab
0 0
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

35. The results show that the added value of
conducting usability evaluations in the field is
very little and that recreating central aspects
of the use context in a laboratory setting
enables the identification of the same usability
problem list.
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware
Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

36. 0 1
The results show that the added value of
conducting usability evaluations in the field is
very little and that recreating central aspects
of the use context in a laboratory setting
enables the identification of the same usability
problem list.
Field Lab
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware
Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

37. according to our study there was no
difference in the number of problems that
occurred in the two test settings. Our
hypothesis that more problems would be
found in the field was not supported
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

38. 0 2
according to our study there was no
difference in the number of problems that
occurred in the two test settings. Our
hypothesis that more problems would be
found in the field was not supported
Field Lab
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

39. evaluations conducted in field settings can
reveal problems not otherwise identified in
laboratory evaluations
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

40. 1
evaluations conducted in field settings can
reveal problems not otherwise identified in
laboratory evaluations 2
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
Field Lab

41. The analyses of the comparison between
usability testing done in two different settings
revealed that there were many more types
and occurrences of usability problems found
in the field than in the laboratory. Those
problems discovered tend to be critical issues.
Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field Tests
H.B Duh, G. C. B. Tan,V. H. Chen, MobileHCI 2006

42. The analyses of the comparison between
2 2
usability testing done in two different settings
revealed that there were many more types
and occurrences of usability problems found
in the field than in the laboratory. Those
problems discovered tend to be critical issues.
Field Lab
Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field Tests
H.B Duh, G. C. B. Tan,V. H. Chen, MobileHCI 2006

43. field vs. lab
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

44. field vs. lab
DI SA GR EE
EXP ER TS
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

45. ... but they all agree
evaluations in the field (are) more
complex and time-consuming
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

46. ... but they all agree
testing in the field requires double the
time in comparison to the laboratory
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

47. ... but they all agree
field-based usability studies are not easy
to conduct. They are time consuming and
the added value is questionable.
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware
Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

48. testing in the lab is
better than no testing

49. remember ...
for most software, lab testing is fine
if you must do field testing
do it late
plan and run pilot tests
be prepared (like the Scouts)

50. which phone?
which context?
which connection?

51. remember ...
do not test over wi-fi
cover participants’ data costs

52. dut = mut + afec
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges

53. tsdohoeaygtrtwt
dut = (mut + afec)
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges

54. tsdohoeaygtrtwt
dut = (mut + afec)
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges
tsdohoeaygtrtwt = the small
detail of how on earth are you
going to record the whole thing

55. why recording?
memory aid
powerful communication tool

56. 4 approaches to the
small detail of how on earth
are you going to record the whole thing

57. 1. wearable equipment
.
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)

58. 1. wearable equipment
.
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)

59. 1. wearable equipment
.
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)

60. 1. wearable equipment
.
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)

61. Figure 10. Video recording with third-person view of participants and close-up view of PDA. Note that
1. wearable equipment
the camera focused on the device screen is turned 90 degrees to optimize use of the Picture-in-Picture
view.
cm—containing video and audio receivers, Picture-in-Picture unit, hard disk recorder, and battery.
e vulnerableas described above, but now weights only reliability, device. Asofwireless video attached to to get any me
capture to unexpected events,the mobile and size the cameras technology
make it impossible
Will Laboratory Test Results be Valid making it Contexts?, A. Kaikkonen, A. Kekäläinen, M. Cankar, T. Kallio, A.
2 kg and measures only 18x14x25 cm, in Mobile
us schedules. These risks shouldgenceR.T.cheap high-endthe M.video cameras 2011
Kankainen and A Field Laboratory for Evaluating in Situ, of Høegh, J. Kjeldskov, test. J. Stage,
from wireless B. Skov,
matures and becomes more widespread, an emer-
for longer periods of time. Powered by only one matching the professional standard of the wireless

62. allows testing in
the field

63. but ...
difficult and time-consuming to set up
intrusive, uncomfortable and heavy

64. 2. screen capture
Mobiola Screen Capture for Blackberry 4.2+ and Symbian S60 v3
http://www.shapeservices.com/en/products/details.php?product=capture&platform=none#

65. 2. screen capture
m o te
Re

66. 2. screen capture
It runs on both PC,
Mac, Linux and mobile environments such as Android, Symbian, iPhone OS and Windows Mobile.
m o te
Re

67. m o te
Re

68. m o te
Re

69. m o te
Re

70. provides high quality
screen recording

71. but participants won’t appreciate
you installing stuff on their phones

72. no application will support all platforms
http://www.shapeservices.com/en/products/details.php?product=capture&platform=none

73. no application will support all platforms
http://www.ovostudios.com/

74. Ovo Studios screen capture application for iOS
http://www.ovostudios.com/

75. fingers are not captured ...
Ovo Studios screen capture application for iOS
http://www.ovostudios.com/

76. .. and that is a big deal
Why AirPlay mirroring is the Biggest Thing to Happen to User Research in 2011
http://www.remoteresear.ch/airplay/

77. .. and that is a big deal
Obviously,
think aloud is critical because I cannot see how the participant is interacting
with his fingers on the touch screen.
I can see using the mirroring in a lab setting to get the signal from the iPad
to the observation room though you still won’t see the physical interaction
with the device, like you would with a device mounted camera (e.g. Noldus).
Why AirPlay mirroring is the Biggest Thing to Happen to User Research in 2011
http://www.remoteresear.ch/airplay/

78. .. and that is a big deal
iPad usability testing - our equipment
http://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/

79. .. and that is a big deal
Recently, we’ve questioned the value of capturing the device screen. It does
after all end up being a video of the screen changing but with no sense
of the participant interacting with it.
iPad usability testing - our equipment
http://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/

80. 3. document cameras
Handheld Usability (page 174), S. Weiss (2002)

81. Google
Towards the Perfect Infrastructure for Usability Testing on Mobile Devices
R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)

82. Nielsen Norman
Elmo TT-02RX Teachers Tool
document camera with autofocus
controlled remotely from a laptop
records screen and fingers
webcam records both video streams
participant’s face fed onto a laptop and
recorded with Morae
Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)

83. usertesting.com
m o te
http://www.usertesting.com/mobile/
Re

84. good recording quality
and easy to set up

85. but it’s not particularly cheap
Elmo TT-12 Document Camera (accessed March 4th 2012)
http://www.bhphotovideo.com/c/product/843500-REG/Elmo_1331_TT_12_Interactive_Document_Camera.html
http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html

86. ea p
IPEVO Point 2 View USB Document Cam (accessed March 4th 2012)
http://www.ipevo.com/prods/Point-2-View-USB-Camera
http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html C h

87. participants must
keep within the
camera range

88. phone must lay on
a desk or be hold
at a flat angle
http://www.flickr.com/photos/zabriskiepoint/2806511301/sizes/m/in/photostream/

89. 4. mounted devices

90. 4. mounted devices
ready-made

91. 4. mounted devices
ready-made
DIY

92. 4a. ready-made mounted devices
Mobile Device Camera by Tracksys
http://www.tracksys.co.uk/product-
details.php?id=9
Mobile Device Camera by Noldus
http://www.noldus.com/human-behavior-research/
accessories/mobile-device-camera-mdc#
Ovo Studios device camera
http://www.ovostudios.com/devicecamera.asp

93. 4b. DIY mounted devices
Little Springs Design
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/

94. 4b. DIY mounted devices
Little Springs Design Nick Bowmast http://www.bowmast.com/mob-device-cam/
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/
Google
Towards the Perfect Infrastructure
for Usability Testing on Mobile Usability Sciences
Devices, R. Schusteritsch, C.Y. Wei, http://www.usabilitysciences.com/services/lab-
M. LaRosa - Google (CHI 2007) based-usability-testing/mobile-usability-testing

95. 4b. DIY mounted devices are blooming!!
Little Springs Design Nick Bowmast http://www.bowmast.com/mob-device-cam/
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/
by curiouslee
Google
Towards the Perfect Infrastructure
for Usability Testing on Mobile Usability Sciences
Devices, R. Schusteritsch, C.Y. Wei, http://www.usabilitysciences.com/services/lab-
M. LaRosa - Google (CHI 2007) based-usability-testing/mobile-usability-testing

96. natural interaction with
the phone

97. but they are not cheap
http://www.godigi.com/products/DigiZoom-MDC.html

98. ea p
Mr Tappy, a kit for filming handheld devices (accessed March 10th 2012)
http://www.mrtappy.com C h

99. messy to build
http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/

100. and ...
if bulky they can prevent single-hand
use
if heavy they can become tiring during
long tests

101. easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

102. screen mounted mounted
wearable document
capture devices devices
equipment applications cameras ready-made DIY
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality

103. screen mounted mounted
wearable document
capture devices devices
equipment applications cameras ready-made DIY
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality

104. screen mounted mounted
wearable document
capture devices devices
equipment applications cameras ready-made DIY
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality

105. the spirit
http://www.wired.com/gadgetlab/2010/07/the-pencil-ipad-stand-smart-enough-to-impress-a-new-yorker/

106. the
ingredients

107. two meccano trunions (part no. A126)

108. two 5 and 6-hole meccano strips (part nos. 5 & 4)

109. one 11-hole meccano strip (part no. 2)

110. six meccano screws & nuts (part nos. 69 & 37h)

111. one 13-20mm jubilee clip

112. one HUE HD webcam

113. a second USB webcam

114. a USB male to female extension cable

115. blu tack (I think you call it mounting putty)

116. an allen key

117. a meccano wrench

118. a screwdriver

119. a Windows computer

120. screen recording software

122. a task
You just moved to Austin, to an old, big
house in Pemberton Heights.You love it,
but there is a problem: mice and rats. The
house is infested!
Go to www.austintexas.gov and find out
how to let the local authorities know
about the infestation.

123. how was that?

125. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

126. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

127. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

128. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

129. 15-model Meccano set 14.99
Hue HD webcam 34.95
Philips webcam 24.32
additional Meccano parts 4.01
blu tack 0.98
jubilee clips (x2) 1.99
USB cable 8.99
screwdriver 6.29
CamStudio 0.00
total (in GBP) 96.52

130. in USD: 151.25

131. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

132. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

138. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

139. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

140. weight: 125 grams

141. weight: 125 grams
an iPhone weighs 137 grams
an iPad weighs 680 grams
I weigh 55,000 grams
a blue whale weighs 136,400,000 grams

142. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

143. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

146. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

147. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

148. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

149. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

153. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

154. we expect much of our buildings: they
need to have firm foundations, solid
structures, pleasing aesthetics. We should
expect the same of emerging mobile
systems.
Mobile Interaction Design, M. Jones and G. Marsden (2005)

155. thanks!
belenbarrospena@gmail.com
b@runningwithbulls.com
@belenpena @bernardtyers
http://belenpena.posterous.com
http://www.runningwithbulls.com