In this webinar, Nate Colker, UX Researcher at UserZoom, talks about different research methods and tools for testing mobile websites and apps. As a case study, Nate goes over a setup and results of a recent mobile benchmark study that we conducted using UserZoom's brand new Mobile Remote Usability Testing solution.

1. Agenda
WEBINAR:
Mobile UX Research Methods and Tools
www.userzoom.com

2. Agenda for today
1. Introductions
2. Mobile UX Research Methods and Tools
3. Case Study: Fashion Industry Mobile/Tablet Benchmark Study
– Details & Design
– Study Results
4. Q&A
www.userzoom.com

3. Speakers
Alfonso de la Nuez
Nate Colker
Sneha Kanneganti
Co-Founder & Co-CEO
UX Researcher
UX Researcher
Moderator
Presenter
Twitter hashtag: #uzwebinar
www.userzoom.com

4. About UserZoom
 All-in-one Enterprise software solution that helps
Businesses cost-effectively test, measure and improve UX
over websites & mobile apps.
 Offer online or remote user research & testing
solutions, saving UXers time, money, effort,
and a lot of actionable insights
 UX Consultants since ‟01, SaaS since „09
 In Sunnyvale (CA) Manchester (UK), Munich
(DE) and Barcelona (Spain)
 90% renewal rate, 50% revenue growth rate
in the last 3 years
www.userzoom.com

5. Mobile UX Research Methods and Tools
www.userzoom.com

6. Mobile UX Research Methods
Mobile UX research method depends on the goals of the study.
• To validate and refine (testing).
• To gather user perspectives.
•
To create user profiles & personas.
• To understand behavior.
There are several methods to test mobile applications. It depends on
your:
• Goal
• Time
• Budget
www.userzoom.com

7. Mobile UX Research – Moderated Studies
www.userzoom.com

8. Mobile UX Research Methods - Moderated
Goal: To understand how users your mobile application, website or services.
Method: Usability Testing in Lab.
Users: 5-15
Project Timeline: 2+ Weeks (execution & analysis)
Cost: $15,000 - $30,000
Pros
Cons
Moderated Lab Usability Testing
• Probing by moderator
• Artificial environment
• Real-time viewing of interactions
• Comparatively more cost and time consuming
www.userzoom.com

9. Mobile UX Research Tools - Moderated
Mr Tappy

A kit for filming mobile phones and tablets from the
user‟s point of view.

Price: $295
(including shipment)
Pros
• Simple to set up
• Allows to test any mobile device you
want
Cons
Moderated Lab Testing Tool: Mr. Tappy
• Gets in the way of the natural use
of the phone
• You need your own webcam to do the
recordings
• The finger movements of the participant might
obscure the view
Review of the product:http://www.actualinsights.com/2012/mrtappy-mobile-usability-testing-rig/
www.userzoom.com

10. Mobile UX Research Tools - Moderated
DIY Document Camera by Objective Digital
You’ll need

GorillaPod Stand by Joby (www.joby.com)

Logitech Webcam Pro 9000

Recording software for Mac or PC
Moderated Lab Testing Tool
Source: http://blog.objectivedigital.com/2010/11/19/diy-document-camera-for-mobile-testing-record/
www.userzoom.com

11. Mobile UX Research Tools - Moderated
Mod 1000

A camera and a rig for recording the user experience with
mobile phones and tablet screens

Cost: $350
Pros
• Comes with its own webcam and
software
• Simple and lightweight design
Cons
• Gets in the way of the natural use
of the phone
• The finger movements of the participant
Moderated Lab Testing Tool: Mod 1000
might obscure the view
www.userzoom.com

12. Mobile UX Research Tools – Moderated
UX Recorder

An iOS app that records on-screen activity and gestures along with video
from the front facing camera (and audio)

Works on mobile websites only, doesn‟t work on mobile apps

Supported on devices running iOS 5.0 and up

Price: single session - $1.99; unlimited sessions - $59.99
Pros
• Records on-screen activity as well as
video from the front facing camera
• Records audio in addition to video
• Allows to export recorded video session
Cons
• Inability to record iOS applications (iOS
limitation)
• No analytics, just a video recording
Moderated Remote Testing Tool:
UX Recorder
to a desktop
www.userzoom.com

13. Mobile UX Research – Moderated Remote
www.userzoom.com

14. Mobile UX Research Methods – Remote Moderated
Goal: To understand how users your mobile application, website or services
Method: Remote Moderated Usability Testing
Users: 5 - 15
Project Timeline: 2+ Weeks (execution & analysis)
Cost: $5,000 - $7,000
Pros
Cons
• Not tied to a location.
• Difficult to observe body language.
• Distributed & diverse user pool.
Moderated Remote Usability Testing
• Technical difficulties.
www.userzoom.com

15. Mobile UX Research Methods – Remote Moderated
Delight.io

Enables you to record a screen of your iOS application and automatically
capture all gestures

Price: 30 hours for $300
Pros
• Allows to record audio in addition to video
• Records user‟s emotions using the
Cons
• You need to install a piece of code into your
app
iPhone‟s front camera
• Has a nice dashboard with all the
recorded sessions
www.userzoom.com

16. Mobile UX Research Methods – Remote Moderated
Reflector app

An app that will broadcast the screen of the iPhone or iPad
onto your computer

Price: Single user license - $12.99; 5 seat license - $54.99
Pros
• You can project your user testing
sessions for stakeholders to see
Cons
• You cannot see user‟s
Moderated Remote Testing Tool: reflector App
gestures
• Very easy to set up Has a nice
http://www.90percentofeverything.com/2013/01/14/combining-reflector-and-silverback-for-ios-usability-testing/
www.userzoom.com

17. Mobile UX Research Methods – Remote Moderated
Mac Hug

Idea developed by MailChimp

User‟s webcam and Skype to moderate the sessions

Sessions recorded at the researcher‟s end(silverback, WebEx
recorder, etc.)
Pros
• Cost Effective
• Doesn‟t require too much
Cons
• Users expressions & body
language lost
http://blog.mailchimp.com/remote-usability-testing-on-mobile-devices/
www.userzoom.com

18. Mobile UX Research – Unmoderated Remote
www.userzoom.com

19. Mobile UX Research Methods – Remote Unmoderated
Goal: To understand how users your mobile application, website or services
Method: Remote Unmoderated Usability Testing
Users: 200+
Project Timeline: 10 days (execution & analysis)
Cost: Variable ($1,000 - $6,000).
Pros
Cons
• Distributed & Diverse Users
• Probing difficult
• Time & cost effective
• Body language & immediacy of qualitative
• Enables to collect quant and qual
Unmoderated Remote Usability Testing
feedback lost
data
www.userzoom.com

20. Mobile UX Research Methods – Remote Unmoderated
Usabilla Mobile

Create visual surveys in iOS or Android browser

Allows to test screenshots of an app or a mobile site

Users can scroll through an image and tap anywhere
to leave a point and type up a note
Results
Charts and Heatmaps
Un-moderated Remote Testing
Tool: Usabilla Mobile
www.userzoom.com

21. Mobile UX Research Methods – Remote Unmoderated
UserTesting Mobile

Create series of tasks for users to perform

Send it out to users from UserTesting panel

Watch recorded videos of the user‟s screen

Downside – limited to their own panel
Results
• Video of user‟s screen and hand motions
• audio
Moderated Remote Testing Tool: UserTesting
Mobile
www.userzoom.com

22. Mobile UX Research Methods – Unmoderated Remote
UserZoom Mobile - New

Create a study with tasks for users to perform in their natural
environment

Test any mobile website and/or a hosted prototype on both
iOS and Android devices

Test your competitors‟ mobile site since no coding is
needed
Results
Quantitative and qualitative data
(success ratios, time on task, number of
clicks, survey responses, clickstreams,
NPS)
Coming Soon: Session Replay &
Unmoderated Remote Testing
Tool: UserZoom Mobile
Heatmaps
www.userzoom.com

23. Mobile UX Research Tools: Opinion Research
UserZoom Mobile Surveys

Conduct various types of research projects, including: satisfaction
surveys, exit surveys and persona definition

Works on iOS and Android devices
www.userzoom.com

24. Fashion Industry Mobile Website UX Benchmark Study (Remote Unmoderated)
www.userzoom.com

25. Study Details
How do fashion retail mobile websites perform on routine purchase related tasks?
4 Brands
3 Tasks
1. Choose a black dress and go through the
purchase process as if to buy it.
2. Look for this* necklace that goes with
your dress and continue to checkout.
Study Design
•
•
•
•
•
Tested Between April 15-21, 2013
Between Subjects Design
200 Users/50 users per brand
3 Tasks and Survey Questions
Recruitment Criteria
 Female
 between 25-45
3. Look for information on how to return an
item ONLINE and whether there is an
associated cost.
*Product name provided
 Regularly make purchases on their
tablet
• Used a panel Vendor (SSI).
www.userzoom.com

26. Mobile Phone/Tablet Optimization
Brands
Tablet
Mobile site
iPad
Android
www.userzoom.com

27. Mobile Phone/Tablet Optimization
www.userzoom.com

28. Study Through Participant‟s Perspective
www.userzoom.com

29. Study Results
www.userzoom.com

30. Results – Top 4 Findings
1. Based on user feedback on 3 tasks, White House Black Market
fares the best in searching and finding products (a black dress &
necklace). It also has the highest Net Promoter Score of +26.
2. Users found it difficult to find return policy information on these
product pages.
3. Users do not like registering and disclosing personal information
before making a purchase.
4. Material, length, weight and how it looks on a person are important
product information users seek while making purchasing decisions.
www.userzoom.com

31. Task -1
www.userzoom.com

32. Results – Task 1
Choose a black dress and go through the purchase process as if to buy it
Reasons for failure
Success
“I was being forced to enter a lot of
personal information before I got to
the payment details page.”
100
90
80
70
60
“The web page would not load.”
50
40
30
20
10
0
52
42
42
53
Kenneth Cole
Guess
Bebe
WHBM
Success rate in percentage
“It takes a very long time to enter
payment information on a tablet
without a full keyboard. It was also
difficult to find a black dress I liked in
a size 10/m.”
“It‟s probably easier on a
computer, but on a mobile device it
was very slow, I had to hit the buttons
several times, and my keyboard would
pop in and out.”
www.userzoom.com

33. Results – Task 1
Choose a black dress and go through the purchase process as if to buy it.
52% Success
Clear Payment Methods
Mentioned.
42% Success
Too much to type on a
touch pad.
www.userzoom.com

34. Results – Task 1
Choose a black dress and go through the purchase process as if to buy it
Efficiency
Satisfaction
7
5
6
4
5
3
4
3
2
2
1
1
6
2.76
2.9
3.28
3.33
Kenneth Cole
Guess
Bebe
WHBM
0
Average Time Taken in Minutes
5.7
Kenneth Cole
Guess
6.5
6.2
0
Bebe
WHBM
Satisfaction on a Scale of 1-7
Guess: “I had to give
registration info to check
out. I don‟t always like
doing that.”
www.userzoom.com

35. Task - 2
www.userzoom.com

36. Results – Task 2
Look for a necklace that goes with your dress and continue to checkout.
Success
Reasons for failure
“It says that the necklace is
available in "one size" but didn‟t
state the actual length.”
“There was a lack of
information on the materials
and I would like a picture of the
item on a neck to estimate
size.”
“Not enough information about
what the products were made
out of. Wax??”
Success rate in percentage
“I scrolled through the 8 pages
but couldn‟t find it. I had to
search for it.”
www.userzoom.com

37. Results – Task 2
Look for a necklace that goes with your dress and continue to checkout.
93% Success
Product materials clearly
listed.
51% Success
Vague product description.
www.userzoom.com

38. Results – Task 2
Look for a necklace that goes with your dress and continue to checkout.
Efficiency
Satisfaction
7
5
6
4
5
3
4
3
2
2
1
0
1
1.96
2.26
2.46
Kenneth Cole
Guess
Bebe
Average Time Taken in Minutes
2.56
WHBM
0
6.4
5.4
6.3
6.2
Kenneth Cole
Guess
Bebe
WHBM
Satisfaction on a Scale of 1-7
WHBM: “Love it. Liked the
suggestions, reviews and
guide to size”
Kenneth Cole: “I like how there are
other pieces of jewelry presented as
suggestions as to what could go with
this particular necklace.”
www.userzoom.com

39. Task - 3
www.userzoom.com

40. Results – Task 3
Look for information on how to return an item ONLINE and whether there is an associated cost.
Reasons for failure
Success
“Too much information!”
“I had to go to the full site
to find the info.”
Success rate in percentage
“Not very easy to find and
probably wouldn‟t have even
tried that hard had it not
been for this survey.”
“I couldn't FIND any information
about online returns, only in
store and by mail.”
www.userzoom.com

41. Results – Task 3
Look for information on how to return an item ONLINE and whether there is an associated cost.
82% Success
Clear online purchase
return information.
40% Success
Too much information to
parse through.
www.userzoom.com

42. Results – Task 3
Look for information on how to return an item ONLINE and whether there is an associated cost.
Satisfaction
Efficiency
7
5
6
4
5
3
4
3
2
2
1
1
0
2.1
2.16
2.4
1.53
Kenneth Cole
Guess
Bebe
WHBM
Average Time Taken in Minutes
5.6
4.5
6.3
6.4
Kenneth Cole
Guess
Bebe
WHBM
0
Satisfaction on a Scale of 1-7
Kenneth Cole:
“It was not clear whether the
cost for shipping returns was
free”
Guess:
“I Had To Go To Check Out To
Get The Return Policy. I Need
It More Accessible.”
www.userzoom.com

43. Net Promoter Score - NPS
26
22
22
2
• While all brands have a positive NPS, White House Black Market has the
highest with 26. (NPS is the measure of how likely a user is to recommend
a site to friends and family.)
Net Promoter Score, Net Promoter and NPS are registered trademarks of Bain & Company, Satmetrix Systems and Fred
Reichheld. © 1996-2013. Bain & Company
www.userzoom.com

44. Mobile Phone/Tablet Optimization
Study simultaneously conducted in Spain & U.K.
www.userzoom.com

45. Take-aways & Recommendations
There are many mobile usability research methods & tools in the market today. There is no
excuse not to conduct user research anymore.
•
Based on user feedback on 3 tasks, White House Black Market fares the best in
searching and finding products (a black dress & necklace). It also has the highest Net
Promoter Score of +26.
•
Crisp product information, precise search results and easy to use filters are aspects
users care about while making purchases on websites.
•
Research what your customers care about.
www.userzoom.com

46. Questions?
Follow us for updates on future webinars!
Twitter:
@userzoom
Linkedin:
http://www.linkedin.com/company/userzoom
Facebook:
https://www.facebook.com/userzoom
www.userzoom.com