2. Intros
Alfonso de la Nuez Kim Oslob
Co-Founder & Co-CEO Sr. Client Services Director
Twitter Hashtag #uzwebinar
3. Overview
What we mean by UX Benchmarking and why it’s important?
How to conduct UX Benchmarking studies, using remote
unmoderated usability testing (aka task-based surveys)?
What are some of the UX metrics you should look for?
How to measure the quality of a website’s user experience across
various competitors?
A case study: American Airlines vs Delta Airlines Websites
4. What is Benchmarking?
Definition of Benchmarking:
The process of comparing the business
processes and performance metrics
including cost, cycle time, productivity,
or quality to another that is widely
considered to be an industry standard
benchmark or best practice.
It provides a snapshot of the
performance of your business and helps
you understand where you are in
relation to a particular standard.
5. What is UX?
Definition of UX:
It’s about how a person feels about using a
system. It highlights the experiential,
affective, meaningful and valuable aspects
of HCI and product ownership, but it also
covers a person’s perceptions of the
practical aspects such as utility, ease of
use and efficiency of the system.
User experience is subjective in nature, as
it is about an individual’s feelings and
thoughts about the system. User
experience is dynamic, because it changes
over time as the circumstances change.
8. What is UX Benchmarking?
The process of comparing a website’s (or any
digital product) performance, specifically in
relation to UX metrics, including among
others effectiveness, efficiency, overall
satisfaction and quality perception, to another
that is widely considered to be an industry
standard benchmark or best practice
UX Benchmarking provides a snapshot of the
performance of your product and helps you
understand where you are in relation to a
particular standard.
9. The question we’d like to address today is:
How do you conduct UX Benchmarking?
10. In order to Benchmark, first you need to measure
11.
12. Why is UX Benchmarking important?
Measuring the quality of UX is more important than ever
Consumers have gained a lot of power in recent years
More competition, more maturity in the online marketplace
You can’t really manage what you can’t measure. UX Pros must
deliver more measured value. Mgmt Teams LOVE metrics…
UX Benchmarking allows you to MEASURE performance in two
ways:
1. Baseline studies: Measure UX over time, compare evolution of your
own product and/or service
2. Against competitors within your industry
13. Common UX research & design challenges
Difficulties obtaining objective, statistically significant, actionable
data
Obtaining idea adoption from stakeholders
Geographic/demographic limitations
UX research is costly and time-consuming
UX Benchmarking is therefore quite challenging
This is where large-scale, remote unmoderated
usability testing really helps
14. 3 pillars of UX Benchmarking
1. Plan &
Goals
UX Benchmarking
2. UX research
3. Recruiting
participants
methods &
tools
15. 4 must-dos with UX Benchmarking
1. Plan ahead:
Define measurable & comparable UX-related KPIs (will show them in the case study ahead)
Select competitors to benchmark against
Define target audience
Identify common site activities/tasks (recommended between-subjects design)
2. Conduct research using task-based testing (see methods and tools later on)
3. Gather both quantitative & qualitative data
Use the quantitative data to test for statistically significant differences between
means or proportions
Use qualitative data to support your findings and understand the why
4. Research at least twice a year (or even quarterly) to measure design changes
and trends in expectations
18. UX research methods & tools
We recommend the following combination:
• Remote Unmoderated Usability Testing (or task-based survey tools)
• Remote Unmoderated Usability Testing Video Tools
19. What about recruiting participants?
3 choices to recruit participants:
1. Panel vendor (SSI, Research Now, etc)
2. Private Mailing List
3. Intercept visitors
21. Study Background
• Competitive benchmark of American Airlines vs. Delta
Study
Air Lines websites
• Using task-based surveys, or Remote Unmoderated
Usability Testing
• Date: April 24 – April 25, 2012
• 100 participants (50 per site)
Panel
• Must fly for business or leisure more than twice a
year
• Must use websites to make travel arrangements
• Panel source: Survey Sampling International (SSI)
Tasks
• Go through the entire process of booking a flight
• Locate the total price for checking in two bags
www.userzoom.com
22. Task-based Surveys
(aka unmoderated large-scale remote usability testing)
Go through the entire process of booking a flight
Go through the entire process of booking a flight
• Large sample of participants take the study simultaneously
• Natural context
• International/domestic testing
• No moderation needed
• Usability + Behavior + Feedback
www.userzoom.com
23. Study Design
American
Airlines Task 1 Task 2
Screener Pre-task questions
Follow-up q’s
Delta
Air Lines Task 1 Task 2
www.userzoom.com
24. Study Design
Pre-task brand attributes: Which of the following words do
Pre-task you associate with the AIRLINE brand?
Go through the entire process of booking a flight
Tasks Locate the total price for checking in two bags
Self-reported success
How satisfied were you with the process of booking a flight?
Ease of Use
Post-tasks Ease of navigation
Clear where to start
Satisfaction with amount of time to complete task
Overall rating
Post-task brand attributes
Wrap-up What did you like the most about the AIRLINE website
What did you like the least about the AIRLINE website
Likelihood to recommend (Net Promoter Score )
www.userzoom.com
25. Participant Profile
Age range Gender
Under 18
14% 18-20
8% 31% 21-30
45% Male
31-40
55% Female
41-50
27%
20% 51-60
60+
How often do you fly yearly (for business or leisure)?
I don't fly regularly
10+ times
Participants were
5-10 times required to book
their travel online in
2-4 times order to qualify
1 time
0% 20% 40% 60% 80%
www.userzoom.com
26. Brand Perception
%
Pre Post Change Pre Post % Change
Friendly 41% 14% -27% 42% 10% -32%
Customer 27% 27% 0% 26% 40% +14%
Focused
Safe 57% 27% -30% 44% 24% -20%
Good 41% 31% -10% 36% 22% -14%
value
www.userzoom.com
27. Task 1:
Finding a round-trip flight
Your Task: Find a round-trip flight departing May 1st and returning May 6th that you would consider booking on Delta Air
Lines/American Airlines website. You are travelling from San Francisco, CA to New York, NY. After that, please go through
the entire process of booking a flight until you get to the "Purchase" button.
Delta Air Lines American Airlines
Task Success Task Success
Median Success Time Median Success Time
3:43 3:56
On both studies the majority of participants self-reported being able to successfully complete the
booking process. There was not a significant difference in success rate or completion time (90%
Confidence Level).
www.userzoom.com
28. Task 1:
Finding a round-trip flight
Post-Task Questionnaire
Top Box Scores*
How satisfied were you with the process
of booking the flight? 50% 55%
Booking the flight was easy
64% 67%
It was easy to navigate the website
64% 65%
It was clear how to start booking the
flight 62% 82%
I am satisfied with the amount of time it
took to complete the task 64% 80%
Participants that completed the American Airlines task reported being clearer on where to start and
more satisfied with the time it took to complete booking (90% Confidence Level). All percentages
represent top box scores.
*Top box scores show the percentage of participants that agreed strongly with the statements (selected 6-7 on a 1-7 scale)
www.userzoom.com
29. Task 2:
Find the price for checking 2 bags
Your Task: Find the total price for checking in two bags for your flight from San Francisco to New York.
You purchased a standard economy class ticket.
Delta Air Lines American Airlines
Task Success Task Success
Median Success Time Median Success Time
1:48 2:00
On both studies the majority of participants were not able to successfully identify the price of
checking two bags. There was not a significant difference in success rate or completion time (90%
Confidence Level).
www.userzoom.com
30. Task 2:
Heatmaps - First Click
On both Delta and American Airlines participants used similar strategies when searching for baggage
checking fees. The most commonly used strategy was to start by clicking the flight booking form
fields/buttons (28% Delta “To Airport” field and 24% American Airlines “Find Flights” button).
Participants that first located and clicked on a “Baggage” had higher success rates (70% Delta and 60%
American Airlines).
www.userzoom.com
31. Task 2:
Find the price for checking 2 bags
Post-Task Questionnaire
Top Box Scores*
How satisfied were you with the process
of searching for the total price for 30% 20%
checking in two bags?
Finding the total price was easy
32% 24%
It was easy to navigate the website
38% 36%
It was clear how to start searching for
the total price 32% 32%
I am satisfied with the amount of time it
took to complete the task 34% 34%
Participants that completed the Delta task reported being more satisfied with the process and ease of
the task. The differences are not significant (90% Confidence Level). All percentages represent top
box scores.
*Top box scores show the percentage of participants that agreed strongly with the statements (selected 6-7 on a 1-7 scale)
www.userzoom.com
32. Final Questionnaire:
What did you like the most about AA/Delta?
Representative Quotes: Representative Quotes:
“very easy to navigate” “the prices”
“easy to find flights” “easy to navigate”
“The modern look of it”
Several of the Delta participants commented that they liked the appearance of the Delta website.
There were multiple American Airlines participants that responded liking American Airline’s pricing.
www.userzoom.com
33. Final Questionnaire:
What did you like the least about AA/Delta?
Representative Quotes: Representative Quotes:
“the pricing” “finding information regarding
“Not easy to find bag fees” baggage pricing”
“Prices”
“finding things other than straight
flight booking”
For both websites participants commented that the pricing was high and that it was difficult to locate
the bag fees.
www.userzoom.com
34. Final Questionnaire:
Net Promoter Score
NPS = % of Promoters
(9s and 10s) - Detractors
(0 through 6)
0 1 2 3 4 5 6 7 8 9 10 NPS
0% 4% 4% 4% 6% 18% 6% 10% 22% 6% 20% -16%
Delta Air Lines
4% 6% 4% 0% 4% 10% 12% 18% 14% 12% 18% -10%
American Airlines
Detractors Promoters
When asked if they would recommend the Delta/AA website to a family member,
friend, or colleague both sites had negative NPS scores.
www.userzoom.com
35. Some takeaways…
1. UX Benchmarking, while challenging, is not mission impossible
2. Given the right plan, methods and tools, results are really valuable
3. UX Pros must deliver more measured value
4. There are plenty of UX metrics to be measured, including:
Task success
Time on task
NPS
Reasons for problems, abandonment
The WHYs
Etc.
37. Thanks so much for your time!
Check out our upcoming news and events at www.userzoom.com
Sunnyvale (USA) Barcelona (Spain) Cheshire (UK)
440 N. Wolfe Rd. Sunnyvale, Av. Diagonal 419 3º 2ª 08008 Booths Park. Chelford Road, Knutsford,
CA 94085 Barcelona Phone: +34 93 414 Cheshire WA16 8GS
Phone: +1 (408) 524 7445 7554 Fax: +34 93 209 8380 Phone: + 44 (0)1565 759890
Contact: Alfonso de la Nuez Contact: Xavier Mestres Cell: + 44 (0)7900 472 920
Contact: Arthur Moan
Hinweis der Redaktion
In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.