1.
M E A S U R I N G U X O N L A R G E S CA L E
How Google Improves User Experience
Submitted by
Priya Saraswat
HCDE 517

2.
S O M E FAC T S
Authors - Kerry Rodden, Hilary Hutchinson and
Xin Fu – quantitative UX Researchers at Google.
The article got published in 2010 at SIGCHI
Conference on Human Factors in Computing
Systems.

3.
U S E R E X P E R I E N C E I S M E A S U R A B L E
If you don’t measure it, you can’t improve it

4.
T H E C H A L L E N G E
• It’s easier to collect attitudinal data like satisfaction
both on a large scale(through surveys) and small
scale(usability testing)
• But behavioral data collection methods are still small
scale e.g lab testing to see effectiveness(task
completion, errors) and efficiency(time on task)

5.
S O H O W TO M E A S U R E U X O N L A R G E S CA L E ?
• A KPIs - Key Performance Indicators
• PULSE - Page Views, Uptime, Latency, Seven Day Active Users and
Earnings
Why they may not solve the purpose?
• Most of these metrics are business centered rather than user
centered.
• The changes in the user experience because of a newly-
implemented product change tend to be generic in nature and
may not directly relate to the quality of the UX or the overall goals
of the project.

6.
T H E H E A RT F R A M E WO R K
The article talks about using data analysis as just another UX
research method to define appropriate metrics that reflect:
• The quality of user experience (the HEART framework)
• The goals of your product or project (the Goals-Signals-
Metrics process)

7.
H A P P I N E S S
What can be measured - satisfaction, visual appeal,likelihood to
recommend and perceived ease of use
Method(s) - A general well designed surveys
Example - After launching a redesign or new feature, asking users a
one line survey to determine the satisfaction.

8.
E N G AG E M E N T
What can be measured - frequency, intensity or depth of interaction
Method(s) - Analytics
Example - Number of visits per user per week or number of photos
uploaded per user per day. Good to calculate average per user
rather than total

9.
A D O P T I O N
What can be measured - how successful you are at attracting new
users.
Method(s) - Analytics
Example - Upgrades to the latest version, New subscriptions
created, Purchases made by new users

10.
R E T E N T I O N
What can be measured - the rate at which the existing users are
returning
Method(s) - Analytics, should be measured over different periods
Example - Number of active users remaining present over time,
Repeat purchases over time

11.
TA S K S U CC E S S
What can be measured - efficiency, effectiveness, error rate
Method(s) - Remote Usability Testing, Benchmarking study
Example - tasks around a multi-step sign-up process to measure
how closely user follow it

12.
• Goals - Articulating the goals of a product or a feature in terms of
user experience
• Signals - Find out what data sources be considered for these
signals
• Metrics - Translating these signals into specific metrics
G OA L-S I G N A L S - M E T R I C S

13.
CO M PA R E D W I T H
Signals
Metrics
Goals
More for less : A novel hybrid method to maximize the impact
of research that talks about how involving team members from
the start help in getting more team buy-in at the end

14.
D I S C U S S I O N Q U E ST I O N
• How HEART framework can be made applicable to
enterprise products where most of the products are
used as tools to get a job done?