This document is a transcript from a workshop on measuring user experience (UX) metrics. It discusses defining key metrics that measure whether a product is achieving its goals and serving users. Participants are guided through exercises to identify a product or feature, its key users and uses. They then brainstorm metrics to measure the key use and select one metric to define more specifically. The document stresses that good metrics are clear, comparable, normalized, and actionable to change behavior. It also discusses qualitative and quantitative research, common metrics, and how to capture metrics through instrumentation or manual tracking over time.

1. True University | Measure What Matters | @katerutter | June 13, 2017
Making the Most of UX Metrics
Tuesday, June 16, 2017 | Kate Rutter | @katerutter
Measure what
Matters

2. True University | Measure What Matters | @katerutter | June 13, 2017
tweet-friendly
Hello!
@katerutter

3. True University | Measure What Matters | @katerutter | June 13, 2017
Say “hi” and
introduce
yourselves.
(These are the folks who will
make your work awesome.)
2 min
Let’s get to know each other.

4. True University | Measure What Matters | @katerutter | June 13, 2017

5. True University | Measure What Matters | @katerutter | June 13, 2017
I am not an expert on your business.
You are.
My goal is to help you explore
ways to measure how well your product
serves real humans...
so you can use metrics to design & improve
the products you build.
What to expect...

6. True University | Measure What Matters | @katerutter | June 13, 2017
What to expect...
You may not like the pace/process/
pressure/perspective, etc. etc.
That is okay.
All I ask is that you
keep an open mind and try it.

7. True University | Measure What Matters | @katerutter | June 13, 2017
how do we know
our work is…
WORKING?
*

8. True University | Measure What Matters | @katerutter | June 13, 2017
How do you
measure UX
now?

9. True University | Measure What Matters | @katerutter | June 13, 2017
3 questions
1.What are we trying to
accomplish?
2.What does that look like?
3.How will we measure the
work & track progress?

10. True University | Measure What Matters | @katerutter | June 13, 2017
Let’s get warmed up
Pick an idea to use
for the day
Task-a-
Doodle
write it on a
stickynote
A product you work on
An idea for a new product
or
Product • Feature • Interaction • Customer problem
1 min

11. True University | Measure What Matters | @katerutter | June 13, 2017
Who are the people?
(the users)
What problems do
they have?
What is a proposed
solution?
Product Snapshot
BE SPECIFIC
2 min

12. True University | Measure What Matters | @katerutter | June 13, 2017
a mobile app for
delegating tasks
Working parents with
with kids
need to know when
something’s done
Too much to do, not enough time
need to share
tasks on the go
Task-a-doodle 5 3 2017Who are the people?
(the users)
What problems do
they have?
What is a proposed
solution?
Product Snapshot
BE SPECIFIC
2 min

13. True University | Measure What Matters | @katerutter | June 13, 2017
3 questions
1.What are we trying to
accomplish?
2.What does that look like?
3.How will we measure the
work & track progress?

14. True University | Measure What Matters | @katerutter | June 13, 2017
Features
Users
Uses
Product
Needs
Company
Purpose
sketches,
prototypes,
pixels
UI
via @luxrco
UX Stack
WHITEBOARDABLE

15. True University | Measure What Matters | @katerutter | June 13, 2017
Meet Erika, a socially engaged college student.
Erika needs to feel closely connected with friends
near and far, every day.
using status updates on her “wall”, messages,
comments and “likes”
in a way that’s universal, clean, consistent, and fast.
With Facebook, Erika can share her latest
thoughts and see what her many friends are up to,
Facebook

16. True University | Measure What Matters | @katerutter | June 13, 2017
Meet Erika, a socially engaged college student.
Erika needs to feel closely connected with friends
near and far, every day.
using status updates on her “wall”, messages,
comments and “likes”
in a way that’s universal, clean, consistent, and fast.
With Facebook, Erika can share her latest
thoughts and see what her many friends are up to,
Facebook
Key
Use

17. True University | Measure What Matters | @katerutter | June 13, 2017
Meet Mary, a working professional with two kids.
Mary needs to share the load of household tasks
with her family, and get help completing tasks.
using the task scheduler, task sharer and done
notifications.
in a way that’s friendly, clear, visual, and fast.
With Task-a-doodle, Mary can create tasks, share
them with her spouse and kids, and know when
something gets done
Task-a-doodle

18. True University | Measure What Matters | @katerutter | June 13, 2017
Meet Mary, a working professional with two kids.
Mary needs to share the load of household tasks
with her family, and get help completing tasks.
using the task scheduler, task sharer and done
notifications.
in a way that’s friendly, clear, visual, and fast.
With Task-a-doodle, Mary can create tasks, share
them with her spouse and kids, and know when
something gets done
Task-a-doodle
Key
Use

19. True University | Measure What Matters | @katerutter | June 13, 2017
What can our
customer do with our
product that they
can’t do without it?
Ask:

20. True University | Measure What Matters | @katerutter | June 13, 2017
With _________, ________ can
________________________.
persona
name
goal statement
product
feature
interaction
Key Use
2 min

21. True University | Measure What Matters | @katerutter | June 13, 2017
3 questions
1.What are we trying to
accomplish?
2.What does that look like?
3.How will we measure the
work & track progress?

22. True University | Measure What Matters | @katerutter | June 13, 2017
Who is our user?
Where do they go &
what do they do? What
do they think/feel/say?
Ask:

23. True University | Measure What Matters | @katerutter | June 13, 2017
Examples from Foodspotting

24. True University | Measure What Matters | @katerutter | June 13, 2017
rapid viz

25. True University | Measure What Matters | @katerutter | June 13, 2017
You can draw a person in 4 simple steps....
Star People

26. True University | Measure What Matters | @katerutter | June 13, 2017
smile frown
up brows
neutral
down
brows
no brows
add a mouth...
add eyebrows...
sketch via @luxrco; also see @austinkleon
Expressions Matrix

27. True University | Measure What Matters | @katerutter | June 13, 2017
Task-a-doodle,
Mary
5 3 2017
Share tasks
with others.
OK to be obvious
5 min
Make a quick
sketch.
What can
someone DO with
your product?

28. True University | Measure What Matters | @katerutter | June 13, 2017
Key
Use

29. True University | Measure What Matters | @katerutter | June 13, 2017
3 questions
1.What are we trying to
accomplish?
2.What does that look like?
3.How will we measure the
work & track progress?

30. True University | Measure What Matters | @katerutter | June 13, 2017
“ ~ Joshua Porter
Your metrics will be as
unique as your business.

31. True University | Measure What Matters | @katerutter | June 13, 2017
“Over 10,000 downloads!”
“Avg time on site is 22 minutes!”
“+450 new sign-ins this week!”

32. True University | Measure What Matters | @katerutter | June 13, 2017
None of these numbers help us know
our product is working for our users.
“Over 10,000 downloads!”
“Avg time on site is 22 minutes!”
“+450 new sign-ins this week!”
Nope.

33. True University | Measure What Matters | @katerutter | June 13, 2017
User Research
Surveys
Usability
Testing
Interviews
Observation
Field
research
Metrics &
Analytics
Quantitative
tells you what is
happening
Qualitative
tells you why
things happen
Generative
helps you come up
with new ideas
Evaluative
helps you assess or
validate something
Big data?
best source
of possible
metrics
NUMBERS STORIES

34. True University | Measure What Matters | @katerutter | June 13, 2017
Metrics &
Analytics
• Click tracking
• Analytics packages
• A/B testing
• KPIs / OKRs
• Usage stats
• Conversion

35. True University | Measure What Matters | @katerutter | June 13, 2017
Common metrics
DAU Daily Active Users
MAU Monthly Active Users
CPA Cost per Acquisition
ARPDAU Average Revenue per Daily Active User
Retention How long they stay & how
frequently they interact
Churn How many users you lose vs. keep
LTV LifeTime Value
NPS Net Promoter Score Sketch via Build Better Products by Laura Klein, Rosenfeld Media

36. True University | Measure What Matters | @katerutter | June 13, 2017
Product
Traffic
Retention

37. True University | Measure What Matters | @katerutter | June 13, 2017
Product
Traffic
Retention
use

38. True University | Measure What Matters | @katerutter | June 13, 2017

39. True University | Measure What Matters | @katerutter | June 13, 2017
…measures the
usage of your
product by a
person.
a good metric…
Clear &
specific
Comparative
Normalized
Actionable
Changes
your behavior

40. True University | Measure What Matters | @katerutter | June 13, 2017
• Know what interactions represent
the desired user behaviors.
• Identify the specific interactions
that support the action or behavior.
• Find the numbers to track
these interactions.
Clear & specific
share a task
*
a good metric…

41. True University | Measure What Matters | @katerutter | June 13, 2017
Normalized
(rate or ratio)
• Percentages are
inherently comparative
• They balance out big
changes in total
quantities.
782 total users
420 share a task
1,275 share a task
6,835 total users
18.6%
53.7%
a good metric…

42. True University | Measure What Matters | @katerutter | June 13, 2017
Comparable
(time-stamped)
• Captured at regular
intervals
• Directly comparable
over a span of time WTFJHT?
6
a good metric…

43. True University | Measure What Matters | @katerutter | June 13, 2017
Actionable, not vanity
• Total number of users
• Downloads
• Page views
• Likes
• Funding press releases
• Articles in media
• etc…
a good metric…

44. True University | Measure What Matters | @katerutter | June 13, 2017
Actionable, not vanity
• Total number of users
• Downloads
• Page views
• Likes
• Funding press releases
• Articles in media
• etc…
a good metric…

45. True University | Measure What Matters | @katerutter | June 13, 2017
“ ~ Alistair Croll
with Ben Yoskovitz
Don’t just ask questions. Know how
the answers to the questions will
change your behavior.
a good metric…

46. True University | Measure What Matters | @katerutter | June 13, 2017
vanity
total number
of registered
users
unhelpful
sign-ups
% of users
who share a
task 3+
times a day,
per week
awesome
better
% of users
who sign in 3+
times a day,
per week
good
% of new
users per
week
a good metric…

47. True University | Measure What Matters | @katerutter | June 13, 2017
= 7+ friends within 10 days
of account creation
= 1 file in one folder
= user comes back the day
after signing up for a game
= The third run Leading,
not lagging

48. True University | Measure What Matters | @katerutter | June 13, 2017
So…
How do we get
our awesome
metric?

49. True University | Measure What Matters | @katerutter | June 13, 2017
a good metric... a great metric...
...makes you look at all the other
metrics and say “none of those
other numbers matter if we don’t
get this right first.”
...measures the usage of your
product by a person. The usage
should be specific to features that
deliver value to your user.
vanity
total number
of registered
users
unhelpful
sign-ups
% of users
who share a
task 3+
times a day,
per week
awesomebetter
% of users
who sign in
3+ times a
day,
per week
good
% of new
users per
week
Metrics checklist

50. True University | Measure What Matters | @katerutter | June 13, 2017
Brainstorm 5 ideas for
metrics
What could you count to make sure
the Key Use is happening?
3 min

51. True University | Measure What Matters | @katerutter | June 13, 2017
Pick one to focus on
.5 min

52. True University | Measure What Matters | @katerutter | June 13, 2017
3. Is there an object
basis?
• “per user” [ ___/user]
• “per user per week” (fancy!)
• Examples: per user, per session,
or by interaction or item
(messages per file, emails
per thread)
* not required, but super-helpful
• “per week” [ ___/wk ]
• “per month” [ ___/mo]
2. Is there a time basis?
• “number of...” [ # ]
• “average number of...” [avg ]
• “percent of...” [ % ]
1. Does the metric
begin with a number?
Make it awesome
% of users
who share a
task 3+
times a day,
per week
awesome
better
% of users
who sign in
3+ times a
day,
per week
good
% of new
users per
week
3 min

53. True University | Measure What Matters | @katerutter | June 13, 2017
Share your work
Review the snapshot
& use.
Peers: make
recommendations
Improve
as a pair
Key Metric 2 min
each

54. True University | Measure What Matters | @katerutter | June 13, 2017
Put it on your dashboard
.5 min

55. True University | Measure What Matters | @katerutter | June 13, 2017
So…
then
what?

56. True University | Measure What Matters | @katerutter | June 13, 2017
Three ways to gather data:
• Non-technical
• Technical “by hand”
• Instrumented with software
Instrumentation
The process of setting up the technology to capture
the numbers that matter to you. This often means
using an analytics package or writing custom code.

57. True University | Measure What Matters | @katerutter | June 13, 2017
UI Sketch
•Sketch the UI -
simple is good!
•Annotate interactions
to capture
•Make notes on how
to calculate.
Where in the UI
can you capture
data?

58. True University | Measure What Matters | @katerutter | June 13, 2017
Metrics Plan
•Set a goal
•Set a time span
•Think about how to
collect the data
How will you track
progress?

59. True University | Measure What Matters | @katerutter | June 13, 2017
Track over time

60. True University | Measure What Matters | @katerutter | June 13, 2017
Track over time

61. True University | Measure What Matters | @katerutter | June 13, 2017
Track over time

62. True University | Measure What Matters | @katerutter | June 13, 2017
how do we know
our work is…
WORKING?
*

63. True University | Measure What Matters | @katerutter | June 13, 2017
3 questions
1.What are we trying to
accomplish?
2.What does that look like?
3.How will we measure the
work & track progress?

64. True University | Measure What Matters | @katerutter | June 13, 2017
Key
Metric
Key
Use

65. True University | Measure What Matters | @katerutter | June 13, 2017
Features
Users
Uses
Product
Needs
Company
Purpose
sketches,
prototypes,
pixels
UI
via @luxrco
UX Stack

66. True University | Measure What Matters | @katerutter | June 13, 2017
Products that
make our
users’ dreams
become real

67. True University | Measure What Matters | @katerutter | June 13, 2017
Geek out & learn more

68. True University | Measure What Matters | @katerutter | June 13, 2017
Kate Rutter
@katerutter
http://intelleto.com
Thanks!
Swearing, drinking & UX…give it a listen!
http://bit.ly/whatiswrongwithux

69. True University | Measure What Matters | @katerutter | June 13, 2017
Attributions
• UX Stack: Adapted from Janice Fraser, Luxr
• Book cover images via Amazon: https://www.amazon.com/
• Purpose statements: Facebook via https://investor.fb.com/resources/default.aspx;
• Sketches via : Alexa Andrzejewski, foodspotting (now OpenTable);
• I <3 Metrics pin: https://www.zazzle.com/i+heart+metrics+gifts
• Joshua Porter: http://bokardo.com/ Quote via https://www.slideshare.net/andrew_null/metrics-driven-design-by-joshua-porter/28-
Saturday_March_12_2011
• OMTM citations: Lean Analytics via http://leananalyticsbook.com/
Nice reads:
• Great metrics article: https://medium.com/the-year-of-the-looking-glass/metrics-versus-experience-a9347d6b80b
• KISSmetrics blog: https://blog.kissmetrics.com/
• Joshua Porter, Bokardo slideshare: https://www.slideshare.net/andrew_null/metrics-driven-design-by-joshua-porter/28-
Saturday_March_12_2011