Unpacking and measuring the impact of UX design. A talk presented at the inaugural UX South Africa Conference on 3 September 2014.

1. The ROI of UX Design
Unpacking and measuring the impact of UX design
Werner Janse van Rensburg
Co-founder, ISOflow

2. Four Parts:
1 The Art of Measuring
2 Data Types
3 Metrics
4 Case Studies

3. Part
1
The Art of Measuring
(the short version)

4. Image: http://www.turbosquid.com/3d-models/3d-ruler-model/371813

5. 1664: Seconds-Pendulum. Distance
for a half-rotation in 1 second
Image: http://en.wikipedia.org/wiki/History_of_the_metric_system
Source: http://en.wikipedia.org/wiki/Seconds_pendulum

6. 1789: Decimal system introduced. Almost time
too (2014-10-03.427 or 42.7% of a day)
Source: http://en.wikipedia.org/wiki/History_of_the_metre

7. 1792–98: One ten-millionth of the distance from
the Earth's equator to the North Pole (at sea level).
Dunkirk to Barcelona, by Méchain & Delambre.
Image: http://www.examiner.com/slideshow/hurricane-arthur-speaks-math-as-well-as-weather

8. 1799: Mètre des Archives
(Platinum Bar)
Source: http://en.wikipedia.org/wiki/History_of_the_metre

9. Meter standard engraved in
marble in 16 locations in Paris
Image: http://fr.wikipedia.org/wiki/M%C3%A8tre

10. The public was educated
on these new standards
Image: http://en.wikipedia.org/wiki/History_of_the_metric_system

11. 1875: BIPM and the International
Prototype Meter (Platinum-Iradium)
Image: http://en.wikipedia.org/wiki/History_of_the_metre

12. 1927: 7th BIPM General Conference:
The unit of length is the metre, defined by the distance,
at 0°, between the axes of the two central lines marked
on the bar of platinum–iridium kept at the Bureau
International des Poids et Mesures and declared
Prototype of the metre by the 1st Conférence Générale
des Poids et Mesures,
this bar being subject to standard
atmospheric pressure and supported
on two cylinders of at least one centi-metre
diameter, symmetrically placed
in the same horizontal plane at a
distance of 571 mm from each other.
,,
Define Source
,,

13. 1960: Krypton standard
and the interferometer.
Later used laser, in 1975.
Source: http://en.wikipedia.org/wiki/History_of_the_metre

14. The length of the path
travelled by light in vacuum
during a time interval of
1/299,792,458 of a second
,,
,,
1983: Speed of light

15. Why am I telling you all this?
We can all
use a ruler
We can all
measure UX

16. Three defining characteristics:
• A user is involved
• That user is interacting with a
product, system, or really anything
with an interface
• The users’ experience is of
interest, and observable or
measurable
Source: http://www.usernomics.com/iaa_aed_2003.pdf
Image: http://defibuk.co.uk

17. Automatic External Defribulators
• Used by untrained persons in public settings.
• Four devices tested by 64 users (16 per device).
• Two devices were successful 100% of the time.
• 9 / 16 (56%) and 4 / 16 users (25%)
FAILED on the remaining two devices
Source: http://www.usernomics.com/iaa_aed_2003.pdf
Image: http://defibuk.co.uk

18. Why is this
important
for us?

19. Part
2
Data Types

20. ISO 9241-11:1998 (Usability)
The extent to which a product can
be used by specified users to
achieve specified goals with
effectiveness, efficiency and
satisfaction in a specified context
of use.
,,
,,
ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability

21. UX Metrics reveals something about:
• Effectiveness
– “Being able to complete a task”
• Efficiency
– “The amount of effort required to complete the
task”
• Satisfaction
– “The degree to which the user was happy with his
or her experience while performing the task”
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

22. Approach: Formative Summative
Image: http://portandfin.com/spaghetti-with-mango-roasted-peppers-basil/

23. Qualitative Data
• Deals with descriptions
• Data can be observed but not
measured
• Colors, appearance, beauty,
smells, tastes etc.
• Qualitative > Quality
Quantitative Data
• Deals with numbers
• Data which can be measured
• Length, height, area, volume,
weight, speed, time,
temperature, humidity, sound
levels, cost, members, ages etc.
• Quntitative > Quanitity
• Gold frame
• Smells Old / Musty
• Nice Texture
• Brush Strokes
• Masterful
• Art: 100 x 140cm
• Frame: 120x160cm
• Weight: 8.5kg
• Surface: 14,000cm2
• Cost: $3m
Image: http://www.artchive.com/meninas.htm Source: http://regentsprep.org/regents/math/algebra/AD1/qualquant.htm

24. • Comprehensive look
at UX Metrics
• Practical Approach
• Help you make the
right decisions
• Examples
• Relevant to many
products and
technologies

25. Helps to answer critical questions:
• “Will the users recommend the product?
• Is this new product more efficient to use than the
current product?”
• How does the user experience of this product
compare to the competition?
• What are the most significant usability problems
with this product?
• Are improvements being made from one design
iteration to the next?”
• … as well as avoid wrong assumptions.
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

26. It’s not the things we don’t know
that gets us in trouble, it’s the
things we do know that ain’t so.
,,
,,

27. To Assume
is to make an
ass out of u and me.

28. Types of Data
Nominal Ordinal Interval Ratio
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

29. Nominal Data
– Just a grouping
– Fruit / Gender / Eye Color
– Type of Car
– Statistics:
• 45% of users are female
• 25% preferred apples
• 200 users had blue eyes
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Image: http://citysportsblog.com/city-sports/fruit-how-much-is-too-much/

30. Ordinal Data
– Ordered data
– Intervals not meaningful
– Order from best to worse
– Rate this site
 Poor  Fair  Good  Excellent
– Statistics:
• 40% rates this site “Good”
• 68% Prefer Option A
• Rated 28th on IMDB Top 250
28
56
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Image: http://www.imdb.com/chart/top

31. Interval Data
– Differences meaningful
– There’s no natural 0
– Temperature, Dates
– System Usability Scale (SUS)
– Statistics:
• Increase / decrease in usability (%)
• Averages / Standard deviation
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Image: http://envirodailyadvisor.blr.com/2013/05/tips-for-safe-mercury-cleanup/ & https://www.iconfinder.com

32. Ratio Data
– Similar to interval data
– With an absolute 0
– E.g. Time, Age, Weight, Height
– Zero has inherent meaning
(Absence of Age / Weight)
– Or no time remaining
– Statistics:
• Twice as fast or half as slow
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Image: http://www.softwareag.com/ & https://www.iconfinder.com

33. Descriptive Statistics
• Describes the data, without saying anything
about the larger population
• Most common measures used:
– Central Tendency
– Variability
– Confidence Intervals
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

34. Measures of Central Tendency
• Mean / Average
• Sum / # of items
• 243/10 = 24.3
• Median
• Middle number when ordered
from small to large. Half of
values are above,
half below.
• 17 (between 16 & 18)
Participant
Task Time
(Seconds)
P1 11
P2 18
P3 12
P4 29
P5 10
P6 53
P7 16
P8 14
P9 22
=AVARAGE function P10 58
=MEAN function
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

35. Participant
Task Time
(Seconds)
P1 11
P2 18
P3 12
P4 29
P5 10
P6 53
P7 16
P8 14
P9 22
P10 58
Measures of Variability
• Range
• Range = Max - Min
• 58-10 = 48
• Variance
• 303.8
• Standard Deviation
• Square root of the variance
• 17.4 Seconds
=MAX and =MIN functions
=VAR and =STDEV functions
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

36. Confidence Intervals
• Confidence Level: 95%
• Want to be 95% Certain
• Willing to be wrong 5% of
time (Alpha)
• Most commonly used:
• Confidence 99%, 95%, 90%)
(Alpha 1%, 5%, 10%)
• E.g. 1% for AED tests
Mean Checkout
Time (Seconds)
(Error Bars Represent 90%
140
120
100
80
60
40
20
=CONFIDENCE(alpha, stdev, sample size) 0
Design A Design B
Checkout Time (Seconds)
confidence intervals)
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

37. Part
3
Metrics
(and how to render results)

38. Categories:
• Performance Metrics
• Issue Based Metrics
• Self Reported Metrics
• Behavioral and Physiological Metrics
• Live Website Data

39. Performance Metrics
• Task Success
• Time on Task
• Errors
• Efficiency
• Learnability
UX

40. Column / Bar Graphs
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

41. Line Graphs
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

42. Stacked Bar / Column Graphs
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

43. Issue-Based Metrics
• Severity
• Frequency
• By Category
• By Task
UX

44. Pie or Donut Charts
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

45. Self-Reported Metrics
• Rating Scales
• Ease of Use
• System Usability Scale
• Satisfaction
• Open Ended Questions
UX

46. Scatterplots
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

47. Source: http://ebusinesshow.blogspot.com/2011/02/third-weekly-task-usability-evaluation.html
62.5%

48. Behavioral and Physiological
Metrics
• Eye Tracking
• Emotion & Stress
UX

49. Image: http://usableworld.com.au/2009/03/16/you-look-where-they-look/

50. Image: http://mashable.com/2014/05/19/tobii-glasses/

51. Image: https://www.flickr.com/photos/56422507@N07/5827330051/

52. Live Website Data
• Basic Web Analytics
• Click-Through Rates
• Drop-Off Rates
• A/ B Tests
UX

53. Image: http://www.thesearchagents.com/2012/04/how-to-create-a-mobile-marketing-dashboard-in-google-analytics/

54. Part
4
Case Studies

55. (Gains – Cost)
Cost
ROI =

56. Every $1 invested in usability
returns between $10 and $100.
,,
,,
IBM, Cost-Justifying Usability

57. And if we don’t…
There’s training, support and maintenance costs.
Costs associated with fixing problems increase:
• $1 to fix in early analysis
• $10 to fix in design
• $100 to fix in a prototype
• $1000 to fix after deployment
Mantei and Teorey (1988)

58. We don’t always measure $
Intranet E-commerce App
Websites
(Content /
Marketing)
Productivity
Time saved
x
hourly cost of
employees
Revenue
Drop-off rates
Conversion Rates
Customer
Satisfaction
Increased
Feature use
Visitors
Pageviews
Customer Loyalty
Sales Impact

60. 10%
Of project budget
The cost of
usability:
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

61. Benefits from usability
160%
72
Usability Metrics
2.6x
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

62. Benefits from usability
109%
72
Usability Metrics
2.1x
With 8 outliers excluded

63. Only one metric, before & after:
• Conversions
• Request for Quote
• Sales
• Reservations
• Help desk calls
• Newsletter signups
• Trial Signups
• Time on Site
• Reduced Exits
• Traffic
• Return Visitors
• Feature Use
• Comments
• Customer Satisfaction
• Task Time
• Success Rate

64. Improvement in Usability Metrics
Metric
Average Improvement
Across Web Projects
Sales / conversion rate 87%
Traffic / visitor count 91%
User performance / productivity 112%
Use of specific (desired) features 174%
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

65. Ways to double your business
B = V x C x L
B = amount of business done by the site
V = unique visitors coming to the site
C = conversion rate
2000-2010
Conversion Decade
L = loyalty
2010-2020
Loyalty Decade
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

66. Skilled-user
Performance
Error-
Avoidance

67. More users, bigger cost impact
• Guest Checkout
• One Click Purchase
• User Ratings

68. Investment would increase further
• Competition
• Return on Investment
• Changing Audience
• Internationalization
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

69. Eurostar
• Usability & IA
• New booking features
• New Brand Identity
• Market-leading experience
• Six-month timeframe
Metric:
Online sales (12 mnths)
Before:
£110 million/year
After:
£136 million/year
Ratio: 124%
Improvement: 24%
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

70. Eurostar
• Issues resolved:
– Error Messages
– Confusing Language
– Confirmation Pages
– User Accounts
– HTML Issues
Metric:
Online sales (12 mnths)
Before:
£110 million/year
After:
£136 million/year
Ratio: 124%
Improvement: 24%
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

71. 100+
Issues

72. 70+
Issues

73. High-fidelity

74. Adobe Kuler
• Color Theme Sharing Site
• Comment Engagement low
• Unconventional Display
Metric:
# of Comments
Before:
6 / day (Average)
After:
37 / day (Average)
Ratio: 617%
Improvement: 517%
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

77. Familiarity does not necessarily
breed contempt in Web design.
Unless a feature offers entirely
new functionality or much
improved intuitive use, it is often
better to stick with a
conventional delivery.
,,
,,
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

78. In closing…

79. Tools, tools, tools…
Remote Usability Testing:
• http://www.optimalworkshop.com/c
halkmark.htm
• http://ethn.io/
• http://www.feedbackarmy.com/
• http://fivesecondtest.com/
• http://www.keynote.com/
• http://www.loop11.com/
• http://www.trymyui.com/
• http://usabilitytools.com/
• http://www.usabilla.com/
• http://www.userlytics.com/
• http://www.usertesting.com/
Offline Usability Testing:
• http://silverbackapp.com/
• http://www.techsmith.com/camtasia.
html
• http://www.techsmith.com/morae.ht
ml
A/B Testing:
• http://www.adobe.com/africa/soluti
ons/testing-targeting.html
• http://useartisan.com/
• http://www.globalmaxer.com/
• http://optimizely.com/
• https://vwo.com/
Source: http://uxpin.com/upload/ux-design-for-startups-marcin-treder.pdf

80. Tools, tools, tools… (more)
ClickTracking:
• http://www.labsmedia.com/clickheat
/index.html
• http://www.clicktale.com/
• http://www.crazyegg.com/
• http://usabilitytools.com/features-benefits/
click-tracking/#tool-description
• http://www.userzoom.com/
Eye Tracking:
• http://eyetrackshop.com/
Session Recording:
• http://www.userzoom.com/
• http://www.ghostrec.com/
• http://mouseflow.com/
• http://www.openhallway.com/
• http://www.tealeaf.com/
• http://usabilitytools.com/features-benefits/
visitor-recording/#tool-description
• http://userreplay.co.uk/
Source: http://uxpin.com/upload/ux-design-for-startups-marcin-treder.pdf

81. Keys to success
to help you with your UX measurements
• Make Data Come Alive
• Don’t Wait to be Asked to Measure
• Measurement is Less Expensive Than You Think
• Plan Early
• Benchmark Your Products
• Explore Your Data
• Speak the Language of Business
• Show Your Confidence
• Don’t Misuse Metrics
• Simplify Your Presentation
10x
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

82. Image: http://fr.wikipedia.org/wiki/M%C3%A8tre

83. Thanks!
Werner Janse van Rensburg
@wernerjvr
@isoflow
www.isoflow.co.za