Jules Skopp gave this workshop at UX STRAT Europe 2021 on the topic: "Design with Data: Objectives and Metrics, the Drivers of UX Strategy"

1. Design
with data
Objectives and metrics,
the drivers of experience strategy

2. About

3. Hello!
Data matters, d’oh!
Confidence (new products)
Sustainable outcomes (existing products)
Good metric
Metrics tree
Demo
9:00
:40
10:00
:40
:50
11:10
:40
:50
12:20

4. About digital products,
transferable elsewhere too

5. Slides and handouts
will be provided

6. Whatever you share here
stays here

7. Add your name to Post Its
I won’t call you out otherwise

8. Hello!

9. Jules’ day

11. 1. Note challenges
2. Group them
3. Talk about the top one

12. Write down your
challenges
5 mins

13. Design
system
adoption
Impact
strategy

14. Group them in
common themes

15. Describe your
top challenge
2 mins each

16. Sketch out
your product
Include objectives

18. Data matters

19. Designers themselves have
been partly to blame in the
past: they have not always
embraced design metrics or
actively shown management
how their designs tie to
business goals.
mckinsey.com/business-functions/mckinsey-design/our-insights/the-business-value-of-design, October 2018

20. Quant
Analytics
Eye tracking
Tree tests
A/B/n tests
Surveys*
Qual
Interviews
Usability tests
Recordings
Surveys*

21. nngroup.com/articles/quant-research-practice

22. What are your
challenges with
metrics/data?

23. Purpose of data

24. Convince
Validate
Monitor

25. Convince
Validate
Monitor

26. Does your data
challenge fall into
“Convince” bucket?

27. Convince
Validate
Monitor
Antitheses

28. Antitheses
Seeking data for other solutions
that solve the same problem,
or data that rejects your
hypotheses
I don’t care
about the
result
I do care
about
specific
result
I do care
about
global
results
only

30. Convince
Validate
Monitor

31. Does your data
challenge fall into
“Validate” bucket?

32. Convince
Validate
Monitor
Hypotheses

33. By doing action
I expect outcome
Hypotheses
By doing _________,
I expect ____________
Available Results

35. Convince
Validate Indecisiveness
Monitor

36. Convince
Validate Disagreement
Monitor

37. Convince
Validate Conversation stopper
Monitor

38. Does your data
challenge fall into
“Monitor” bucket?

39. Convince
Validate
Monitor

40. Convince
Validate
Monitor
Predict

41. Predict
What happened/will happen
if a metric changes?
Repeat rate
Conversion

42. Purpose of data:
Convince, validate, monitor

43. Confidence

44. Remember any
prioritization
methods?

45. Kano model
Must-haves, performance, and
delighter features
Jared Spool’s Medium article

46. Mo-st have
Should have
Could have
Won’t have
productplan.com/glossary/moscow-prioritization

47. Potential
Impact
Ease
widerfunnel.com/pie-framework

48. (Reach)
Impact
Confidence
Ease
productplan.com/glossary/ice-scoring-model

49. 3+1 questions:

50. Are you defining
what to build next?

51. Sketch how does
your org innovate

52. What is the next
thing you’d
like to pursue?

53. Big decision:
Merge products with similar goals?

54. Rate analysis
Performance

55. How confident
are you about
your strategy?

56. Confidence
0
10
Deploy
itamargilad.com/resources/confidence-meter-calculator

57. Confidence
0
10
My idea is
the best
Deploy

58. Confidence
10
OKRs
0

59. Confidence
10
0.5
OKRs

60. Confidence
10
1 Competitors/market

61. Confidence
10
3
I dig it
Me too!

62. Confidence
10
5
A/B tests
MVP light launch

63. Confidence
Monitor
Validate
Convince
Release

64. Are you familiar
with the local
maxima problem?

68. Good enough

69. Consider multiple directions

75. The approach that matters
Ideas you explored

76. Confidence

77. Stay two steps ahead with
Confidence tracking
Multiple strategies

78. Sustainable
outcomes

79. 1. Conversion rate
2. % of users reached
3. Revenue potential
4. Page views
5. Trust

82. 1. Users’ revenue
2. Travelers’ conversion
3. Traveler page views
4. Trust

83. What are the
measurable
outcomes of your
product?

84. Why not?
Net promoter score

85. Promoters
>35%
retently.com/blog/good-net-promoter-score
NPS survey interpretation,
measure of success
Promoters
NPS = % promoters - % detractors
Detractors
8
6

86. retently.com/blog/good-net-promoter-score

87. Why not?
Product/market fit

88. How would you feel if you could
no longer use the product?
Very
disappointed
Not
disappointed

89. Promoters
>40%
review.firstround.com/how-superhuman-built-an-engine-to-find-product-market-fit
Product/market fit survey
interpretation, measure of success
Very disappointed

90. review.firstround.com/how-superhuman-built-an-engine-to-find-product-market-fit

91. Promoters

92. Source: Internal report—please don’t share publicly

93. You are accountable
for these metrics
NOT

94. Prioritization
—how do you
participate?

95. Organizational priorities:
Objectives and key results

97. Vanity metrics
Bounce rate
Time
Team velocity
Page views
Users
Revenue
productplan.com/glossary/vanity-metrics

98. Principle:
Make it about the user success

99. Good metric

100. Return to your
previous exercise
and check for
these factors...

101. Log in
Dashboard
To detail
Take action
Behavior-based
Conversion funnel
1.
2.
3.
4.

103. Log in : take action
Ratio or percentage

104. Product view
Sat on it
per

105. Pulse

107. Per week
Per time frame

108. Plot on the timeline

109. Not enough traffic
Leading indicators

110. 47 minutes

111. More than one metric

113. Visits
Product view

114. Login : calendar
Always more than one metric
Login : create
Login : invite
Login : duplicate
Login : view item

115. Behaviour
Ratio
Time-framed
More than one

116. Why… yes!
Conversion

117. Conversion
Affecting behavior, removing
Friction towards objective, is our job
60 838
12 40
15 190

118. ...the system automatically
adds 30 business metrics to
each test, just to make sure
you don’t mess up anything
The well-known rumor about Booking.com

121. Yes again!
Satisfaction

122. Satisfaction
“CSat” monitors quality of service
qualtrics.com/uk/experience-management/customer/
measure-customer-satisfaction/
Overall, how satisfied are
you with “La Jolla Grove
restaurant”?

125. YES!
Attitudes

126. For example Google’s HaTS or relevant qual. research
Challenge: feature creep
Solution: attitudinal research

127. Attitudes
The highest goal of experience is to be
able to change the opinion of users
Google’s HaTS

128. Alright, yes!
Competitive UX comparison

129. Velocity
New users
Time spent
Features

130. Principle:
Make it about the user success

131. Users’ money made
Users’ phys. betterment
Users’ goals reached
...

133. Good metrics:
Documenting user success
Behaviors
Attitudes/satisfaction
People-centric framework, not yet copyrighted, by Jules

134. Metric tree

135. Increase
revenue
Decrease
costs
Increase
new business
Increase
existing business
Increase
shareholder value
Jared Spool: UX strategy means business

136. North star

137. Increase
revenue
Decrease
costs
Increase
new business
Increase
existing business
Increase
shareholder value
The north star metric can be… well... not exciting

138. Chair

139. Pick one Discover store
Sit on it Compare Walk in
Sell chairs

140. Product view Impression
Sat on it Comparisons Visits
Chairs sold

141. Think about the
North-Star metric
for your company

142. North star

143. Pick one Discover store
Sit on it Compare Walk in
Sell chairs

144. Pick one Discover store
Sit on it Walk in
Sell chairs

145. Fill in each next lane
with the behavioral
metrics leading into
previous row’s
metrics

147. Discussions matter

148. Appendix: Demo

152. Install Event tracking tracker extension
View on Chrome Store

153. Install GA debugger extension
View on Chrome Store

154. Event category
This is a product space, like
“ShoppingCart”
Disclaimer: This might differ from Partner Central
Shopping cart
Next

155. Event action
All the taps/clicks captured
User.Click.Cart.Next
1. Shopping cart
2x 3x
Next

156. Event label
Distinguish between same events
User.Click.Cart.Next label = step1
1. Shopping cart
2x 3x
Next

157. 50$
Event value
Attach value to events
User.Click.Cart.Next step 1 value = 50
1. Shopping cart
2x 3x
Next

158. Event
Action: User.Click.Cart.Next
Label: User.Click.Cart.Next=step2
Value: User.Click.Cart.Next step 2 = 50
2. Checkout
Next
Insurance
+$10

159. Event action
Ticking the box action
User.Click.Cart.Insurance
2. Checkout
Next
Insurance
+$10

160. Event action
Ticking the box action
User.Click.Cart.Insurance
2. Checkout
Next
Insurance
+$10

161. Custom
dimension
Logs states as users do stuff
IsInsured = True
2. Checkout
Next
Insurance
+$10

162. 3. Summary
2x 3x
Order $60
All together
Event category: Shopping.Cart
Event action: User.Click.Cart.Next
Event label: Step3
Event value: 60
Custom dimension: IsInsured=True

164. Remove all of these

165. Name your
exploration

166. Add user
segment

167. Add user
segment:
All...

168. A/B/n testing

169. Original Var A Var B Var C

170. Multivariate testing

171. Original Var A Var B Var C

172. Multivariate testing
NOT

173. Original Var A Var B
Var B

174. Factorial experiment
en.wikipedia.org/wiki/Factorial_experiment

175. Original Var A.1 Var A.2 Var B.1 Va

177. Other metric