Nailing Measurement: a Framework for Measuring Metrics that Matter

1. JOSEPHINE LEE | PRODUCT MANAGER | ATLASSIAN Nailing Measurement A Framework for Measuring Metrics that Matter

2. CONTEXT

3. Focusonlocalizationin Confluence. CONTEXT

4. GOAL

5. Deliver10languagesin Confluencebytheendof2018. GOAL

6. Photo credit: memegenerator.net

7. Photo credit: wallpaperbro.com

8. What went wrong?

9. OutcomeOutput

10. OutcomeOutput Deliver10languagesin Confluencebyendof 2018

11. OutcomeOutput Deliver10languagesin Confluencebyendof 2018

12. OutcomeOutput Deliver10languagesin Confluencebyendof 2018 Increaseusersatisfactionby20% withournon-Englishspeaking customersbytheendof2018.

13. OutcomeOutput Deliver10languagesin Confluencebyendof 2018 Increaseusersatisfactionby20% withournon-Englishspeaking customersbytheendof2018.

14. Measuring the right thing Goals, Signals, and Metrics Learnings Agenda

15. Measuring the right thing Goals, Signals, and Metrics Learnings Agenda

16. How do you choose the right metric?

17. What does feature success look like?

18. What does feature success look like? user

19. What does user success look like?

20. How can we measure that? What does user success look like?

21. Agenda Measuring the right thing Goals, Signals, and Metrics Learnings

22. Goals Signals Metrics

23. Goals Signals Metrics What does user success look like?

24. Goals Signals Metrics How can we measure that? What does user success look like?

26. Goals are what success looks like for your user Goals Signals Metrics

27. Goals are what success looks like for your user Goals Signals Metrics E.g. Give a useful talk at Atlas Camp

28. Goals are what success looks like for your user Goals Signals Metrics E.g. Give a useful talk at Atlas Camp

29. Goals are what success looks like for your user Signals are predictors of the associated goals Goals Signals Metrics E.g. Give a useful talk at Atlas Camp

30. Goals are what success looks like for your user Signals are predictors of the associated goals Goals Signals Metrics E.g. Give a useful talk at Atlas Camp E.g. Audience is smiling and nodding

31. Goals are what success looks like for your user Signals are predictors of the associated goals Metrics are measurable & define what success looks like Goals Signals Metrics E.g. Give a useful talk at Atlas Camp E.g. Audience is smiling and nodding

32. Goals are what success looks like for your user Signals are predictors of the associated goals Metrics are measurable & define what success looks like Goals Signals Metrics E.g. Give a useful talk at Atlas Camp E.g. Audience is smiling and nodding E.g. 4.5+ session approval rating

33. GOAL

34. Deliver10languagesin Confluencebytheendof2018. GOAL

35. Goals Goals are what success looks like for your user

36. GSM Play Ingredients

37. Post-its Sharpies Whiteboard Markers GSM Play Ingredients

38. Setting Goals

39. Individual brainstorming Setting Goals

40. Individual brainstorming Shareback & grouping Setting Goals

41. Individual brainstorming Shareback & grouping Discussion & prioritisation Setting Goals

42. Goals Ensure non-English teams can collaborate and get work done with Confluence

43. Goals Signals Ensure non-English teams can collaborate and get work done with Confluence Signals are predictors of the associated goals

44. Individual brainstorming Shareback & grouping Discussion & prioritisation Setting Signals

45. Goals Signals Ensure non-English teams can collaborate and get work done with Confluence

46. Goals Signals Ensure non-English teams can collaborate and get work done with Confluence Users working with a non- English language, use the translations.

47. Goals Signals Ensure non-English teams can collaborate and get work done with Confluence Users working with a non- English language, use the translations.

48. Goals Signals Ensure non-English teams can collaborate and get work done with Confluence Users working with a non- English language, use the translations. Users do not revert back to English.

49. Goals Signals Metrics Ensure non-English teams can collaborate and get work done with Confluence Metrics are measurable & define what success looks like Users working with a non-English language, use the translations. Users do not revert back to English.

50. Individual brainstorming Shareback & grouping Discussion & prioritisation Setting Metrics

51. Goals Signals Metrics Ensure non-English teams can collaborate and get work done with Confluence Users working with a non-English language, use the translations. Users do not revert back to English.

52. Goals Signals Metrics Ensure non-English teams can collaborate and get work done with Confluence Ratio of users in same language as browser is greater than 30%. Users working with a non-English language, use the translations. Users do not revert back to English.

53. Goals Signals Metrics Ensure non-English teams can collaborate and get work done with Confluence Ratio of users in same language as browser is greater than 30%. 50% of users that ‘switch' to a translation do not revert to English within 6 months. Users working with a non-English language, use the translations. Users do not revert back to English.

54. MetricsSignals

55. MetricsSignals Show us whether we’re on the right track

56. MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful

57. MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful

58. SHORT TERM SUCCESS MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful

59. SHORT TERM SUCCESS LONG TERM SUCCESS MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful

60. SHORT TERM SUCCESS LONG TERM SUCCESS MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful E.g. Audience is smiling and nodding

61. SHORT TERM SUCCESS LONG TERM SUCCESS MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful E.g. Audience is smiling and nodding E.g. 4.5+ session approval rating

62. FEATURE DRIVEN GOAL

63. Deliver10languagesin Confluencebytheendof2018. FEATURE DRIVEN GOAL

64. USER DRIVEN GOAL Ensurethatnon-English speakingteamscancollaborate andgetworkdonewith Confluence

65. MetricsGoals Signals

66. Metrics Goals Signals

68. Jira Email Notifications

71. Goals Signals Metrics ITERATION 1

72. Goals Signals Metrics Users are notified for things they care about ITERATION 1

73. Goals Signals Metrics Users are notified for things they care about ITERATION 1

74. Goals Signals Metrics Users are notified for things they care about Users are opening their notifications ITERATION 1

75. Goals Signals Metrics Users are notified for things they care about Users are opening their notifications ITERATION 1

76. Goals Signals Metrics Users are notified for things they care about 20% increase in users clicking through to links from a notification Users are opening their notifications ITERATION 1

77. Goals Signals Metrics Users are notified for things they care about Users are opening their notifications 20% increase in users clicking through to links from a notification ITERATION 1

78. Goals Signals Metrics Users are notified for things they care about Users are opening their notifications 20% increase in users clicking through to links from a notification ITERATION 1

79. Informative

80. ActionableInformative

81. ActionableInformative

82. ActionableInformative Notifications that are valuable to users by delivering information

83. ActionableInformative Notifications that are valuable to users by delivering information Issue was transitioned to ‘Done’

84. ActionableInformative Notifications that are valuable to users by delivering information Issue was transitioned to ‘Done’

85. ActionableInformative Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’

86. ActionableInformative Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’ You were mentioned on a comment

87. ActionableInformative Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’ You were mentioned on a comment

88. ActionableInformative NOTIFICATION WAS OPENED Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’ You were mentioned on a comment

89. ActionableInformative NOTIFICATION WAS OPENED NOTIFICATION WAS CLICKED Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’ You were mentioned on a comment

90. Goals Signals Metrics Users are notified for things they care about 20% increase in users clicking through to links from a notification Users are opening their notifications ITERATION 1

91. ITERATION 1 Goals Signals Metrics Users are notified for things they care about 20% increase in users clicking through to links from a notification Users are opening their notifications

92. ITERATION 1 Goals Signals Metrics Users are notified for things they care about 20% increase in users clicking through to links from a notification ITERATION 2 Users are opening their notifications

93. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users clicking through to links from a notification Users are clicking through from their actionable notifications ITERATION 2

94. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications Users are clicking through from their actionable notifications 20% increase in users opening informative 20% increase in users clicking through an actionable notification ITERATION 2

95. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users clicking through an actionable notification ITERATION 2

96. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users clicking through an actionable notification ITERATION 2

97. ActionableInformative NOTIFICATION WAS OPENED NOTIFICATION WAS CLICKED Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’ You were mentioned on an Issue Issue was shared with you

98. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you

99. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you

100. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you ACTIONED

102. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you ACTIONED Notification Product Action

104. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you Comments on an issue Likes a comment Transitions an issue Edits an issue Dwells on an issue (30 sec) ACTIONED

105. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users clicking through to links from a notification ITERATION 2

106. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users clicking through to links from a notification ITERATION 2

107. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users clicking through to links from a notification ITERATION 2ITERATION 3

108. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users actioning a notification ITERATION 3

111. Agenda Measuring the right thing Goals, Signals, and Metrics Learnings

112. Learnings Understand your tradeoffs Counter metrics Avoid x% Be specific Qualitative Keep the balance

113. Learnings Understand your tradeoffs Metric: Reduce email volume by x% Counter metrics Avoid x% Be specific Qualitative Keep the balance

114. Learnings Understand your tradeoffs Metric: Reduce email volume by x% Counter metric: 0% decrease in Jira MAU Counter metrics Avoid x% Be specific Qualitative Keep the balance

115. Learnings Set a target for your team Counter metrics Avoid x% Be specific Qualitative Keep the balance

116. Learnings Set a target for your team Counter metrics Avoid x% Be specific Qualitative Keep the balance

117. Learnings Set a target for your team What is the minimum % that you would be disappointed with? Counter metrics Avoid x% Be specific Qualitative Keep the balance

118. Learnings Set a target for your team Countermetric:0%decreaseinJiraMAU Metric:Reduceemailvolumeby20%x% Counter metrics Avoid x% Be specific Qualitative Keep the balance

119. Learnings Set a target for your team Metric:Reduceemailvolumeby20% Countermetric:0%decreaseinJiraMAU Counter metrics Avoid x% Be specific Qualitative Keep the balance

120. Learnings Be clear with what you're setting out to achieve Counter metrics Avoid x% Be specific Qualitative Keep the balance

121. Learnings Be clear with what you're setting out to achieve Metric: increase email open rates by 20% Counter metrics Avoid x% Be specific Qualitative Keep the balance

124. Learnings Be clear with what you're setting out to achieve Relative: 20%→24% Absolute: 20%→40% Metric: increase email open rates by 20% Counter metrics Avoid x% Be specific Qualitative Keep the balance

125. Learnings Counter metrics Avoid x% Be specific Qualitative Pair quantitative metrics with qualitative analysis Keep the balance

126. Learnings Counter metrics Avoid x% Be specific Qualitative Pair quantitative metrics with qualitative analysis Quantitative: Tellsyouwhat’shappening Qualitative: Allowsyoutounderstandwhy Keep the balance

127. Learnings Balance product execution vs setting the perfect metric Counter metrics Avoid x% Be specific Qualitative Keep the balance

128. Takeaways

129. 1 Start by determining user success Takeaways

130. What does user success look like?

131. How can we measure that? What does user success look like?

132. 1 Start by determining user success Takeaways

133. 1 2 Start by determining user success Use a framework & iterate Takeaways

134. http://go.atlassian.com/goalssignalsmetrics

135. http://go.atlassian.com/goalssignalsmetrics http://go.atlassian.com/goalssignalsmetrics

136. 1 2 Start by determining user success Use a framework & iterate Takeaways

137. 1 3 2 Start by determining user success Use a framework & iterate Set a target & be specific Takeaways

138. 1 3 2 Start by determining user success Use a framework & iterate Set a target & be specific Takeaways

139. 1 4 3 2 Start by determining user success Use a framework & iterate Set a target & be specific Balance execution & learning with setting the perfect metric Takeaways

140. Thank you! JOSEPHINE LEE | PRODUCT MANAGER | ATLASSIAN

Nailing Measurement: a Framework for Measuring Metrics that Matter

Du liest eine Vorschau.

Hier ansehen

Nailing Measurement: a Framework for Measuring Metrics that Matter

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Nailing Measurement: a Framework for Measuring Metrics that Matter (20)

Weitere von Atlassian (20)

Aktuellste (20)