“Every line is the perfect length if you don't measure it.”
- Marty Rubin
So your organization has embarked upon a transformation to be more nimble and responsive by employing the latest tools and thinking in the Agile and DevOps arena. In this transformational context, how do you know that your initiatives are effective? Empirical measurements should provide insights on business value flow and delivery efficiency, allowing teams and organizations to see how they are progressing toward achieving their goals, but all too often we find ourselves mired in measurement traps that don't quite provide the right guidance in steering our efforts.
Rooted in contemporary thinking and tested in practice, this talk explores the principles of good measurement, what to measure, what not to measure, and enumerates some key metrics to help guide and inform our Agile and DevOps efforts. If done right, metrics can present a true picture of performance, and any progression, digression of these metrics can drive learning and improvement.
4. What is Measurement?
Measures
• I have 5 apples
• # incidents Metrics
• I have 5 more
apples than
yesterday
• % of Sev1 incidents
since feature
rollout
ObservaSon that reduces uncertainty…
4
10. 1. Business Success
• Starts with Measurable Goals
• Indicators for success
• Market share
• New business service enablement
10
11. 2. Customer/User Success
• Frequency of key
transacSons
• Amount of Sme spent in
the applicaSon
• User saSsfacSon survey
results
• A/B test results
• Customer Scket volume
www.glasbergen.com
11
12. 3. Operations Success
• UpSme (availability)
• Performance (response
Sme)
• Resource uSlizaSon
• Database query Smes
• Mean Sme to detecSon
• Support
12
13. 4. Development Success
• Lead Sme for changes
(from development to
deployment)
• Deployment frequency
• % Failed deployments
• Incident severity
• Outstanding defects
14. • Average Revenue
per User
• Customer Lifetime
Value
• Daily/Monthly
Active Users
• Average Session
Time
• % Change in
Customer Volume
• Feature Metrics
• Development
Lead Time
• Idle Time
• Cycle Time
• Work in
Progress-
Technical Debt
• Rework
• Idle Time
• MTTD
• Defects
• Deployment
Lead Time
• MTTR
• % of Failed
Deployments
• Deployment
frequency,
duration
• MTTR
• Performance
- Response
Time
• Availability
• Security Pass
Rate
• Customer Ticket
Volume
• Net Promoter
Score
• Net Value Score
Deploy
Business
Success
QA
Development
Release
Customer
Success
Operate
• Release
Frequency
• Time/Cost
per Release
• Predictability
Accelerate Value Delivery
Balance speed, risk, quality & costReduce time to obtain/respond to customer
Now, All Together
14
OperaSons Success Development Success
15. Monitoring Framework
• Aggregate metrics from
all success contexts
• See the enSre system
• Enables visualizaSon,
anomaly detecSon,
trending, alerSng…
Source Adaptation: Turnbull, The Art of Monitoring, Kindle edition, chap. 2.
Business
Success
Customer
Success
Development
Success
Operations
Success
15
Event Aggregator
17. 16 11 19 20 3 8 4 19 6 2
88
94
99
109
118 121
128
134
138 138 138 138
16
27
46
66 69
77
81
100
106 108
0
20
40
60
80
100
120
140
160
78 (6/8-6/21) 79 (6/22-7/5) 80 (7/6-7/19) 81 (7/20-8/2) 82 (8/3-8/16) 83 (8/17-8/30) 84 (8/31-9/13) 85 (9/14-9/27) 86 (9/28-10/11) 87
(10/12-10/25)
88 (10/26-11/8)89 (11/9-11/22)
#ofStories!
Sprints !
Stories Completed Per Sprint Scope (Total Stories Planned) Total Completed (Cumulative)
Projected (Optimistic) Projected (Pessimistic) Projected (Median)
Empirical Data Drives Enquiry and Adjustment
17
Q: When are all planned stories estimated to be completed?
A: End of Sprint 90 (12/06/16)
Q: What is the total number of stories estimated to be
completed by 11/08/2016?
A: 119 Stories based on average velocity
Q: Can we deliver all current planned stories by 10/16/2016?
A: Not Likely, given the current trend
Fixed Scope
Fixed Date
Fixed Scope
& Date
