This presentation works through important questions that people have about metrics on agile teams, principles around how to be effective with your agile metrics strategy, how to measure agile teams following a lightweight approach to GQM, potential metrics to collect about agile teams, and how to support IT governance through effective metrics rollups.
Optimizing AI for immediate response in Smart CCTV
Â
Measuring Agile: A Disciplined Approach To Metrics
1. Š Disciplined Agile Consortium 1
Measuring Agile
A Disciplined Agile
Approach to Metrics
2. Agenda
â˘âŻ Important questions
â˘âŻ Principles
â˘âŻ Measuring agile teams
â˘âŻ Potential metrics
â˘âŻ Supporting IT governance
â˘âŻ Parting thoughts
â˘âŻ Questions
Š Disciplined Agile Consortium 2
3. Important Questions
â˘âŻ How do you measure solution delivery
teams?
â˘âŻ What metrics should we collect?
â˘âŻ How do you roll up team measures to
the portfolio level?
â˘âŻ What metrics are right for you?
â˘âŻ How do agile and traditional metrics fit
together?
â˘âŻ What are the success factors for metrics
programs?
â˘âŻ What are the failure factors?
â˘âŻ How does this fit into your overall
governance strategy?
Š Disciplined Agile Consortium 3
14. Principle: Adopt Common Metrics Categories
Across Teams
Š Disciplined Agile Consortium 14
Time to
Market
Financial
Stakeholder
Satisfaction
Quality
Staff
Morale
Transform
21. Goal Question Metric (GQM)
A lightweight GQM process:
1.⯠Identify improvement goals
2.⯠Generate questions that define those goals as completely as possible in
a quantifiable way
3.⯠Identify potential measures needed to be collected to answer those
questions
4.⯠Introduce mechanisms for data collection
5.⯠Collect, validate and analyze the data in real time to provide feedback to
teams for corrective action
Similar to Googleâs OKR (Objectives and Key Results)
Š Disciplined Agile Consortium 21
22. â˘âŻ Some goals are qualitative:
â⯠Invest in IT wisely
â⯠Improve staff morale
â⯠Improve stakeholder satisfaction with delivered solutions
â˘âŻ Some goals are quantitative:
â⯠Decrease cycle time by 20% this quarter
â⯠Reduce technical debt by 10% by year end
â⯠Get our net promoter score (NPS) to 60% by quarter end
â˘âŻ Our advice:
â⯠Short time frames (less than three months) work well
â⯠Quantitative goals are generally better than qualitative
Š Disciplined Agile Consortium 22
Identify Improvement Goals
24. Goal: Invest in IT Wisely
Questions
1.⯠How effective is the
investment in IT activities?
2.⯠What future spending do we
require?
3.⯠Is productivity increasing?
Potential Metrics
1.⯠Value delivered, net present
value (NPV), return on
investment (ROI)
2.⯠Effort/cost projection (burn
ups/burn downs)
3.⯠Acceleration, business value
delivered, velocity (trend)
Š Disciplined Agile Consortium 24
25. Goal: Decrease cycle time by 20% this
quarter
Questions
1.⯠Is the team working at a
sufficient pace?
2.⯠Is the team working together
effectively?
3.⯠Are changing requirements
putting the release date at
risk?
Potential Metrics
1.⯠Release burndown, ranged
release burndown
2.⯠Team morale, stakeholder
satisfaction, blocking work
items
3.⯠Age of items, ranged release
burndown
Š Disciplined Agile Consortium 25
26. Goal: Reduce Technical Debt by 10% by Year
End
Questions
1.⯠What is our current level of
technical debt?
2.⯠Are we improving quality over
time?
Potential Metrics
1.⯠Code quality, defect density
2.⯠Build health, defect density,
test coverage, code quality
Š Disciplined Agile Consortium 26
28. Iteration Burndown
â˘âŻ Indicates the likelihood of
delivering as promised this
iteration
â˘âŻ Application:
â⯠By team to manage their
time
â˘âŻ Advantages:
â⯠Helps to avoid surprises at
end of iteration
â⯠Motivates a continuous
and sustainable pace
â˘âŻ Disadvantages:
â⯠Overhead to update
estimated work remaining
Š Disciplined Agile Consortium 28
29. Velocity
â˘âŻ Velocity is a measure of the teamâs delivery
capacity for each iteration measured in âpointsâ
â˘âŻ Options:
â⯠Gross velocity: Number of points delivered in an
iteration
â⯠Net velocity: Delta in the number of points on the
work item list
â⯠Smoothed velocity: Velocity averaged over several
iterations
â˘âŻ Application:
â⯠By team to extrapolate the schedule. e.g. If total
points in Work Item List is 400, with velocity of 40
there is 10 iterations left
â˘âŻ Advantages:
â⯠Simple way to project schedule/cost
â˘âŻ Disadvantage:
â⯠Easy to manipulate because points are manually
identified
â⯠Not easily comparable across teams
Š Disciplined Agile Consor1um 29
32. Ranged Release Burndown
â˘âŻ A ranged estimate of number of iteration required to complete work
â˘âŻ Range of uncertainty decreases over time
â˘âŻ BUT⌠many people donât like ranged estimates
Š Disciplined Agile Consortium 32
34. Acceleration
â˘âŻ The change in velocity over time
â˘âŻ Application:
â⯠Use to calculate productivity improvement (acceleration * number of
people on team * charge rate)
â˘âŻ Advantages:
â⯠Comparable across teams
â⯠Easy to calculate
â˘âŻ Disadvantages:
â⯠Easy to manipulate because
points are manually identified
Š Disciplined Agile Consortium 34
0
5
10
15
20
25
30
Velocity
Iterations
35. Value Delivered â Value points
â˘âŻ Assign âvalue pointsâ to a story that captures the
estimated financial value of the delivered
functionality.
â˘âŻ Application:
â⯠Estimate the value delivered by a team
â˘âŻ Advantage:
â⯠Provides insight into the value being produced by a
team
â˘âŻ Disadvantage:
â⯠Easy to game as itâs a manual estimation
â⯠Doesnât take cost of delay into account (without re-
estimating)
Š Disciplined Agile Consortium 35
36. Code Quality Metrics
â˘âŻ A category covering a very wide range of metrics that provide insight
into how well source code has been written.
â˘âŻ Options
â⯠Halstead, Cyclomatic complexity, Defect density, Coupling, and many
more
â˘âŻ Applications:
â⯠Determine the existing quality of code
â⯠Pinpoint potential code quality problems
â˘âŻ Advantages:
â⯠Easy to automate
â⯠Many commercial and open source tools
â˘âŻ Disadvantage:
â⯠Easy to get overwhelmed by the data
Š Disciplined Agile Consortium 36
37. Defect Trends
â˘âŻ Track when and where defects are
found against when they are
addressed
â˘âŻ Options:
â⯠By severity
â⯠By source
â˘âŻ Application:
â⯠Monitor defects/bugs that escape the
delivery team
â˘âŻ Advantage:
â⯠Motivates teams to fix defects
â⯠Can provide insight into where you
need to improve your quality strategy
â˘âŻ Disadvantage:
â⯠Very easy to game because defects
are often identified manually
Š Disciplined Agile Consortium 37
38. Test/Code Coverage
â˘âŻ Calculated as the number of lines invoked by tests divided
by the total number of lines of relevant code
â˘âŻ Application:
â⯠Determine how much of your code base has automated tests
â˘âŻ Advantages:
â⯠Easy to automate
â⯠Low test coverage tells you that you need to automate more of
your tests
â˘âŻ Disadvantage:
â⯠High test coverage may not tell you much because this metric
is easy to game with low-quality tests
Š Disciplined Agile Consortium 38
39. Build Status
â˘âŻ Monitor the status of the build
â˘âŻ Options:
â⯠Trend
â⯠Current status
â˘âŻ Advantage:
â⯠Trend report useful to teams new to
continuous integration (CI)
â⯠Very useful indicator if build broken
for long time
â˘âŻ Disadvantage:
â⯠Once you have implemented CI
effectively the trend metric loses
value
Š Disciplined Agile Consortium 39
40. Net Promoter Score (NPS)
â˘âŻ Net Promoter Score (NPS) measures the loyalty that exists between
a provider and a consumer
â˘âŻ Application:
â⯠Gauge the loyalty of your customers
â˘âŻ Advantage:
â⯠Indicates potential for revenue growth
â˘âŻ Disadvantage:
â⯠Manually generated and easy to âfixâ with enticements
Š Disciplined Agile Consortium 40
41. Resource Utilization
â˘âŻ Measure of the percentage of time that people
spend working. Typically calculated as
number of hours worked divided by number of
hours they should have worked
â˘âŻ Application:
â⯠Matrix organizations where some (or most)
people arenât âself startersâ who need to be
assigned to work
â˘âŻ Advantages:
â⯠Identify people available for work
â˘âŻ Disadvantages:
â⯠Motivates you to build teams out of whom you
have available instead of whom would be a
good fit
â⯠Increases stress and motivates people to fill in
their time instead of to add value
â⯠Often results in little or no slack time
Š Disciplined Agile Consortium 41
42. Skill Level
â˘âŻ People are rated against desired
skills on a scale of 1 to N
â˘âŻ Application:
â⯠Identify skills gaps
â⯠Justify training or coaching efforts
â˘âŻ Options:
â⯠Individual or Team
â⯠Self or external assessment
â˘âŻ Advantages:
â⯠Focuses training/coaching efforts
â˘âŻ Disadvantages:
â⯠Easy to game, particularly with self-
assessment
Š Disciplined Agile Consortium 42
43. Disciplined Agile Certifications
â˘âŻ Measure adoption rate of the DA certifications
across your organization
â˘âŻ Application:
â⯠Indicates skill and experience level of team
members
â˘âŻ Advantage:
â⯠Measured from outside so difficult to game
â⯠When combined with other metrics can provide
an accurate estimate of the value of your
investment in training and coaching
â⯠Focuses training and coaching efforts
â˘âŻ Disadvantage:
â⯠Need to invest in the training and coaching of
your staff over a long term basis
Š Disciplined Agile Consortium 43
â˘âŻ Knowledge
â˘âŻ Knowledge
+ 2+ years experience
â˘âŻ Knowledge
+ 5+ years experience
+ Give back
44. Lead and Cycle Time
â˘âŻ Measure the length of time to develop a feature from concept to
cash
â˘âŻ Application:
â⯠Identify places where you need to improve your process
â⯠Indication of time productivity: (Cycle Time-Internal Wait Time Within
Cycle)/Lead Time
â˘âŻ Advantages:
â⯠Measures time-to-market, a critical business success factor
â⯠Looks at overall process, not just development
â˘âŻ Disadvantage:
â⯠Can lead to finger pointing
Š Disciplined Agile Consortium 44
45. Cost of Delay
â˘âŻ A measure of the impact of time on the business outcomes we hope
to achieve
â˘âŻ Application:
â⯠Input into work (feature or project) prioritization
â˘âŻ Advantages:
â⯠Surfaces important financial considerations
â˘âŻ Disadvantages:
â⯠Easy to game as you need to guesstimate the value curve(s)
â⯠Requires understanding of basic financial concepts
Š Disciplined Agile Consortium 45
46. Cumulative Flow Diagram
â˘âŻ Visualize the work effort of a team over time
â˘âŻ Advantages:
â⯠Indicates potential bottlenecks in your process (bumps on the diagram)
â⯠Simple report commonly generated by agile management tools such as
JIRA, Rally, or VersionOne
â˘âŻ Disadvantage:
â⯠Easy to âfixâ because it shows
points of work, which are
manually guesstimated
Š Disciplined Agile Consortium 46
0
10
20
30
40
50
60
70
80
90
May
'16
June
'16
July
'16
Aug
'16
Sept
'16
Oct
'16
Backlog
In Progress
Done
48. IT Intelligence (Automated Dashboards)
â˘âŻ Automated dashboard that summarizes the status for all of IT
â˘âŻ Shows the entire portfolio:
â⯠Potential/suggested endeavors
â⯠Ongoing development endeavors
â⯠Operational solutions
â˘âŻ Enables drill down into details
Š Disciplined Agile Consortium 48
49. Rolling Up Metrics
Š Disciplined Agile Consortium 49
Portfolio
Team
â˘âŻ Measure against organizational goals
â˘âŻ Overall indicator or category indicators
â˘âŻ Organizational goals evolve slowly (~annually)
Program
â˘âŻ Measure against program goals
â˘âŻ Category indicators per sub-team
â˘âŻ Program goals evolve quarterly (or thereabouts)
â˘âŻ Measure against team goals
â˘âŻ Specific metrics per team
â˘âŻ Metrics evolve regularly as the situation evolves
â˘âŻ Unique dashboard per team
*
0..1
*
11
0..*
51. Metrics Principles
â˘âŻ There is no easy answer
â˘âŻ Every metric has strengths and weaknesses
â˘âŻ Compete against yourself
â˘âŻ Measure to improve
â˘âŻ You get what you measure
â˘âŻ Measure outcomes at the team level
â˘âŻ Each team needs a unique set of metrics
â˘âŻ Team use metrics to self organize
â˘âŻ Trust but verify
â˘âŻ Adopt common metrics categories across teams
â˘âŻ Donât manage to the metrics
â˘âŻ Automate wherever possible
â˘âŻ Prefer trends over scalars
â˘âŻ Prefer leading over trailing metrics
â˘âŻ Prefer pull over push
Š Disciplined Agile Consortium 51
57. Shuhari and Disciplined Agile Certification
At the shu stage you are beginning to
learn the techniques and philosophies of
disciplined agile development. Your
goal is to build a strong foundation from
which to build upon.
At the ha stage you reflect upon and
question why disciplined agile strategies
work, seeking to understand the range
of strategies available to you and
when they are best applied.
At the ri stage you seek to extend and
improve upon disciplined agile
techniques, sharing your learnings with
others.
Š Disciplined Agile Consortium 57
58. Scott Ambler + Associates is the thought leader behind the Disciplined
Agile Delivery (DAD) framework and its application. We are an IT
management consulting firm that advises organizations to be more
effective applying disciplined agile and lean processes within the
context of your business.
Our website is ScottAmbler.com
We can help
Š Disciplined Agile Consortium 58