Nowadays, as the software industry is slowly becoming more mature, software measurement and performance measurement are becoming increasingly important. Organizations need to know their productivity and competitiveness in software development projects for various reasons. In many software development contracts, targets are set for the suppliers to reach. These targets are based on software metrics like productivity, speed of delivery and software quality. In order to check if the targets are reached, it is necessary to measure the functional size of the software product that is delivered and also the functional size of the software development project that is carried out, as there is usually a difference between these two sizes. To be able to use functional size in contracts, it must be measured in an objective, repeatable, verifiable and therefore defensible way. That being the case, the industry’s best practice is to use an ISO/IEC standard for functional size measurement, e.g. Nesma, COSMIC or IFPUG function points. However, these methods only measure the functional user requirements from the total software requirements to be delivered. In activities like project estimation and productivity measurement, the influence of the non-functional requirements is expressed in the Project Delivery Rate (PDR) which is expressed in effort hours per function point. If more than the average amount of non-functional requirements need to be realized in a project (or more severe non-functional requirements), the PDR used should also be higher. In the industry it is customary to set productivity targets based on an average (or calibrated) influence of non-functional requirements and this works quite fine in traditional software projects. In software development projects that are executed in an agile way, this is not always the case. When working agile, there are forces that influence the traditional way of performance measurement significantly, resulting in a number of serious issues. In this paper these issues are explained and a method to overcome these issues is proposed.
2. |
Performance measurement
of agile teams
continuous development of a product –
performance measured by sprint
Cracow Poland, October 7 2015
2Performance measurement of agile teams
Harold van Heeringen, METRI
Theo Prins, Sogeti Nederland
Edwin van Gorp, Sogeti Nederland
3. |
Sizing, Estimating & Control
Sogeti department responsible for:
Functional size measurement: Nesma, IFPUG and COSMIC;
Estimating new projects, releases and contracts;
Project control based on software metrics;
Performance measurement;
Benchmarking;
Performance reporting.
This presentation is about one engagement where the SEC department
was asked to help solving the issues with regard to the productivity of a
Sogeti agile team in a government project…. measured per sprint!
Performance measurement of agile teams 3
4. |
Background
RfP of Dutch government agency in 2014
Agile development of a product, taken over from another supplier;
Performance measurement of the team per sprint;
Based on ISO/IEC norm, in this case Nesma function points;
Realistic targets set, based on Sogeti historical data, external
benchmark data and historical data of previous supplier!
Sogeti offered to measure performance based on Project Delivery
Rate (PDR), measured in project function points delivered per sprint;
Politics: Customer needs to prove the new supplier performs better!
Customer selects Sogeti because of proved historical performance
and specialized department Sizing, Estimating & Control.
4Performance measurement of agile teams
5. |
The problem
The application turned out to be very complex. Even the OTAP
environment was very complex to handle. The product backlog
consisted of many non-functional backlog items;
Functional size measurement is a powerful way to objectively size
progress when it comes to functionality added, changed, or deleted,
but not for measuring non-functional sprint backlog items processed by
the team;
The customer (product owner) decides which functional and non-
functional backlog items are put on the sprint backlog;
Traditional data and performance measurement methods turned out to
be ineffective when measuring performance on sprint level when more
than average non-functional backlog items are put on the sprint
backlog.
5Performance measurement of agile teams
6. |
Two types of agile projects
1. Development of a set of specific requirements, prioritized on a backlog,
realized in a specific duration by a specific team in a specific amount of
effort hours and cost. At one point, the project, and the product, is
finished. (in traditional terms: new development or release).
2. Continued development (evolving) an existing application. No definite
end goal when the project or product is finished. A year divided into X
sprints of Y weeks and a fixed team working to deliver the sprint backlog
items. This only ends when the organization decides maintenance is no
longer needed. (in traditional terms: maintenance).
This industry paper investigates Type number 2.
6Performance measurement of agile teams
7. |
First supplier
Big international system integrator;
Application developed from scratch;
About 15 sprints;
Sprints with fluctuating functional sizes in Nesma FP;
High complexity;
Average PDR about 17 hours/FP.
Almost all backlog items of functional nature!
However, the customer was not happy and decided to select a new
supplier.
7Performance measurement of agile teams
8. |
After transition
From Q3 2014 onwards, Sogeti took over;
Sprints of 3 weeks;
Sogeti team, product owner supplied by customer;
Product backlog contains many non-functional items:
Scrum team works hard, but hardly delivers function points;
PDR (h/FP) relatively high, not in line with target PDR;
Customer contract manager blames Sogeti for not being productive;
Contract under pressure, media attention, pressure and politic issue;
Sogeti department Sizing, Estimating & Control asked to analyze the
performance and to propose improvements.
Performance measurement of agile teams 8
9. |
Function points
Important to understand, using a functional size measurement method
(an ISO/IEC standard) means measuring the size of the functional user
requirements that are implemented in the software;
Non-functional requirements are not measured at all!
More non-functional work in a sprint means that less functionality is
realized, and therefore a higher PDR (hours/FP)/lower productivity.
In project estimation and benchmarking, the influence of NFR is
accounted for by the historical data used or the parametric model used,
or the peer group that is constructed based on projects with similar
characteristics.
Performance measurement of agile teams 9
10. |
Story points
Usual way to estimate effort in agile teams;
Team members assign story points to each backlog item, reflecting the
amount of work needed to realize the item;
Subjective, not repeatable, not verifiable and not defensible, but
mainstream practice in agile teams because of ease of use;
Story point-based metrics can not be compared with any measurements
or metrics outside the team;
Story point measurement is not a standard, but SP do take into account
the effort spent on non-functional backlog items.
Performance measurement of agile teams 10
11. |
Non functional backlog items are important
11Performance measurement of agile teams
Sprint X FP SP Effort hours
Backlog item 1 4 4 90
Backlog item 2 0 6 120
Backlog item 3 0 2 45
Backlog item 4 5 3 65
Backlog item 5 4 3 80
Total 13 18 400
PDR = 400/13 = 30,8 Hours/FP
Ratio F/NF SP backlog items: 10/8
13. |
Issue
Team spends a lot of time on non-functional backlog items;
PDR is not good enough to reach target;
Stakeholders don’t understand this ‘technical issue’ and only see metrics
on the dashboard PDR significant worse than expected;
But… customer product owner decided on the product backlog items to
put on the sprint backlog!
So, disappointing PDR mainly caused by the number of functional and
non-functional backlog items put in the sprint by the product owner (=
the customer!).
Sogeti SEC wishes to address this issue and to come up with a proposal
for a more accurate performance measurement method.
13Performance measurement of agile teams
14. |
SEC proposal
Agile Normalized Size (ANS)
Functional size that could have been realized if the product owner only had
put functional backlog items in the sprint backlog.
Based on this ANS, a PDR (hour/FP) can be determined that can be
compared to the PDR’s in the databases with historical data.
14Performance measurement of agile teams
15. |
Method
1. Measure the functional size of the realized functional backlog items with
a standard method (Nesma/IFPUG FPA, COSMIC, …);
2. Determine whether the realized backlog items are functional or non-
functional;
3. Determine the number of story points of the functional backlog items
realized in the sprint;
4. Determine the total number of story points realized in the sprint;
5. Determine the agile normalized size:
(functional size / # functional story points)
* total # story points
15Performance measurement of agile teams
17. |
The effect in multiple sprints
Sprint Size (FP) Functional SP Non-functional SP Total SP ANS (nFP)
16 20 32 12 44 27,5
17 25 28 16 44 39,3
18 18 24 20 44 33,0
19 29 35 4 39 32,3
20 4 6 36 42 28,0
21 15 16 24 40 37,5
17Performance measurement of agile teams
18. |
Advantages / disadvantages
Advantages
Reduced influence of non-functional backlog items;
The use of an ISO/IEC FSM standard – ability to benchmark.
Disadvantages
Depending on accurate story point assignment (subjective);
Possible for the team to tweak the performance figures;
As the product owner is present, this risk is considered to be small;
Impossible to measure ANS when the functional size delivered is 0.
Performance measurement of agile teams 18
23. |
Issue: completely non-functional sprints
In sprint 22, zero function points were delivered;
Size in FP is 0, ANS impossible to determine (dividing by zero);
Impossible to determine productivity.
Solution: progressive approach.
23Performance measurement of agile teams
24. |
Progressive approach
Size measurement and productivity measurement not per sprint, but until
the last sprint;
Does not focus on sprint, but on overall performance
(∑1-n functional size / ∑1-n functional story points)
* ∑1-n total story points
24Performance measurement of agile teams
27. |
Starting points
Documentation
After each sprint the functional documentation should be made up-to-
date and it must be clear:
Which functionality was added in the sprint;
Which functionality was changed in the sprint and in which way;
Which functionality was deleted in the sprint;
This should be part of the definition of done.
Effort administration
The effort hours need to be booked in the effort administration in such a
way that it is possible to clearly identify the effort hours in scope and out
of scope of the performance measurement.
Performance measurement of agile teams 27
28. |
Conclusions and recommendations
The productivity of an agile team in a contract can be measured and
benchmarked while taking into account the effect of non-functional
requirements;
The customer now understands that non-functional backlog items have
impact on the PDR when using only Nesma/IFPUG function points in agile
projects when measuring on a sprint level. Customer is able to explain
that internally and politically. Pressure is less now, because targets are
met.
The method can help other organizations as well!
Performance measurement of agile teams 28