Point of View on Developing Software Project Cost Contingencies

1. D. B. Skillern – Delivery Excellence Manager
2010 February 14
Point of View on
Developing Software Project Cost Contingencies
© 2010 IBM Corporation

2. Over a quarter of project failures are due to budgeting and planning errors
Analysis completed by a variety of experts as well as anecdotal experience has shown that the typical
project cost contingencies are inadequate. For budgeting purposes, the authorized project cost for most
software development projects include some element of a cost contingency. The typical contingency is
based on internal policy and approval guidelines. Experience has noted that the range of variance to a
single project estimate on any project at the early stages of the development life cycle (e.g., feasibility and
requirements definition) can lie in a wide range well beyond the typical contingency. The statistical range
for variance from the initial estimate of a software project cost ranges from 60 percent higher than the
estimate to 30 percent lower than the original estimate. The large range of differences between estimated
costs and actual costs for IT projects was first identified in a study done by Barry W. Boehm, in 1981
“Software Engineering Economics1”. This analysis has been reviewed many times by other experts both
in the software industry and academia since publication. IBM has confirmed and further extended the
findings of Mr. Boehm’s analysis.
In recent studies done by The Gartner Group, project failures range from 20% to 40% of all IT projects
depending on size1. Even at companies that aren’t among the top 25% of technology users, three out of
ten IT projects [on average] fail. Translation: an amazing 30% experience project failures.2 Over a quarter
of these failures were due to significant budgeting and planning errors.
2 © 2010 IBM Corporation

3. Typical cost contingencies are often inadequate
The typical budget for an IT project includes cost contingency. The cost contingency is based on internal
guidelines. The guidelines usually limit the contingency to 10% to 15% of the budgeted cost estimate.
The percentage contingency factor is designed to limit cost over-runs to minor scope changes and
estimate differences, before supplemental approval is required.
Based on academic analysis and practical experience, this level of cost contingency is often inadequate
to absorb large estimate variances that occur. However, there are approaches that can limit the risk of
cost over-runs. These approaches include use of a packaged software solution (aka, Commercial Off-
the-Shelf Software), application of proper estimation techniques, installation of a sound project
management control system and deployment of a skilled project manager. While these approaches can
reduce the amount of contingency needed, the requirement for cost contingency is not eliminated. The
following describes approaches to improve cost contingency estimation and reduce potential over-runs to
the budget project cost estimate.
3 © 2010 IBM Corporation

4. Realize that all software project estimates are approximations
All estimates of software development costs are approximations. The estimate is an educated guess of
what might happen in the future, expressed with a stated degree of confidence (i.e., potential variance),
for a specific scope of work. The degree of confidence that the estimator has in the estimate, as
expressed in the variance, depends on the techniques used in the estimate process and the knowledge of
what is required. The variance around the cost estimate measures the potential range of results that can
be expected. Basically, this is how much actual results can differ from estimated results. The degree of
confidence in the accuracy of a particular estimate (and potential estimate variance) depends on the
amount of uncertainty existing when the estimate is developed. Typically, early in a project life-cycle,
considerable uncertainty exists around the requirements for the project. Extensive analysis has been
completed regarding the potential range of variance for a software development project over the
development life-cycle. The analysis indicates that while there may be different statistical distributions of
variance around the estimate, a variance, none-the-less, will exist. There are certain actions that can be
taken to mitigate the impact of the variance. The actions involve the project management control system
deployed, the estimation techniques used, the project manager assigned and the cost contingency
maintained. The following discusses estimate variance in software development projects and the possible
actions that can be taken to mitigate the impact of such variance.
4 © 2010 IBM Corporation

5. Understand that estimate differences will occur
Estimation accuracy in software development is a problem that vexes every project and program
manager. The topic has been studied extensively. Different models have been developed for software
development estimation – ranging from statistically based mathematical calculations to judgment based
analogies. Each type of estimation technique has inherent limitations. The various studies done on the
accuracy of software development cost estimation have reached similar fundamental conclusions.
Regardless of technique used to develop the estimate, a variance will exist for the estimated effort and
cost of the project. The variance range will narrow depending on the knowledge of what is needed and
how the work is completed. Essentially, estimate accuracy and confidence will improve as more is known
and uncertainty is reduced.
Customer Acceptance Defects by Effort Hours Development Hours Required By IT Organization CMMI Level
250 9000
8000
200
7000
CMMI level 1
6000
150
5000
4000
100
3000
CMMI level 4
2000
50
1000
NB Shows Linear and Order 2
polynomial trend lines.
0 0
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
0 2000 4000 6000 8000 10000 12000 14000 16000
Source: Software Engineering. A Practitioners Approach, Roger S.
5 Source: IBM Analysis, 2008. Pressman, Pressman, ISBN 0-07-707936-1. © 2010 IBM Corporation

6. Recognize that the cost differences (variance) can be large
The original analysis of software development cost variance was completed by Mr. Barry W. Boehm in
19813. The analysis was based on certain Department of Defense information. The analysis considered
software project cost estimates as a function of the life-cycle phase of the project. This function has been
empirically proven in further completed by Mr. Luiz A. Laranjeira, in “Software Estimation of Object-
Oriented Systems4”. The work has also been reviewed and validated by various other academicians and
practicing consultants, including IBM since its initial release. Historical experience with software
development cost variances indicate that larger variances exist for software development efforts and
smaller variances will occur for simple package software installations. The software development cost
variance analysis (based on custom development) are depicted below:
3,4
6 © 2010 IBM Corporation

7. Be thorough in building the original cost estimate
In addition to the complexity of the project driving the need for a higher contingency to address estimate
variance, the quality of the original estimate will impact the amount of cost contingency required.
Accordingly, the contingency for high quality estimate will be lower than the contingency on a quick back
of the envelope number, so it is critical that the estimate process be thorough. There are an number of
simple rules5 that can be applied to improve the estimate quality. Some examples of these rules follow:
– Use two or more estimating approaches, viewing the problem from different angles improves the accuracy
and the confidence in the estimate. Three example methods are: the analogy, metrics based and duration
based models (with time constraints). There are inherent limitations in all estimation approaches, but the
limitations associated with each method can be understood and considered in the process.
– Examine the requirements and translate into a scope of work within a given solution with specific
measurements. The scope can be quantified to reflect the effort or cost drivers that influence work to be
done. Typically such quantification involves 'counts' of work efforts: number of prototype sessions, number
of core business processes, number of reports, number of interfaces, etc.
– Use an estimate model when possible, software packages can be purchased to complete complex estimate
development for custom software projects (e.g., COCOMO), but for a estimating the effort to implement an
accounting software package (e.g., Great Plains) a spreadsheet will suffice.
– Use the expertise of person experienced in developing the particular software in the project. Experience
build on many projects will improve the quality of the estimate. In software development, the best predictor
of future performance is how it was done in the past. Supporting an estimate with past metrics removes
subjectivity. Also recognize that estimates developed by software sales people may be biased due to their
desire to make a sale software sale.
– Recognize what you don’t know. Identify the key assumptions and risks factors associated with the project.
These elements represent elements to be controlled to minimize the variance to the cost estimate.
7 © 2010 IBM Corporation

8. Understand what do know and how you can control it
What you don’t know about project definition (requirements, complexities, scope, resource skills, etc.)
represents uncertainty that drives estimate variances. What you know you should control. What you don’t
know represents risks you should mitigate.
Recent analysis completed by Mr. Murray Cantor, an IBM Distinguished Engineer, has indicated that there
may be multiple probability distributions impacting project estimates. Mr. Cantor’s work, summarized in an
IBM technique paper6, generally validated the work of Mr. Boehm. Essentially, at the outset of a project,
very little is known for sure – the effort duration and expected cost are, at best, educated guesses. In
mathematical terms, the factors and values of an estimate are made up of known quantities (such the
software product to be used) and random variables (such as the changes need to the software product to
meet requirements). The cost of a project will vary depending on the number of random variables and the
possible values of these random variables. The analysis of Mr. Cantor supports the position that certainty
increases around these project parameters as time passes. The amount of time to complete certain tasks
becomes known, so the estimate variance (and range in the probability distribution) for the estimate and
required contingency decreases. The analysis suggests that many different statistic distributions can exist
for the project parameters (normal-bell curve, log-normal, triangular, etc.). Even though many statistical
distributions exist depending on the specific facts and circumstances of a project, it is critical to recognize
that project cost variance will exist and techniques need to be applied to control the variance. Developing
a cost contingency recognizes the impact of limitations to mitigating risks that drive estimated cost
variances.
8 © 2010 IBM Corporation

9. Identify a realistic cost contingency
Build a cost contingency for a project that recognizes the Risk Drivers
risks associated with the project. Realize that the size of Organization Change Level
Organization Priority 80 Package Softw are Custom izations
this contingency may be larger than the typical business Technology Platform 70 Softw are Integration
control contingency percentage. The business control User Readiness
60
50
Staff Capabilities
contingency cap approach is designed to avoid the need SME Availability
40
30
Project Manager
for supplemental approvals for minor variances and Data Quality
20
10 Project Managem ent System
properly manage funding requirements. The alternative 0
Decision Making Speed Technical Com plexity
to managing with a larger cost contingency is to
supplement with additional authorized costs as cost
over-runs when risks become realized project issues. Leadership Support Size and Breadth
Either approach requires proper expectations Tim eline Constaints
External Requirem ents
management. Softw are Developm ent Maturity
In developing the right contingency for a project, the risk
profile should be considered. The potential risks should be scaled to model the potential risk contingency
required. The scaling depends on risk assessment (probability of occurrence, impact of risk, etc.) for a
particular effort and how well these risks can be mitigated. After the risk contingency estimate is determined,
evaluate the business case with the worst case risk scenario to determine what is acceptable.
The risk contingency can also be measured in mathematically terms through calculation of variance using
more complex estimation techniques such as the PERT technique or the Delphi method. The final amount of
contingency maintained for any project is a matter of judgment and philosophy as well as the internal
practices of the organization involved. The critical factor is to recognize that a contingency will be needed for
any project effort. Based the contingency on known risks and minimize the use of that contingency with an
effective project management control system.
9 © 2010 IBM Corporation

10. Understand that a sound project management control system reduces
inherent uncertainty that a savvy project manager can not do alone
Project cost contingencies will be required to address project risks. Project Management Success
Effectively managed projects tend to face lower variances to the original
cost estimate (and use less contingency) for a project. Effective project Highly
Skilled
management execution depends on both:
Possible Highly
- the skills of the project/program management team and Success Probable
Success
Manager
Project
Skills
- the quality of the project management control system.
A sound Project Management Control System (PMCS) represents the
tools and techniques to manage and mitigate risks that drive cost Success Possible
estimate variance. The risks for any project arise in a number of areas. Unlikely Success
One area of risk is the program will not achieve the cost and schedule Limited
Experience
targets. A typical PMCS includes certain control activities focused on
Poor Quality High Quality
these objectives. Example PMCS activities include Project Governance,
Work Plan Management and Risk Mitigation provide tasks to reduce Project Management Control System
uncertainty and cost variance. Structure
The key to reducing the cost variances is to reduce the uncertainty. “Even if, at the onset of the project, the initial
variance is high, the project could be managed so that early in the project, the uncertainties are removed so that
the variance is low. In fact, the key to project success is removing the uncertainty as soon as possible, resulting
in the increased ability to make accurate and aggressive plans. However, since the ability to reduce variance is
based on knowledge gained through project experience,”6 only in the most simple projects can the uncertainties
be removed quickly. Control what you can and plan for what you can’t control. Plan for the uncertainty inherent
to software development and implementation projects. Building a quality project cost estimate supported by
realistic project contingencies, a sound project management controls and a qualified project manager are proven
steps to address uncertainty and the risks of project failure.
10 © 2010 IBM Corporation

11. Notes:
1
Gartner Group Studies, including “The User’s view of Why IT Projects Fail”, 2003, 2004, and 2005.
2
“Survey shows common IT woes”, Julia King, Computerworld, June 2003
3
Software Engineering Economics, Barry W. Boehm, Prentice Hall PTR, 1981
4
“Software Size Estimation of Object-Oriented Systems”, IEE Transactions on Software Engineering, Vol. 16, No.5, May 1990
5
“Estimating – Top Ten Best Practices”, Estimating at IBM, October 27, 2009
6
”Estimate variance and governance”, IBM Developer Works, March 15, 2006
11 © 2010 IBM Corporation