The difference between expert estimates and parameteric estimates, and a study to see which ones are more accurate and faster to do. ISPA/SCEA conference, Brussels (May, 2012)
1. Estimate software projects
Faster, Cheaper and Better!
Can parametric estimating ‘beat’ the experts?
H.S. van Heeringen
@haroldveendam Brussels, May 2012
2. Overview
• Sogeti
• Expert vs. Parametric Estimates
• Chalenge for parametric estimates
• Estimating Wizard
• Study: Expert accuracy vs. Parametric accuracy
2
3. Sogeti
• Sogeti is the ‘local IT’ brand of Capgemini
• Over 20.000 people in 15 countries
− Belgium, Denmark, Finland, France, Germany, India,
Ireland, Luxemburg, Netherlands, Norway, Spain, Sweden,
Switzerland, UK and the US.
• Sogeti Nederland: ca. 3.000 people
• Local IT service provider:
− Close to the clients;
− Our mission is to be a leading provider of professional
technology services. Our vision is to be perceived as the
best supplier on each local market;
− Tmap (test), DYA (architecture), ViNT (research), etcetera.
3
4. Project Estimates
• Two types of project estimation:
− Expert estimation
− Parametric estimation
• Expert estimates
− Knowledge and experience of experts
− Assign effort hours to tasks (bottom-up)
− Subjective, but always applicable
• Parametric estimates
− Size measurement, historical data, CER’s and tooling
− Size measurement methods: NESMA FPA, COSMIC, IFPUG
− Objective, but well documented specifications required
4
5. Software Size measurement
• Software is hard to measure before the project
starts
• But size is usually the main cost driver
• Best practice: measure functionality requested
by the users
− NESMA / IFPUG / COSMIC function points
• Objective, verifiable, repeatable methods
• Although ISO standards, there is always an
inaccuracy margin
− Documentation is not complete or not detailed enough
− Measurers don’t have the required level of expertise
− During the projects, changes will happen
5
6. Comparing the two types
• Expert Estimates
− Bottom-up estimation
− Usually optimistic (up to 30% under estimation is common)
◦ Forgotten activities
◦ Hard to defend
◦ The expert is not going to do all the work
◦ The expert may not be an expert on the new project
• Parametric Estimates
− Top-down estimation
− Estimating & Performance Measurement process needed
− Effort = size * Productivity
◦ Size is objectively measureable (COSMIC, FPA)
◦ Productivity from historical data (organization / ISBSG)
6
8. Cost of high and low estimates
Non-linear extra costs
-Planning errors
-team enlargement more expensive, not faster
-Extra management attention / overhead
-Stress: More defects, lower maintainability !!
Linear extra kosten
Extra hours will be used
8
9. Sogeti AS SEC
• Division Application Services
− projects and maintenance (fixed price/ fixed date)
• CoE Sizing, Estimating & Control (SEC)
− Parametric estimation, control, historical data collection
− External services, project estimation, benchmarking,
consultancy
• CoE: Microsoft, Java, Oracle
− Onshore / offshore teams
− Expert estimates (technical architects / programmers)
• Challenge of parametric and expert estimate
9
10. Challenge – ‘sell parametric estimates’
• Project/Bid management still believe experts
more than parametrics
− ‘More detail must mean more accurate, right’?
− ‘This project is very different from past projects’
− ‘I see that we don’t get any function points for module XYZ,
but we have to do a lot of work for that!’
− ‘I think the project is quite easy and I think that the
parametric estimate overestimates the effort’
− But what about: team size, duration, forgotten activities,
past performance.
• How can we convince project management ??
10
12. Project at different durations
Plan A
Duration: 6 months
Effort: 4.500 hours
Max. team size: 5,8 fte
MTTD: 1,764 days
Team size (fte)
Plan B
Duration: 7 months
Effort: 2.400 hours
Max. team size: 2,7 fte
MTTD: 2,816 days
Which duration have the experts in
Duration mind?? 12
Size and productivity being constant
13. Assignment 2005
• Before 2005: estimation maturity level = 1
• Build estimation instrument
− Gain time and effort in estimating bids
− Accurate enough to depend and rely on
− Flexible:
◦ Estimate onshore / offshore and hybrid projects
◦ Calculate different test strategies
◦ Take into account complexity
◦ Implement Deming cycle (PDCA)
• Give scenario’s for duration !!!
13
14. First version Estimating Wizard (2005)
• Try to put the duration / effort tradeoff in a
model
− Tuned with experience data
Hour/FP: Average Complexity
Duration in months 3½ 4 4½ 5 5½ 6 6½ 7 7½ 8
0-250 FP 10,1 8,9 8,1 7,7 6,9
250-500 FP 9,1 8,0 7,3 6,9 6,2
500-750 FP 8,6 7,6 6,9 6,5 5,9
750-1000 FP 8,3 7,3 6,6 6,3 5,6
1000 -1250 FP 8,1 7,1 6,5 6,2 5,5
1250 - 1500 FP 7,9 6,9 6,3 6,0 5,4
14
15. Estimating Wizard 2011
Estimating Wizard Powered by: Sizing, Estimating & Control Data version: 24-11-2010 Model version: 17
Input
Functional design parameters
Functional Design Yes Step 1: Is there a functional design phase?
Overlap Yes, calculated 5 Step 2: In case of overlap between the functional design phase and building
to let the wizard calculate the overlap, or to enter the number of we
Language English Step 3: Enter the language in which the functional design should be written.
Availability key users Normal Step 4: Enter the availability-rate of the key users.
Location Sogeti office Step 5: Enter the location where the functional design should be written.
Build and test parameters
Development tool Java Step 6: Select the development tool.
Onshore Offshore
C onstruction 35% 65% Step 7: Enter the percentage of construction work that is done onshore.
Translation FD required No Step 8: Is a translation of the functional design required.
System test approach TMap Medium Step 9: Select the TMapfactory system test approach.
System test strategy Scripting and design NL, excecution in India Step 10: Select the system test strategy.
Tools/methodologies Unknown Step 11: Rate the level of tools and methodologies to be used for the develop
C omplexity Unknown Step 12: Rate the technical complexity of the project.
Development team Unknown Step 13: Rate the competence, experience and skill level of the development
Reuse Unknown Step 14: Rate the quantity and complexity of integrating reused, unmodified
General parameters
Size 643 C OSMIC Step 15: Enter the functional size and select a unit of size
(FP= function points, C FP=C OSMIC functionpoints).
Start date 01-01-11 Step 16: Enter the start date of the project.
Risk surcharge (%) 10 % Step 17: Enter the risk surcharge percentage.
Warranty (%) 4 % Step 18: Enter the warranty surcharge percentage.
Organization type Banking Step 19: C hoose the organization type.
Quality documentation 6 Step 20: Rate the quality of the documentation.
Non functional req. Average (0) Step 21: What influence do the non functional requirements have on the effo
Scenario interval 2,0 Step 22: Enter the number of weeks for the step size between the seven sce
15
16. EW - output
• Functional Design – no schedule scenarios
Functional design phase
Duration in weeks 17,6
Design complete 4-05-11
Total effort 1.975
Effort per FP 2,53
Effort cost € 208.531
Additional cost € 14.815
Totaal cost € 223.346
C ost per FP € 286
Average team size 2,80
Data altered due to company security reasons 16
19. ISBSG (www.isbsg.org)
• International Software Benchmarking Standards
Group
• Not-for-profit, members are mostly national
software measurement organizations
− NESMA, IFPUG, DASMA, JFPUG, GUFPI-ISMA, AEMES, etc.
• New developments & enhancements repository
− About 6.000 projects now
• Maintenance & Support repository
− Over 500 applications now
19
22. Estimating Wizard 2012
• Mature tool, proven value
• Effort Estimation Relationships are tuned to
historical data
• Size is the main input parameter, and it takes
time to measure size
• Parametric estimating is still not done in one
click, and perceived as expensive by internal
clients
• Experts are still perceived more credible and in
most challenges, the expert estimate is taken
• study: comparing experts and parametrics
22
23. Can parametrics beat the experts?
• Study of 20 completed projects
• Different sizes, languages, data normalized
− Expert estimate
− Estimating Wizard estimate (FP or CFP)
− Estimates vs. Actual results
• Calculate
− Effort Accuracy (Effort estimate / Actual effort)
− Duration Accuracy (Duration Estimate / Actual Duration)
− Cost Accuracy (Cost Estimate / Actual Cost)
• <1 underestimation, >1 overestimation
23
25. Estimation Accuracy results
• Average time spent
− Expert estimate: 34,3 hours
− Parametric Estimate (including size measurement): 22,1 hours
Expert Estimate Estimating Wizard Estimate
Effort Accuracy
Average 0,776 0,878 Closer to 1 means better!
St.Dev. 0,236 0,156
Median 0,752 0,908
Duration Accuracy
Average 0,755 0,953
St.Dev. 0,318 0,247
Median 0,791 0,998
Cost Accuracy
Average 0,775 1,127
St.Dev. 0,237 0,331
Median 0,763 1,122 25
26. Results
• Expert estimates take a lot of time too!
− on average 55% more than Parametric Estimates
− More than 1 expert has to read through all the
documentation, discussions, meetings, etcetera
• Parametric estimates are more acurate!
− Effort and duration estimates on average still optimistic, but
less optimistic than expert estimates
− Cost estimates are pessimistic, but still closer to actuals
than Expert estimate
• Experts might win the project, but the result will
be overruns! 26
27. Cost of high and low estimates
Non-linear extra costs
-Planning errors
-team enlargement more expensive, not faster
-Extra management attention / overhead
-Stress: More defects, lower maintainability !!
Linear extra kosten
Extra hours will be used
27
28. Conclusions
• Estimating wizard
− Higher effort estimation accuracy
− Higher duration estimation accuracy
− Higher cost estimation accuracy (although >1)
− Less hours spent than expert estimates
− Standard WBS, historical data collection and parameter
calibration improve the maturity of the process
• Next steps
− Analyze why costs are overestimated
− Try to identify the projects where EW estimate is enough
− Use the results to convince project management and bid
management to use parametric estimating even more !!
28
29. Thank you for your
attention !!
@haroldveendam Local touch - Global reach
Senior Consultant Software Metrics
Sogeti Nederland B.V.
NESMA board member
ISBSG president
COSMIC International Advisory Council, representing the
Netherlands
T: +31 (0)88 660 6600 + 3165 (dial)
M: +31 (0)6 52 32 73 30
e-mail: harold.van.heeringen@sogeti.nl 29
Hinweis der Redaktion
Good afternoon Ladies and Gentlemen. Welcome to this presentation on the estimation of software projects. I’m very glad to be in this conference. Actually, this is the first time that I’m attending a conference that is not completely dedicated to software estimation or software metrics. This presentation is about the differences between expert estimates and parametric estimates.
This is the agenda for the presentation. First I would like to say a few things about the company that I work for, as most of you probably don’t know the name Sogeti. After that, I am going to show the differences between expert estimates and parametric estimates in software projects. I’m working in the field of Parametric Estimates and I want to share one of our main challenges with you. Next I’m going to introduce the Sogeti estimating wizard and of course the results of the study we did on the accuracy differences between expert estimates and parametric estimates.
Ok, first a few remarks on Sogeti. We are a daughter company of Capgemini and we are the local IT brand of Capgemini. Close to our customers, but quite big in total. In the Netherlands, about 3000 people, but over 20.000 worldwide. We are well known for our Crraftsmenship and thought leadership in different areas, like testing, architecture and industry research… but also in parametric estimating (at least in the Netherlands we are)
1.01 Meten & Begroten dagdeel 1 v1.0 Sogeti Nederland B.V. v1.0
As I mentioned on the previous slide, it’s very important to size the software that has to be realized in an accurate way. As this is actually the main cost driver for all our models and tools. Unfortunately, unlike for instance in the construction industry where one can draw a building plan and then measure the surface or volume and other physical things, software is a non-physical thing. We cannot grab and hold a piece of software and therefore it’s hard to measure. In software industry, a best practice has been evolved over the last few decades to measure the functionality of the requirements that the user wishes to be fulfilled in the software. These requirements should be known beforehand, right? That’s why an ISO standard for functional size measurement methods has been drawn up and a number of functional size measurement methods comply to this standard. In order to be able to use the size measure in collecting historical data, the building of parametric models and the estimation of software projects, of course the method must be objective, repeatable and verifiable. The three methods listed here are all ISO standards and the measure is called function points. However, even the ISO standard does not guarantee a completely accurate result. Functional user requirements are usually described only high level when the bid has to take place and changes are likely to occur during the project.
1.01 Meten & Begroten dagdeel 1 v1.0 Sogeti Nederland B.V. v1.0
1.01 Meten & Begroten dagdeel 1 v1.0 Sogeti Nederland B.V. v1.0
Sogeti Nederland B.V.
I am working in the department Sizing, Estimating & Control and I’m involved in the quotations we are giving our customers. We have a number of specialists who are experts in sizing the requirements of the clients. The size of the user requirements is actually a main cost driver in many projects, so this is an important part of our work. When the size is measured, we have the parametric models and tools to estimate characteristics like effort, duration, team size, quality and costs. In our division, every quotation is supported by both a parametric estimation and an expert estimation, which we call technical calculation. The expert calculation is carried out by technical architects in one of our Centers of Excellence. Sogeti Nederland B.V.
Dan kijken we naar een andere belangrijke factor: de doorlooptijd. We nemen als uitgangspunt de software equation die door Putnam senior is gepubliceerd en die ook in de QSM SLIM tooling wordt gebruikt. U ziet de formule staan. Het idee is dat bij een gegeven omvang en een gegeven productiviteit, het altijd mogelijk is om verschillende inspanning / doorlooptijd scenario’s te maken. De omvang is objectief gemeten. De productiviteit komt uit de ervaringscijfers. U ziet hier 2 verschillende plannen staan. Omdat de formule een zwaarder gewicht toekent aan de doorlooptijd is dit een belangrijke factor. De doorlooptijd iets verkorten leidt tot een onevenredige toename van het totaal aantal benodigde uren.
De reden daarvoor wordt duidelijk als we naar deze figuur kijken. Ik heb de twee plannen uit de vorige figuur hier iets verder uitgewerkt. We zien dat we voor plan A een groter team nodig hebben dan voor plan B. Dit team is misschien groter dan u zou verwachten. Dit komt omdat iedere extra persoon in een team de overall productiviteit verlaagd. Er worden extra communicatiepaden gecreëerd en het wordt moeilijker om het project te managen in die zin dat iedereen nuttig aan het werk gehouden moet worden. Een groter team betekent ook meer defects, wat hier tot uiting komt in een lagere mean-time-to-defect. Iedere defect moet worden gelogd, terug naar de bouw, geanalyseerd, opgelost, geunittest en weer aan de systeemtest worden opgeleverd. Veel extra werk per defect dus.