In many organizations, bottom up estimation of software development projects is still the way to go. Management feels that top-down (parametric) estimation models require too much effort and cost. In this presentation, a random selection of 10 bids are analyzed and the bottom-up estimates and top-down estimates are compared with regard to accuracy and effort spend.

1. Estimate Faster, Cheaper…
and Better!
H.S. van Heeringen
SMEF 2011 (Rome)
@haroldveendam

2. Overview
• Expert vs. Parametric Estimates
• Chalenge for parametric estimates
• Estimating Wizard
• Study: Expert accuracy vs. Parametric accuracy
2

3. Project Estimates
• Two types of project estimation:
− Expert estimation
− Parametric estimation
• Expert
− Knowledge and experience of experts
− Assign effort hours to tasks (bottom-up)
− Subjective, but always applicable
• Parametric
− Size measurement, historical data and tooling
− Size measurement methods: NESMA FPA, COSMIC, IFPUG
− Objective, but well documented specifications required
3

4. Comparing the two types
• Expert Estimates
−
−
−
−
−
−
Bottom-up estimation
Usually optimistic (up to 30% under estimation is common)
Forgotten activities
Hard to defend
The expert is not going to do all the work
The expert may not be an expert on the new project
• Parameteric Estimates
− Top-down estimation
− Estimating & Performance Measurement process needed
− Effort = size * Productivity
◦ - Size is objectively measureable (COSMIC, FPA)
◦ - Productivity from historical data (organization / ISBSG)
4

5. Expert Estimation
Task– Code and en Unit test of module XYZ
Probability
100%
90% . . . . . . . .
75% . . . . . . . .
Realistic estimate 50% . . . . . . . .
10% . . . . . . . .
Expert estimate
0% . . . . . . . .
0%
Effort
. . . . . . . . 24 hours
. . . . . . . . 22 hours
. . . . . . . . 20 hours
. . . . . . . . 18 hours
. . . . . . . . 16 hours
. . . . . . . . 14 hours
. . . . . . . . 12 hours
. . . . . . . . 10 hours
. . . . . . . . 8 hours
. . . . . . . . 6 hours
. . . . . . . . 4 hours
. . . . . . . . 2 hours
Expert Estimate: 1st possibility of success
50/50 median result
5

6. Eestimate & Performance Measurement
Start: Periodically
Start: Estimate request
Finetune Estimation model
Analyse productivity,
Report productivity
Size measurement: FPA
Historical data
Estimation tools
Result:
-Management report,
-Adjusted model
ACT
Adjust &
Report
PLAN
Estimate Results:
- Parametric Estimation
- Expert Estimation
Start: Project completed
Data collection and
administration
• Collect project data
• Measure size
• Benchmark the project
Result:
-Growing project DB,
-Performance measurement
-Updated expert knowledge
Expert Estimate
Start: Project start
Evaluate
CHECK
Administrate
DO
Continuous data collection
• effort hours registration
• defect registration
• change measurement
• project characteristics
Result: Project data
6

7. Challenge – ‘sell parametric estimates’
• Project/Bid management still believe experts
more than parametrics
− ‘More detail must mean more accurate, right’?
− ‘This project is very different from past projects’
− ‘I see that we don’t get any function points for module XYZ,
but we have to do a lot of work for that!’
− ‘I think the project is quite easy and I think that the
parametric estimate overestimates the effort’
− But what about: team size, duration, forgotten activities,
past performance.
• How can we convince project management ??
7

8. Software equation(Putnam)
Size/productivity
1/3
* duration4/3
Effort
= Effort
Plan A: 6 months, 4.500 hours
Plan B: 7 months, 2.400 hours
Duration
8

9. Project at different durations
Effort (hours)
Plan A
Duration: 6 months
Effort: 4.500 hours
Max. team size: 5,8 fte
MTTD: 1,764 days
Plan B
Duration: 7 months
Effort: 2.400 hours
Max. team size: 2,7 fte
MTTD: 2,816 days
Duration
Size and productivity being constant
Which duration have the experts in mind??
9

10. Sogeti AS SEC
• Division AS – RVO’s (fixed price/ fixed date)
• Sizing, Estimating & Control
− 11 (COSMIC) Function Point Analysts
− 2 metrics consultants
• Responsible for metrics part of a quotation.
− Size: FPA/COSMIC
− Estimation: SEER-SEM / QSM / Sogeti tool / ISBSG
− Product: Methodical Estimation Report (scenario’s)
• Centers of Excellence: MS.Net, Java, Oracle
• Experts (technical architects / programmers)
10

11. Assignment 2005
• Build estimation instrument
− Gain time and effort in estimating bids
− Accurate enough to depend and rely on
− Flexible:
◦ Estimate onshore / offshore and hybrids
◦ Calculate different test strategies
◦ Take into account complexity
◦ Implement Deming cycle (PDCA)
− Give scenario’s for duration !!!
11

12. First version Estimating Wizard (2005)
• Try to grasp the duration / effort tradeoff in a
model
− Tuned with experience data
Hour/FP: Average Complexity
Duration in months
3½
0-250 FP
250-500 FP
500-750 FP
750-1000 FP
1000 -1250 FP
1250 - 1500 FP
10,1
4
8,9
9,1
4½
8,1
8,0
8,6
5
7,7
7,3
7,6
8,3
5½
6,9
6,9
6,9
7,3
8,1
6
6,2
6,5
6,6
7,1
7,9
6½
5,9
6,3
6,5
6,9
7
5,6
6,2
6,3
12
7½
5,5
6,0
8
5,4

13. Estimating Wizard 2011
Estimating Wizard
Powered by: Sizing, Estimating & Control
Data version: 24-11-2010
Model version: 17
Input
Functional design parameters
Functional Design
Yes
Overlap
Yes, calculated
Language
Availability key users
Location
English
Normal
Sogeti office
Build and test parameters
Development tool
C onstruction
Translation FD required
System test approach
System test strategy
Tools/methodologies
C omplexity
Development team
Reuse
General parameters
Size
Start date
Risk surcharge (%)
Warranty (%)
Organization type
Quality documentation
Non functional req.
Scenario interval
5
Java
Onshore
35%
No
Step 6: Select the development tool.
Offshore
65%
TMap Medium
Scripting and design NL, excecution in India
Unknown
Unknown
Unknown
Unknown
643 C OSMIC
01-01-11
10 %
4 %
Banking
6
Average (0)
2,0
Step 1: Is there a functional design phase?
Step 2: In case of overlap between the functional design phase and building
to let the wizard calculate the overlap, or to enter the number of we
Step 3: Enter the language in which the functional design should be written.
Step 4: Enter the availability-rate of the key users.
Step 5: Enter the location where the functional design should be written.
Step 7: Enter the percentage of construction work that is done onshore.
Step 8: Is a translation of the functional design required.
Step 9: Select the TMapfactory system test approach.
Step 10: Select the system test strategy.
Step
Step
Step
Step
11:
12:
13:
14:
Rate
Rate
Rate
Rate
the
the
the
the
level of tools and methodologies to be used for the develop
technical complexity of the project.
competence, experience and skill level of the development
quantity and complexity of integrating reused, unmodified
Step 15: Enter the functional size and select a unit of size
(FP= function points, C FP=C OSMIC functionpoints).
Step 16: Enter the start date of the project.
Step 17: Enter the risk surcharge percentage.
Step 18: Enter the warranty surcharge percentage.
Step 19: C hoose the organization type.
Step 20: Rate the quality of the documentation.
Step 21: What influence do the non functional requirements have on the effor
Step 22: Enter the number of weeks for the step size between the seven scen
13

14. EW - output
• Functional Design – no schedule scenarios
Functional design phase
Duration in weeks
Design complete
Total effort
Effort per FP
Effort cost
Additional cost
Totaal cost
C ost per FP
Average team size
17,6
4-05-11
1.975
2,53
€ 208.531
€ 14.815
€ 223.346
€ 286
2,80
Data altered due to company security reasons
14

15. EW - output
• Main build – 7 schedule scenarios
Build and test phase
Duration in weeks
Start phase
Effort
Effort per FP
Effort cost
Additional cost
Totaal cost
C ost per FP
Average team size
Risk and warranty
Risk hours
Risk cost
Warranty hours
Warranty cost
Total
Duration in weeks
Delivery for acceptance
Total effort
Effort per FP
Totaal cost
C ost per FP
Average team size
Average hourly rate
20,0
18-02-11
9.794
28,06
€ 550.720
€ 56.446
€ 607.167
€ 1.740
12,24
22,0
18-02-11
6.690
19,17
€ 376.152
€ 41.090
€ 417.242
€ 1.196
7,60
24,0
18-02-11
4.723
13,53
€ 265.590
€ 31.364
€ 296.954
€ 851
4,92
26,0
18-02-11
3.429
9,83
€ 192.826
€ 24.963
€ 217.789
€ 624
3,30
28,0
18-02-11
2.550
7,31
€ 143.360
€ 20.611
€ 163.972
€ 470
2,28
30,0
18-02-11
1.935
5,54
€ 108.787
€ 17.570
€ 126.357
€ 362
1,61
32,0
18-02-11
1.495
4,28
€ 84.036
€ 15.393
€ 99.428
€ 285
1,17
834
€ 54.301
209
€ 13.575
586
€ 39.107
146
€ 9.777
429
€ 29.484
107
€ 7.371
325
€ 23.150
81
€ 5.788
255
€ 18.845
64
€ 4.711
205
€ 15.836
51
€ 3.959
170
€ 13.682
43
€ 3.420
27,0
8-07-11
11.470
32,87
€ 746.634
€ 2.139
10,64
€ 65
29,0
22-07-11
8.055
23,08
€ 537.717
€ 1.541
6,96
€ 67
31,0
5-08-11
5.892
16,88
€ 405.400
€ 1.162
4,76
€ 69
33,0
19-08-11
4.469
12,80
€ 318.319
€ 912
3,39
€ 71
35,0
2-09-11
3.501
10,03
€ 259.120
€ 742
2,50
€ 74
37,0
16-09-11
2.825
8,09
€ 217.744
€ 624
1,91
€ 77
39,0
30-09-11
2.340
6,71
€ 188.122
€ 539
1,50
€ 80
Data altered due to company security reasons
15

16. Calibrating the model
• Collect historical data
• Use ISBSG data
16

17. Calibrating the model
• Collect historical data
• Use ISBSG data
17

18. Pricing model
• Sogeti Pricing model
18

19. Results
• Study of 10 completed projects
− Expert estimate
− Estimating Wizard estimate (FP or CFP)
− Estimates vs. Actual results
• Calculate
− Effort accuracy (Effort estimate / Actual effort)
− Duration Accuracy (Duration Estimate / Actual Duration)
− Cost Accuracy (Cost Estimate / Actual Cost)
19

20. Study
• 10 projects selected randomly
• Different sizes, programming languages, etc
• Data normalized when necessary (to compare
apples with apples)
20

21. Results
Project
Project 1
Project 2
Project 3
Project 4
Project 5
Project 6
Project 7
Project 8
Project 9
Project 10
Expert
Effort
Size (FP) Accuracy
277
0,675
359
0,579
347
0,589
1.178
0,414
951
1,430
295
0,763
790
0,717
350
1,258
746
0,586
2.293
0,766
Expert
Duration
Accuracy
1,545
0,951
0,142
0,557
0,997
0,857
0,850
0,800
0,296
0,421
Expert
Cost
Accuracy
0,467
1,139
0,615
0,312
0,946
0,619
0,976
1,309
0,545
0,797
Expert
Est. Wizard Est. Wizard Est. Wizard Est. Wizard
Effort
Duration Cost
Time
Time Spent Accuracy
Accuracy Accuracy
spent
30
0,477
1,204
0,501
17
35
0,707
0,775
1,170
26
40
1,067
0,996
1,283
14
60
0,774
0,590
0,862
55
34
1,067
0,877
1,718
24
26
0,881
1,200
0,845
6
34
0,926
0,865
1,132
27
28
1,203
1,096
1,318
20
34
0,826
0,385
1,953
22
40
0,931
0,632
1,058
14
• Calculate
− Effort accuracy (Effort estimate / Actual effort)
− Duration Accuracy (Duration Estimate / Actual Duration)
− Cost Accuracy (Cost Estimate / Actual Cost)
21

22. Estimation Accuracy results
• Average time spent
− Expert: 36,1 hours
− EW (including FPA / COSMIC measurement): 22,5 hours
Expert Estimate
Est. Wizard Estimate
Effort Accuracy
Average
St.Dev.
Median
0,778
0,319
0,696
0,886
0,207
0,904
Duration Accuracy
Average
St.Dev.
Median
0,742
0,405
0,825
0,862
0,272
0,871
Cost Accuracy
Average
St.Dev.
Median
0,772
0,316
0,708
1,184
0,423
1,151
22

23. Results
• Expert estimates take a lot of time too!
− on average 60% more than Parametric Estimates
− More than 1 expert has to read through all the
documentation
• Parametric estimates are more acurate!
− Effort and duration estimates on average still optimistic, but
less optimistic than expert estimates
− Cost estimate pessimistic, but still closer to actuals than
Expert estimate
• Experts might win the project, but the result will
be overruns!
23

24. Cost of high and low estimates
Non-linear extra costs
-Planning errors
-team enlargement more expensive, not faster
-Extra management attention / overhead
-Stress: More defects, lower maintainability !!
Linear extra kosten
Extra hours will be used
24

25. Conclusions
• Estimating wizard
−
−
−
−
−
Higher effort estimation accuracy
Higher duration estimation accuracy
Higher cost estimation accuracy (although >1)
Less hours spent than expert estimates
Standard WBS, historical data collection and parameter
calibration improve the maturity of the process
• Next steps
− Analyze why costs are overestimated
− Try to identify the projects where EW estimate is enough
− Use the results to convince project management and bid
management to use parametric estimating more
25

26. Thank you for your attention !!
Local touch - Global reach
@haroldveendam
26