Abstract—This article provides insight into a ‘best practice’ used for the selection of software suppliers at the largest Dutch telecom operator, KPN[1]. It explains the metrics rationale applied by KPN when selecting only one preferred supplier (system integrator) per domain instead of the various suppliers that were previously active in each domain. Presently (Q2 2012) the selection and contracting process is entering its final phase. In this paper, the model that was built and used to assess the productivity of the various suppliers and the results of the supplier selection process are discussed. In addition, a number of lessons learned and recommendations are shared.

2. Metrics Based Software Supplier Selection
Best practice used in the largest Dutch telecom company
Hans Kuijpers
Harold van Heeringen
Assisi, October 2012

3. Introduction
Harold van Heeringen
Hans Kuijpers
Sizing, Estimating & Control
Program Assurance & Methods
Harold.van.heeringen@sogeti.nl
hans.tm.kuijpers@kpn,.com
@haroldveendam
@_hanskuijpers
@Sogeti_SEC
Sogeti Nederland BV: KPN Nederland:
Senior Consultant Software Metrics Manager Metrics Desk
ISBSG: President Certified Scope Manager
NESMA: Board Member QSM: Special Interest Group Agile
NESMA: Working Group Chair COSMIC
NESMA: Working Group Chair Benchmarking
COSMIC: IAC Member
Metrics Based Software Supplier Selection Pagina 2

4. Agenda
• Introduction and Context
• Phases and Timeline
• The Model
• Results
• Conclusions & Recommendations
Metrics Based Software Supplier Selection -3-

5. Introduction and Context
Revenues: €13.000m
.Why supplier selection? KPN Board
EBITDA: € 5.100m
Employees: 31.000
• Consolidation # suppliers
• Cost reduction
Consumer Business Corporate KPN
• Supplier acts as SI & MSP Market Market Market NetCo E-Plus
Belgium
• 5 Year investment
• SPM is a best practice at KPN Customer
Fixed Mobile Wholesale IT Operations
Experience
Generic &
OSS  Domains  BSS
Traditional
Problem: no more competition between suppliers
 instrument needed to avoid excessive cost  Unit of Work pricing
Metrics Based Software Supplier Selection -4-

6. Agenda
• Introduction and Context
• Phases and Timeline
• The Model
• Results
• Conclusions & Recommendations
Metrics Based Software Supplier Selection -5-

7. Phases and Timeline
Why Productivity metric added?
• Objective selection criteria
• Supplier willingness to show their transparency
• Basis for productivity baseline
• Insight in quality level
• Negotiations for year on year cost reduction
• Relation to continous improvement steps
Metrics Based Software Supplier Selection -6-

8. Requested project information
• Data of 6 historical projects, max 3 KPN projects
• In scope of current technology domain
• Range 300 – 1000 FP
• Sizing method NESMA 2.1 or IFPUG 4.x
• DCF must be completely filled out
• No other template is allowed
In BAFO phase suppliers should show evidence of the size and productivity
figures by releasing FPA-reports, Data Collection Forms and/or insight in their
administrative systems.
Metrics Based Software Supplier Selection -7-

9. Historical Project Data form (1)
Metrics Based Software Supplier Selection -8-

10. Historical Project Data form (2)
Per data field
requirements are
mentioned in the
template.
Metrics Based Software Supplier Selection -9-

11. Agenda
• Introduction and Context
• Phases and Timeline
• The Model
• Results
• Conclusions & Recommendations
Metrics Based Software Supplier Selection - 10 -

12. The Model
Characteristics:
• Degree of openess and compliancy
• Completeness and cohesion of submitted data
• Productivity benchmark against each other and industry
• Delivered Quality
• During the RFP phase the data will be considered as correct, but will be
checked on reality
The 3 test criteria:
A. Compliancy value (10%)
B. Reality value (30%)
C. Productivity - Quality value (60%)
Metrics Based Software Supplier Selection - 11 -

13. Used metrics and benchmarks
 Project Delivery Rate (PDR) = spent project effort related to function point (h/FP)
 Productivity Index (PI) = metric from QSM, derived from size, duration and effort
 Quality: delivered defects per FP
Benchmarks:
• PI against the QSM Business trend line
• PDR against ISBSG Repository
• Adjusted = normalised to Construction+Test activities
Metrics Based Software Supplier Selection - 12 -

14. Compliancy value (10%)
Suppliers start with 10 points
The compliancy value is substracted with 2 points for each violation of rule:
a)Range 300 – 1000 Function Points
b)Method NESMA 2.1 or IFPUG 4.x
c) Each field of “Historical Project Data”-form must be filled out
Maximum value = 10, minimum value = 0
Metrics Based Software Supplier Selection - 13 -

15. Reality value (30%)
PI vs. Functional size (FP)
35
Unrealistic projects are discarded Unrealistic
from further analysis: 30
• PI > +2 sigma (95%) 25
• PDR < P25 ISBSG (best in class 20
projects)
PI
15
The reality value is substracted with 10
Supplier A
Supplier B
2 points for each unrealistic project 5
Supplier C
Supplier D
Supplier E
QSM Business
Maximum value = 10 0
Avg. Line Style
2 Sigma Line Style
50 150 250 350 450 550 650 750 850 950
Minimum value = 0 Effective FP
Functional Size (FP)
Metrics Based Software Supplier Selection - 14 -

16. Productivity - Quality value (60%)
Productivity - Quality value =
(Points PI score * 0,5) + (Points PDR score * 0,3) + (Points Quality score * 0,2)
ID PDR (h/FP) PDR ISBSG median PDR score ID Defects/FP Quality score
7 5,9 8,6 -2,7 15 41,7
8 6,0 8,6 -2,6 18 13,9
9 6,9 8,6 -1,7 21 66,7
11 6,2 8,6 -2,4 22 4,0
12 7,3 8,6 -1,3 23 10,0
Average: -2,1
Median 13,9
The lowest average most points The lowest median most points
For PI and PDR the average of the distance to the benchmark value is determined
For the quality the median is dertermined
The highest average most points
Metrics Based Software Supplier Selection - 15 -

17. Agenda
• Introduction and Context
• Phases and Timeline
• The Model
• Results
• Conclusions & Recommendations
Metrics Based Software Supplier Selection - 16 -

18. Results of Compliancy (1)
Projects discarded:
• Projects on going (4)
• Project sized in COSMIC (1)  Result: 1 supplier has 3 violations,
the others 5 or more
Blank crucial fields
• Defect data Supplier Compliancy Value
• Effort data Supplier A 0
• Dates Supplier B 0
Supplier C 4
Other violations Supplier D 0
• Primary Language (example English) Supplier E 0
Metrics Based Software Supplier Selection - 17 -

19. Results of Compliancy (2)
Metrics Based Software Supplier Selection - 18 -

20. Results of Reality
Projects unrealistic:
• 3 according to PI
• 1 according to PDR
Unrealistic Unrealistic
 Discarded for further analysis projects PI projects PDR
Supplier criterion criterion Reality Value
Supplier A 1 1 6
Supplier B 1 0 8
Supplier C 0 0 10
Supplier D 0 0 10
Supplier E 1 0 8
Metrics Based Software Supplier Selection - 19 -

21. Results of Productivity / Quality
Rank Points PDR Rank Points
Supplier PI score PI score PI score Supplier score PDR score PDR score
Supplier A 3,9 2 8 Supplier A -3,2 1 10
Supplier B 5,0 1 10
+
Supplier B -2,1 2 8 +
Supplier C 3,4 3 6 Supplier C 16,6 4 4
Supplier D 3,0 5 2 Supplier D 6,2 3 6
Supplier E 3,2 4 4 Supplier E 18,3 5 2
Quality Rank Points Points Points Points Productivity/
Supplier Score Quality score Quality score Supplier PI score PDR score Quality score Quality value
Supplier A 3,1 1 10 Supplier A 8 10 10 9,0
Supplier B 13,9 2 8 Supplier B 10 8 8 9,0
Supplier C 52,6 3 6 = Supplier C 6 4 6 5,4
Supplier D 1000,0 5 2 Supplier D 2 6 2 3,2
Supplier E 94,6 4 4 Supplier E 4 2 4 3,4
weight 50% 30% 20%
Metrics Based Software Supplier Selection - 20 -

22. Results of Total Assessment
Recommendation from Metrics Desk: Supplier B and A score best in the model
Compliancy Reality Productivity/ Total
Supplier value value Quality value Points Rank
Supplier A 0 6 9,0 7,2 2
Supplier B 0 8 9,0 7,8 1
Supplier C 4 10 5,4 6,6 3
Supplier D 0 10 3,2 4,9 4
Supplier E 0 8 3,4 4,4 5
weight 10% 30% 60%
However suppliers B and C were selected for the next BAFO phase
Metrics Based Software Supplier Selection - 21 -

23. Findings BAFO phase
Metrics Desk investigated the provided project data of the selected suppliers B + C:
• Size
• Dates
• Hours
• Defects
Supplier B:
• Resistance: confidentiality clause with clients
• Client site visit
Supplier C:
• Size measurement by junior not certified measurers
Metrics Based Software Supplier Selection - 22 -

24. Agenda
• Introduction and Context
• Phases and Timeline
• The Model
• Results
• Conclusions & Recommendations
Metrics Based Software Supplier Selection - 23 -

25. Conclusions and recommendations
• The productivity assessment influenced the total outcome significantly
• The assessment and discussions afterwards gave insight in:
Conclusions:
• Transparency and CMMI level
• The results are used in the negotiations phase to maximize the baseline value
• Make sure the parties understand:
• the purpose of the assessment
• the use of the “Historical Project Data” form
• that the disclosed data will be validated and should not be confidental
• the consequences of violating the governance rules (e.g. penalty points)
Recommendations: • Because of many violations of the compliancy rules, consider 1 penalty point
per violation
• Construct model beforehand, but don’t communicate the model with suppliers
• Bring site visits when offered. This gives extra information next to the
productivity validation
Metrics Based Software Supplier Selection - 24 -

26. Productivity: don’t trust it, check it
Harold van Heeringen Hans Kuijpers
Sizing, Estimating & Control Software Metrics Consultant
Harold.van.heeringen@sogeti.nl hans.tm.kuijpers@kpn,.com
@haroldveendam @_hanskuijpers
@Sogeti_SEC
Metrics Based Software Supplier Selection - 25 -

27. Back-up sheets
Metrics Based Software Supplier Selection - 26 -

28. Productivity index
(PI)
500 FP
Effort 17 h/FP
Productivity Index=18
14 h/FP Productivity Index=16
10 h/FP 9 h/FP
7 h/FP
Duration
- 27 -