Wouldn’t it be nice if you could have more insight into the quality of a product, while it is developed, and not afterwards? Would you like to be able to estimate how many defects are inserted in the product in a certain phase, and how effective a (test) phase is in capturing these defects? To optimize your test phases regarding focus and effort in relation to how many defects they will find? This presentation will show a simple but very effective model that makes it possible: The Project Defect Model. The aim of the Project Defect Model is to track product quality, take corrective actions and reduce quality risks. To get more insight into the quality of the product during development, it is needed to measure the software development processes with two views: Introduction and detection of defects. Introduction is done during the specification, design and coding phases; defects are either introduced into documents or into the actual product. Detection of defects is done via inspections and test during all the phases of the project. A tool was developed using a spreadsheet. The purpose of the tool was to estimate the number of defects per phase, and to track all defects discovered in inspections and tests against these estimates. The tool supported analysis of the data with both calculated values and graphs comparing actuals to estimates in terms of current status and trends over time. The Project Defect Model has been beneficial to projects. It has helped estimating, planning, and tracking quality during the project, including an estimate of the the number of defects left in the released product. The quality data has been used in the project together with time and cost data, to take better decisions on test, review and inspections, and design. Also it has identified quality risks at an early stage, helping the project take corrective actions and decisions on product release and maintenance capacity planning. Finally the model provided insight into the effectiveness of the verification activities, supporting effective process improvement. Paper: The presentation is on a defect planning/tracking tool and approach. Focus will be upon: • Goals: What was the purpose of the model, why developed, what did we want to reach? • How: Show the definition of the model and its implementation and application. • Tools: The tool that was developed to implement the model, how it works, strengths. • Results: How did the model and tool help the project? Did it live up to its purpose? • Success factors: What were key issues that we have dealt successfully with? • Future: How is this model used in future projects, what could further increase its benefits?

1. Controlling Product Quality during Development with a Defect Model 1 February 9, 2004 Ben Linders
Controlling Product Quality during
Development with a Defect Model
ICSTest 2004 Conference,
Düsseldorf, April 22
Ben Linders
Operational Development & Quality
Ericsson R&D, The Netherlands
ben.linders@ericsson.com, +31 161 24 9885

2. Controlling Product Quality during Development with a Defect Model 2 February 9, 2004 Ben Linders
Overview
• Why a defect model?
• How does it work?
• Experiences from projects
• Conclusions
Measurements for product quality
and process effectiveness

3. Controlling Product Quality during Development with a Defect Model 3 February 9, 2004 Ben Linders
Ericsson, The Netherlands
• Main R&D Design Center
• Full product responsibility for Intelligent Networks
– Strategic Product Management
– Provisioning & total project management
– Development & maintenance
– Supply & support
• 1400 employees, of which 350 in R&D
Projects: Quality next to Lead-time and Costs

4. Controlling Product Quality during Development with a Defect Model 4 February 9, 2004 Ben Linders
Purpose Project Defect Model
Why?
– to control quality of the product during development
– and improve development/inspection/test processes
Business Benefit:
➨ Higher test efficiency
➨ Better planning & tracking
➨ Early risks signals
➨ Better focus on important issues
➨ Save time and costs
➨ Happy customers!

5. Controlling Product Quality during Development with a Defect Model 5 February 9, 2004 Ben Linders
Test Efficiency
• How much testing is needed?
– Too little: Product quality risk
– Too much: Money/time wasted
• Aim/Focus for each test phase?
• Budget/time/resource constraints?
• Incremental deliveries: What can be tested?
Plan Testing & Track/Adjust on defects found.

6. Controlling Product Quality during Development with a Defect Model 6 February 9, 2004 Ben Linders
Defect Flow
• Prevent defect insertion
• Detect & remove defects where most economical
• Track inspection/test progress

7. Controlling Product Quality during Development with a Defect Model 7 February 9, 2004 Ben Linders
Planning & Tracking of Quality
• Plan Quality Up Front
– Documents/code (# defects made)
– Inspection & Test effectiveness (% detection rate)
Quality consequence of project decisions
• Track Quality during project
– Actual # defects found (inspection/test)
– Estimate remaining defects: to be found / delivered
'Real-time' prediction of product quality possible
Quality view of design/test progress
Quicker escalation of quality risks

8. Controlling Product Quality during Development with a Defect Model 8 February 9, 2004 Ben Linders
Measurements: Defect Insertion
Defect insertion Target Defect Density: Max 1 major/page per document!
Phase
Expec-
ted
#def
Expec-
ted
size
Expec-
ted
DD
Act.
Size
Fnd
#def
DD
Act
Not
found
yet
%
Foun
d
% Exp
of
total
Specification 4 10 0.4 10 4 0.40 0 100% 4%
High Level Design 12 107 0.112 107 10 0.09 2 83% 12%
Detailed Design 12 47 0.255 47 10 0.21 2 83% 12%
Im plem entation 70 15000 4.667 13000 18 0.00 52 26% 71%
Total 98 42 56 43% 100%
• Input: Design team estimates # of defects inserted & expected size
• Gathered data during project: Actual defects & size
• For each delivery, # of defects in product is calculated:
– Indication of the delivered product quality
– Input to the test planning: How many defects can be found?
– Input to project management to staff/dimension test teams accordingly

9. Controlling Product Quality during Development with a Defect Model 9 February 9, 2004 Ben Linders
Measurements: Defect Detection
Defect detection Target detection rate: 70% document, 60% code, 50% test!
Avail-
Phase Def. Det # Goal %Det % Left Det # Det % Cum %
Specification 4 2 70% 50% 2 2 50% 50%
High Level Design 14 11 70% 79% 3 11 79% 69%
Detailed Design 15 6 70% 40% 9 6 40% 61%
Implementation 79 40 60% 51% 39 6 8% 23%
Unit test 39 8 20% 21% 31 6 15% 30%
Function test 31 15 50% 48% 16 3 10% 33%
System Test 16 9 50% 56% 7 3 19% 36%
Network Test 7 2 40% 29% 5 2 29% 14%
Installation 5 1 15% 20% 4 2 40% 12%
First Customer 4 1 10% 25% 3 1 25% 10%
Average/Total: 95 42% 42 31%
Actual totalExpected in phase
• Input: Nr of defects expected to detect & detection rate goal
• Gathered data during project: Actual defects & detection rate
• Track: Defects found = Product Quality, Test progress

10. Controlling Product Quality during Development with a Defect Model 10 February 9, 2004 Ben Linders
Implementation
• Tool: Excel based defect data base & estimation
• Frequent estimation & analysis sessions
– Ones per week/2 weeks, per project
– Duration: 10 minutes – 1 hour, usually ½ hour
– Attending: Design, test, quality
• Weekly tracking & reporting of product quality
• Includes proven techniques: ODC, requirement coverage, test matrices
Tailored per project, flexible, result oriented
Overall data based on all projects: Planning constants
Quality data, additional to time & costs!

11. Controlling Product Quality during Development with a Defect Model 11 February 9, 2004 Ben Linders
Experiences in Pilot Project
• Quality tracked during the project:
– Specification defects slip through: Clarified requirements in feasibility
– Design defects (inspection): Re-enforced design rules
– Code quality (inspection/test): Base Product risk, design rules
– Test efficiency, defect slip though: Better inspection/Unit Test
– Release Quality per requirement: Test focus, risk management
• Prediction nr of defects at First Customer Delivery and Release:
– Decisions on delivery/release, design follow up and maintenance planning
– Actual defects: Expected 21, actual 20 (in 6 months operation)
Pilot Project Defect Detection rate: 95% (best in class)!

12. Controlling Product Quality during Development with a Defect Model 12 February 9, 2004 Ben Linders
Experiences from ongoing projects
• Classification/analysis of defect with Design & Test
Leaders provides very valuable information.
• Feedback sessions with Project Management Group
are essential for conclusions, and taking actions.
• Estimated latent defects supported release decisions.
• Defect data improved time & costs planning, and risk management.
Also historical data of project support planning of new projects.
Though some conclusions from the model are not unique, they
would have been missed or discovered too late without the model.

13. Controlling Product Quality during Development with a Defect Model 13 February 9, 2004 Ben Linders
Conclusions
Project Defect Model helped the project to:
– Estimate/track defects: Improve product release quality, save time/cost
– Design/test progress: Better planning, risk management, decisions
– Setup Organization: Dimension project teams/maintenance teams
Future
– Internal & Industry data: Improve estimates
– Extend model with cost & planning data
– Exchange experiences with similar models?
Questions?