2. 2 QUALITY. PRODUCTIVITY. INNOVATION.
Software Quality
What are Testing Metric?
Why we need metrics and why
managers like them?
Important Metrics and how to
identify the ones we need
Metrics Report Example
Can Metrics Cheat?
Testing Metrics
3. 3
3 QUALITY. PRODUCTIVITY. INNOVATION.
What makes software development projects successful?
• Product quality
• Optimize implemented processes
• Increase Team efficiency
• Reduce cost and time needed to deliver
• Keep customers happy
Software Quality
4. 4
4 QUALITY. PRODUCTIVITY. INNOVATION.
“When you can measure what you are speaking about, and can express it in
numbers, you know something about it; but when you cannot measure it,
when you cannot express it in numbers, your knowledge is of a meager and
unsatisfactory kind.”
-- Lord Kelvin, a physicist.
Software testing metric is a quantitative measure of the degree to which
a system, component, product or process possesses a given attribute. In
other words, metrics helps estimating the progress, quality and health of a
software testing effort.
Example of software testing metric: Total number of defects found
What is Software Testing Metric?
5. 5
5 QUALITY. PRODUCTIVITY. INNOVATION.
• "We cannot improve what we cannot measure“
• Display past and present performance
• Predict future
• Understand what needs to improve
• Decide what to do next
Why do Test Metrics?
6. 6
6 QUALITY. PRODUCTIVITY. INNOVATION.
“All they (managers) care about are numbers…”
• Your team is doing a great job – show it!
• Inefficiency affects schedules and product quality
• Need to have a way to measure QA process quality
• Metrics help you generate good test reports
Why Managers Like Them?
7. 7
7 QUALITY. PRODUCTIVITY. INNOVATION.
Metrics can be sorted into groups based on different criteria:
• Project-, department- and company-level metrics
• Process (process efficiency), product (improve software quality) and
project metrics (measure team or tools efficiency)
• Base (raw data collected during execution) and calculated metrics
(derive from data in base metrics usually for test reporting purpose)
Metric Types
8. 8
8
Important Metrics (1)
1. Test cases:
• How many test cases designed per requirement?
• How many test cases executed/pending?
• How many test cases passed/failed/blocked? How many failed due to
one particular defect?
2. Defects:
• Total number of defects identified
• Defect distribution per Severity, per Priority, per Component/Module
• Defect Aging
• Defect Density and Defect Density in released code
3. Automation:
• Number of defects found in automated tests (Are they still useful?
Watch for pesticide paradox!)
• Percent of automated test coverage
• How long it takes to run test plan and how often we do it?
4. People/QA Team:
• Execution by user: How many test cases executed per person?
QUALITY. PRODUCTIVITY. INNOVATION.
9. 9
9
Important Metrics (2)
1. Defect distribution by status and phase
• Expected to decrease as the software quality improves toward the
project end
2. Defect open and close rates
• Insight into the ability of testers and developers to work together to
identify and address software issues
3. Defect removal efficiency (DRE)
• Rate at which team is able to adequately fix identified program flaws
4. Burn down chart
• Visual representation of the amount of work yet to be completed
5. Defect severity index
• Shine some light on the effectiveness of development team
QUALITY. PRODUCTIVITY. INNOVATION.
10. 10
10
Important Metrics (3)
1. Number of issues reported by customers
• Determine effectiveness of QA departments (risk-based testing to reduce
gaps)
2. Mean Time to Detect (MTTD) and Mean Time to Repair (MTTR)
• MTTD - how long it takes QA professionals to find a problem
• MTTR - amount of time needed to effectively address it
3. Number of system outages and downtime
• The number/frequency of system outages and length of downtime
experienced by end user
4. COQ (Cost of Quality) and COPQ (Cost of Poor Quality)
• COQ: Total effort put into quality-related activities (development, testing,
reviews)
• COPQ: Cost of fixing defects, updating docs, re-testing, patch distribution
QUALITY. PRODUCTIVITY. INNOVATION.
11. 11
11 QUALITY. PRODUCTIVITY. INNOVATION.
A few tips how to identify correct metrics:
• Set the target audience
• Define the goal for metrics
• Add only relevant metrics based on your project needs (less is more)
• Analyze the cost benefits aspect of each metric
How to identify the right metrics?
12. 12
12 QUALITY. PRODUCTIVITY. INNOVATION.
Steps to create Metrics Report
Identify the key
processes to be
measured
Set the target value for
metrics (baseline)
Identify how often to
collect data and who
should do it
Collect actual data,
calculate and interpret
the results
Create report and
present to stakeholder
with reasonable
conclusions
If things don’t look
good, investigate
reasons and suggest
how to improve
13. 13
13
Metrics Report Example
QUALITY. PRODUCTIVITY. INNOVATION.
METRIC ACTUAL VALUE TARGET
Total number of open defects 18 20
Incoming defect rate 7
Defects found by automation
tests 3
Breakage introduced by new
features (regressions) 5
Outgoing defect-fix rate 5
Severity 1 (catastrophic outage) 0 0
Severity 2 (severe breakage) 2 0
Severity 3 (moderate issue) 3 2
Severity 4 (cosmetic issue) 12 Best Effort
Documentation defects 40 Best Effort
Defects found by customers 4 Best Effort
MTTR (days) 7 10
Customer satisfaction 4.3 4.8
14. 14
14
Metrics – Graphical Representation (1)
QUALITY. PRODUCTIVITY. INNOVATION.
Defects per Severity
Total number of open defects
15. 15
15 QUALITY. PRODUCTIVITY. INNOVATION.
Metrics – Graphical Representation (2)
Open vs Resolved Defects
Defects per Component:
1. Cumulative Chart
2. JIRA Pie Chart
16. 16
16 QUALITY. PRODUCTIVITY. INNOVATION.
• Yes, metrics can cheat too! Usually when they are taken out of
context…
• Need reliable input data
• Cheat accidentally (incorrect input data) or cheat on purpose
(‘tweak’ your queries to get better stats)
• True story example (on the next slide)
• Project metric that did not take into account that there
was a new defect tracking tool introduced!
Can Metrics Cheat?
19. 19
19 QUALITY. PRODUCTIVITY. INNOVATION.
• The bigger the number – the better we are ?
• The component where we find most defects is the worst coded
• Metrics are there for managers to point fingers at people
• “All my filters and queries are correct but I still get very strange
output results”
• There are no "universal metrics“
Common Deceptions about Metrics
22. 22
22 QUALITY. PRODUCTIVITY. INNOVATION.
• If you have any
questions,
please contact
the presenter:
Thank You!
Aleksandra Petrovic
Test Consultant
aleksandra.petrovic@endava.com