2. Cost of Defects Fixing
It can take one person 10 minutes to find a
defect in requirements, 10 minutes to fix the
document. Cost £10 per defect.
Finding the same defect in Functional Test
and then Fixing. Cost £211 per defect.
Finding the same single defect in System
Integration Testing. Cost £587 per defect.
Cost following delivery £? + £Reputation.
2
3. Add Success to Testing
There are usually quick wins, however
do we review the success of our testing?
3
4. Quick Wins
How many defects could have been
avoided by changing the review
practices and acceptance of
Requirements into Development?
Measure by Audit of 1 product:
1. Audit Defect Logs
2. Audit Customer Service Reports.
3. Audit Requirements and their Review.
4
5. RCA vs TEA
Root Cause Analysis (RCA) allows us to
learn, rectify and avoid future defect
injection. Providing we use the data.
Test Escape Analysis (TEA) in contrast
allows us to become more efficient at
testing and avoid future defects from
escaping detection. Provided we use
the data.
5
6. TEA Applied to Defect History
Applying TEA to defect history can help
to analyse where resources and
changes in procedure should be applied
for maximum return on investment.
Normally one would focus upon:
a single typical representative project.
High and Critical defects, to reduce sample size.
TEA can be integrated into the defect
tracking system.
6
7. Purpose of TEA
Root Cause Analysis is aimed at the code
programming – why was the defect
introduced.
Analogy why did the tightrope walker fall
TEA in contrast is about why a defect was not
detected.
Analogy why did we not catch the falling tightrope
walker.
We do TEA to learn how to detect more
defects and how to detect them sooner and
smarter. The ultimate goal is to drive down
the number of defects leaking to the customer
by finding those defects ourselves first.
7
8. Benefits
The benefits of early detection are:
Reduced costs.
Improved reputation.
Reduced loading on engineers.
Improved output from testing team.
8
9. Trends
With TEA we are interested in trends.
Is there a pattern in the defects that are being let through?
Is there a type of test that could help catch more defects.
We target TEA at a sample of defects. This sample
includes:
Defects raised for example by the Customer Which are either
Critical or
High
The knowledge that we gain from TEA is applied to future
Test Plans:
Trend Reports in product TEA - produced by the QA Engineers.
Product Teams are expected to produce their own reports on their
individual trends and apply lessons learned to their Test Plans.
9
10. Resources
TEA Meetings are linked to Company Targets
Are to occur at least monthly for each team.
Expectations
For this activity teams usually need a developer and
tester to attend the monthly meeting.
In practice a TEA meeting usually takes an hour per
month.
In looking at a TEA report one may initially spend
around 15 minutes per record, however it is possible
to do these in just 5 minutes each.
Remember we are looking at a sample – not all.
10
11. Managing TEA
There are 2 ways that TEA
meetings run. Either:
Hold a meeting at which the TEA
reports are completed.
Complete the reports before the
meeting and at the meeting these are
audited.
The TEA meetings ensure that there
is a consistent approach and it also
acts as a mentoring exercise in
supporting the rollout of TEA.
11
12. TEA Assumptions
In carrying out TEA there is an assumption that:
The review team have some minimal familiarity with
the code impacted by the defect.
The review team are familiar with testing techniques.
This can be obtained by either:
Reading BS7925-2;
Laying on an internal course 2 or 3 days;
Attending an ISEB/IQTB Test Foundation Course.
12
13. Management of TEA and RCA
Root Cause Analysis if employed
improves development through the
application of lessons learned. RCA is
usually managed through the defect
tracking tool employed.
TEA in contrast improves testing.
However it to can be managed through
the defect life cycle.
13
14. Summary of TEA Data Fields
The following Fields are used within TEA:
TEA Status
Reason Introduced
Development Phase
Code Designed From
Test Environment
Test Tools Required
Test Technique
Test Type
Free Text Notes Field
All fields are mandatory, but there is a “not
known” option available.
14
15. Reason Introduced
A shared field with the main Defect and RCA. The record can be changed
from any of the reports.
Fields include:
Code Missing
Code not fully built or tested Stage Defect Introduced:
Coding Error “Earliest stage in the processes
Design Incorrect that the defect could have been
Design Missing or Incomplete prevented”.
Design Unclear Reason Introduced:
Environment Unavailable “Provide finer detail as to the initial
root cause of the defect”.
Initial Fix Incomplete
Initial Fix Incorrect
Requirement Incorrect
Requirement Missing or Incomplete
Requirement Unclear
Standards Not Followed
Typing Mistake
Other (free text)
15
16. Development Phase
Requirements Review,
Design Review, What was the appropriate appraisal
process (testing or reviews) that
Code Review, would have detected this defect
earlier?
Unit Test,
Component Test,
Component Integration Test,
System Test
16
17. Code Designed From
API Spec,
What was the source used,
Detail Design, from which the code with the
defect was created?
Requirement,
Standards,
Functional Specification,
Developer Led Requirements,
Other (Selecting Other allows a free text
field to be used).
17
18. Test Environment
Sufficient,
Customer Device,
External Server,
System Component,
Stub,
HRP (Hardware Reference Platform) Feature,
OS / HRP Config,
Platform, What other Test Environment would have
helped to detect the bug earlier? (other than
External Device, what was planned or available)
Live Environment,
Other (Selecting Other allows a free text field to be
used).
18
19. Test Tools Required
What other Test tools would have helped to
Sufficient, detect the bug earlier? (other than what was
planned or available)
Test Execute Framework Extension, Test
Driver Extension,
Code Analyser,
Protocol Tester,
Other (Selecting Other allows a free text
field to be used).
19
20. Test Technique
Boundary Analysis (See BS7925-2)
Cause Effect Graphing (See BS7925-2)
Classification Tree
Equivalence Partitioning (See BS7925-2)
Out Of Memory
Random Testing
Risked Based Targeted Exploratory Testing
State Transition Testing (See BS7925-2)
Static Analysis (Reviews and the use of Static
Analysis Tools)
20
21. Test Type
API Validation Test In Real Environment
Binary Compatibility testing Test on Emulator
Compliance Testing
Dependency Testing
Test Tools – Code Coverage
Design Verification Test Tools –Static Binary
Error Guessing Compatibility
Exploratory Testing Test Tools – Codenomicon
Functional Testing Test Tools – Copyright
Interoperability Testing Test Tools – Coverity
Intrusive Testing
Performance Testing
Test Tools – Lint
Platform Verification Testing Test Tools – Win Runner
Product Configuration Testing Test Tools – Load Runner
RAM / ROM Testing Test Tools – Other
Recovery (Robustness) Testing Open Source Check
Requirements Validation
Security Vulnerability Testing
Static Analysis
Static Testing
Negative Testing
21
22. Evidence from TEA
TEA can be useful for highlighting unforeseen problems. We might have
predicted that inadequate negative tests are being written (and this is
supported by TEA in this example) but there is also much evidence to
suggest that we need to write more positive test cases.
450
Test Techniques - Test Technique that would find the Defect
418
400
350
Nos. of Defects
300
244
250
200
150
96 93
100 61 56 53 46 44 37 35
50 26 17 16 15 7 5 5 2 0
0
Re st
ua s
Eq ery
lity
Re r if
a t a rt
cy
fo e r)
st
n
e
ce
ga est
BC
ns
dA M
pe ity
S e ls
pe e
gr is
M view
St a nc
Co s sio
e
A c l Te
nc
Ve
o
O
Re lys
en
Pe lia n
vP
bi
ra
De cu r
eT
Pl (o th
v
Ne e T
To
O
In epta
co
ra
rm
eT
nd
n
rm
t iv
ui
e
iv
ic
p
m
at
an
sit
rfo
c
ro
un
at
St
Po
te
Bo
22