Testing has evolved. From being an activity that “just had to be done” to a profession. And from just a profession to a fizzing profession with its own methods, tools and techniques on test execution, management and even on strategic levels. Still, a question often posed to the test professionals is: “Why does testing cost so much money?”.

1. TESTING
Identifying the benefits
of testing
René Tuinhout, Richard Janssen

2. 02
Testing has evolved. From being an activity that “just had to be done” to a profession. And
from just a profession to a fizzing profession with its own methods, tools and techniques
on test execution, management and even on strategic levels. Still, a question often posed
to the test professionals is: “Why does testing cost so much money?”.
The perception of testing is that it’s expensive. While we, the test profession, have become
quite versed in explaining why testing has a certain cost, often we have been not been
able to change this question into the far more interesting question: “What does testing
contribute to the stakeholders?” Or, in other words “What costs does testing prevent?”.
By answering this particular question to our stakeholders we would rapidly change the
perception of testing as a necessary evil into that of testing as a contributing factor in the
production process.

3. 03
And we do have the tools and knowledge available to answer this question. Some attempts have even been
made: For example, it´s possible to create a ‘history database’, a database containing data regarding
previous projects and failures, and their costs. By analysing the database for failure patterns and connected
costs, failures occurring often and leading to grave losses can be identified. Then, tests can be designed to
try and detect those failures before they end up in production. Basically the rationale used boils down
to:“after the last project, X failures of type Y occurred in production, costing Z. In this project, by testing we
found A Y-type failures, so we saved our organisation A*(Z/X)”. However, setting up such a database takes
time. And are failures found in different applications really that alike? Can they really be compared to one
another that easily? Too often, these questions lead to lengthy and unresolved debates.
This article suggests another way of calculating the benefits of testing. One, in which the costs saved are
not directly related to failures (expected to be) found, but to risks. It adds to a technique we as testers use
already: risk and requirement based testing.
What we as testers have been doing in the past years, is finding out the risks involved in applications and
linking a test priority to that risk and the associated requirement(s). Is the risk high? And is the
requirement(s) an absolute must? Then it is something that must be tested!
The process of finding risks and prioritising them has evolved in the years past1. Before, it was common
practice to interview one or more stakeholders and discuss the risks involved in the application. This has
evolved into the practice of interviewing stakeholder management not regarding specific risks, but regarding
the risks on a conceptual level: All management involved (stakeholders) together create a “risk
thermometer” specific for their organisation. With risk being defined as probability times impact,
management is asked which high level impact(s) they can envisage for the application(s) under test. Being
smart testers, we do not only ask for impact in terms of money, but also in terms of loss of image and non-
compliance to law or regulations. Stakeholder management will, after some discussions, come up with an
“impact-thermometer” (Figure 1).
Impact
Non-compliance to
Loss of Money Loss of image
law/regulations
* All potential clients and
existing clients affected
*Total loss of >=M€5
10 *Loss of >= k€100 per incident * In the newspapers A,B,C,D or E Board of diectors in jail
9 ... * In two main newspapers ...
8 ... ... ...
7 ... ... ...
6 ... ... ...
* One customer affected
5 *Total loss of >k€1 and<=k€10 and displeased * warning is received
4 ... ... ...
3 ... ... ...
2 ... ... ...
1 No significant loss None None
Figure 1: The impact-thermometer, an example.
1
This article will focus on the risk part of risk & requirement based testing.

4. 04
After having created the impact-thermometer, stakeholder management is asked to create the probability
thermometer in much the same way. Not only the frequency of certain actions should be considered in this
activity, but also the urgency: the probability of a certain action occurring shortly after going live.
For example: An end-of-year-run that’s executed once a year on January 2nd has a very low (annual)
frequency, which would suggest a low score on the probability-thermometer. However, when the go live is
scheduled for December 27th, one of the first things that’s going to happen after the go live is this end-of-
year-run. Which should increase the score on the probability-thermometer (Figure 2).
Probability
Frequency Urgency
5 * Occurs >= 10 times each day * will occur within 1 week of going live
4
3 * Occurs once a week * will occur within three months of going live
2
1 * Occurs once a year * will occur more than a year after going live
Figure 2: The probability-thermometer, an example.
By combining the probability and the impact thermometers the risk thermometer is produced (figure 3).
Loss of Money
Impact
Loss of image
Non-compliance to Risk
law/regulations
* All potential clients and
existing clients affected
*Total loss of >=M€5
10 *Loss of >= k€100 per incident * In the newspapers A,B,C,D or E Board of diectors in jail 10
9 ... * In two main newspapers ...
9
8 ... ... ...
7 ... ... ...
6 ... ... ...
5 *Total loss of >k€1 and<=k€10
* One customer affected
and displeased * warning is received
8
4 ... ... ...
3 ...
2 ...
...
...
...
...
7
6
1 No significant loss None None
5
Probability
Frequency Urgency 4
5 * Occurs >= 10 times each day * will occur within 1 week of going live
4
3
3 * Occurs once a week * will occur within three months of going live 2
2
1
1 * Occurs once a year * will occur more than a year after going live
Figure 3: Combine probability and impact to risk.

5. 05
Based on the risk-thermometer, discussions with stakeholder
management can commence, leading to a MoSCoW division of the Risk
risk-thermometer: Which risk-rating would, for you as the
10
stakeholders, lead to a “must-test” risk? And which to a “should-
9 Must test
test”? (figure 4).
8
7
Having created the risk thermometer with management, the risk 6
Should test
perception of management has been visualised and made 5
Could test
measurable. Consequently, an organisation-specific model has been 4
created: A model indicating the managements view on risks on a 3
2 Won’t test
conceptual level.
1
This model can be used to create an inventory of the actual, real,
risks in the application under test itself. This inventory is created in
discussions with stakeholder staff, the actual users2 of the application Figure 4: MosCow division in the
under test. In a workshop risks are identified and, using the risk thermometer.
probability- and impact-thermometers, the degree of risk can be
calculated for each identified risk. This results in an overview of
Risk 17: Calculation
identified risks, and a degree of risk per each risk identified (figure 5). Risk of fee failure. Affects
all customers,reaches
main newspapers.
10
Abiding by this process has enabled us as testers not only to 9
Must test
prioritise tests better and in alignment with stakeholders and 8
stakeholder management views. It also enabled us to detect possible 7
Should test
6
threats to the test process itself in an early stage: If, for example, the
5
majority of the identified risks are in the must test area, and the time 4
Could test
provided for testing is very limited, insight to stakeholder 3
management can be created (using the thermometers shown), 2 Won’t test
demonstrating either that more testing time or testing staff is 1
needed. Or that stakeholder management might want to reconsider
their MoSCoW-division of the risk thermometer. After all, by shifting Figure 5: Identified risks on the
the risk-ratings, some risks end up in other MoSCoW-categories, risk thermometer, represented by
leading to another test-effort. (In the example in figure 6, stakeholder red dots. Risk 17 shown as an
management shifting the Must-test boundary upward will lead to less example.
must-test test cases, but to more should-test test cases3.)
Based on the organisation specific risk model created by Risk
management, and based on the specific risks prioritised according to
10
the model created by staff, testing can now be prioritised and 9
Must test
managed. 8
7
Should test
But, there’s another consequence of using the process described 6
above: By creating test cases in such a way that it’s traceable which 5
Could test
risk(s) are associated with a test case, now reporting on risk 4
3
coverage becomes possible: Instead of reporting “test case XYZ
2 Won’t test
failed” as was the case in the early years of testing, nowadays “risk
1
ABC isn’t (fully) covered yet”4 can be reported. This way of reporting
on risk coverage is, compared to the reporting on failures, far more
insightful for testing’s stakeholders. Figure 6: A shift in the risk-model,
decided upon by stakeholder
management.
2
By actual users all (future) application users are ment, e.g. front end users, maintenance staff, management in a user-
role. Depending on the test level to execute, development staff could, of course, also be involved in the workshop.
3
The test plan, of course, should define which techniques (and which test coverage and depth) should be used for Must-
test test cases, for Should-test test cases etc. For this article it’s assumed that testing a Should-test test case requires
less effort than testing a Must-test test case (as is usually the case).

6. 06
However, even creating this insight does not provide an answer to the question posed at the outset of this
article:“What costs does testing prevent?”.
‘Adhering’ to the method described above, a short answer to the question could be:”In the impact
thermometer, a ‘loss of money’-column is defined. So when a test case fails, find the risk related to the test
case, then find the loss of money related to this risk, and report it as costs prevented by testing.” And
basically, it is that simple!
Unfortunately, this simple answer often leads to discussions: It will be argued the test case failed and
therefore indeed the risk is not covered, but that the failure is not that severe. Suggesting not the entire risk
would actually have occurred when the failure would have ended up in production. Furthermore, the loss of
image and the non-compliance to law/regulations often is not defined in a monetary way. This can lead to
discussions about which costs were prevented by detecting a failure related to the risks defined in these
ways.
To avoid these discussions, we propose to add a bit more information to the risks identified: The minimum
and maximum cost of failure. While in discussion with stakeholder management, apart from creating the
classic organisation specific risk model, in this model the minimum and maximum costs related to the risk
classes identified should be included (figure 7).
A last suggestion: When minimum and maximum costs per failure detected are included in the defect
Impact
Non-compliance to Minimum Maximum
Loss of Money Loss of image
law/regulations cost cost
* All potential clients and Total value of
existing clients affected organisation
*Total loss of >=M€5 (wash-out)
10 *Loss of >= k€100 per incident * In the newspapers A,B,C,D or E Board of diectors in jail M€5
9 ... * In two main newspapers ... M€3 M€5
8 ... ... ...
7 ... ... ...
6 ... ... ...
* One customer affected
5 *Total loss of >k€1 and<=k€10 and displeased * warning is received
4 ... ... ...
3 ... ... ...
2 ... ... ...
1 No significant loss None None
Figure 7: The impact-thermometer including minimum and maximum cost, an example.
4
It’s assumed test case XYZ failed because of a fault, not because the test case itself was flawed. This assumption will
continue throughout this article: Whenever a failed test case is mentioned, it’s assumed it was a failure caused by a fault,
not by a flawed test case.

7. 07
This model should then be used as a basis for the discussions
with stakeholder staff in which the real risks are identified. For Risk 17: Calculation of fee failure. Affects all
customers, reaches all main news papers.
Risk Minimum cost €M 8, Maximum cost €M 12
these discussions we again suggest not only identifying the actual (costs like: damage control, clients leaving, no
new clients, damages)
risks, but also the minimum and maximum costs related to each 10
Must test
risk identified (figure 8). 9
8
This will lead to discussions, especially regarding the costs of less 7
Should test
6
measurable topics such as “loss of image” and “non-compliance
5
to law/regulations”. Questions such as “how to measure this in a Could test
4
monetary way” will arise. However, experience shows these
3
difficulties can be overcome in discussions amongst the
2 Won’t test
stakeholders, leading to defined minimum and maximum costs
1
the stakeholders agree to5.
Identifying these minimum and maximum costs in the early
stages of the test project allows the test manager to report not
Figure 8: maximum cost, an
only on faults detected or (via traceability to risks) on risks still not
example.
(entirely) covered. It also allows the test manager to report on the
minimum and maximum benefits testing has had for the
organisation. Not by reporting the benefits the test manager has
identified him/herself (which would lead to heated discussions),
but based on information provided by the stakeholders
themselves.
reporting, this has two advantages: The stakeholder representative(s) related to the test project for deciding
on what to do with defects found, are informed of the (mimimum and maximum) costs of a failure. This is
generally viewed as a welcome extra input for the decision process. Furthermore, including the minimum
and maximum costs per failure detected in the defect reporting allows the test manager to report on the
benefits testing has had fairly easily.
In short: Including the minimum and maximum cost of a risk occurring will take some time in the early
stages of a test project, when discussing the organisation specific risk model with stakeholder management
and specific risks with stakeholder staff. However, it prevents (much more) discussions later on in the
project about the costs involved if a risk occurs.
And, more importantly, it will allow us, professional testers, to finally report on the benefits of testing.
5
And, sometimes, even to an organisation specific model defining loss of image and non-compliance –risks in monetary
units.

8. René Tuinhout is a Practice Lead for Logica Netherlands’ Testing & Quality
Management practice. He´s a senior test advisor and test coach with over
14 years of experience in software testing. René advises organizations on
test management, test change management, structured testing, and he
manages test (change) projects. René tutors courses in Testing Techniques,
test management and several other test related subjects and is a speaker on
(inter)national conferences on Testing.
Richard janssen is a practice lead for logica netherlands’ Testing & Quality
Management practice. Richard works on strategy development, marketing
and solution development for logica’s t&qm business. Richard is a corporate
generalist with a broad view on techno – economical trends, organizational
change and business/technology integration. He has a degree in electrical
engineering & computer science from university of twente and an mba from
tsm business school.
Copyright statement
Logica Copyright © 2011 Logica
Tel: +31 (0)88 564 0000 All rights reserved. This document is protected by international copyright law and may not be reprinted, reproduced, copied or utilised in
whole or in part by any means including electronic, mechanical, or other means without the prior written consent of Logica.
testing@logica.com Whilst reasonable care has been taken by Logica to ensure the information contained herein is reasonably accurate, Logica shall not,
under any circumstances be liable for any loss or damage (direct or consequential) suffered by any party as a result of the contents of this
publication or the reliance of any party thereon or any inaccuracy or omission therein. The information in this document is therefore provided
on an “as is” basis without warranty and is subject to change without further notice and cannot be construed as a commitment by Logica.
Logica is a business and technology service company, employing 41,000 people. It provides business consulting, systems integration
and outsourcing to clients around the world, including many of Europe’s largest businesses. Logica creates value for clients by
successfully integrating people, business and technology. It is committed to long term collaboration, applying insight to create innovative
www.logica.com/testing answers to clients’ business needs. More information is available at www.logica.com.
CODE 3623 1011