From the original EuroSTAR 2002 Keynote abstract: The value of testing is a difficult thing for a tester to define – as a supplier we are not independent! It might help if we think like marketers for a moment and view testing as a service (or product?) provided by a supplier to its customers. Would that help? Maybe if we think a little more like marketers we can enhance not only the value of our testing, but also of our customers’ perception of it? Who are the customers that benefit from our testing efforts? The sponsors of a development project, the business users, project management, development staff and others are all our customers in (sometimes) different ways. What do they VALUE in testing? We must understand our product too. Where does the value actually get ADDED? Running a test has little value if it isn’t repeatable or we don’t remember which tests pass and fail. Perhaps it is not doing testing but the information we generate that matters? Are test results the only product of value? Our product (or is it a service?) comes in different flavours: static and dynamic, functional and non-functional, planned and exploratory, manual and automated testing are sometimes alternatives, sometimes complementary. Are we confusing our customers by offering such diversity in the market? This talk looks at the value of testing through the eyes of a marketer and offers some insights to how we might better brand, market and deliver test to our markets. 3 Key Points * How do we measure the value of testing? * What we need to change to add value to our testing. * How we can improve our customers’ perception of the value of testing?

1. ©2003 Systeme Evolutif Ltd Slide 1we’ll change the way you think about testing
What Is the Value of Testing
and How Can We Increase It?
Paul Gerrard
Technical Director
Systeme EvolutifSysteme Evolutif Limited
9 Cavendish Place
London W1G 0QD, UK
Tel: +44 (0)20 7636 6060
email: paulg@evolutif.co.uk
http://www.evolutif.co.uk/

2. ©2003 Systeme Evolutif Ltd Slide 3we’ll change the way you think about testing
Themes
“[A cynic is] A man who knows the price of
everything but the value of nothing”
Oscar Wilde (1854-1900), Lady Windermere’s Fan.
“To win without risk is to triumph without
glory”
Pierre Corneille (1606-84), The Cid.

3. ©2003 Systeme Evolutif Ltd Slide 4we’ll change the way you think about testing
Agenda
Let’s play Marketing
Quantity, Quality, Process or Product – Where is
the Value?
The Most Valuable Product of Testing is Project
Intelligence
Influences on the Value of Testing
Increasing the Value of Testing
Process “Improvements” and the Value of Testing
Summary.

4. ©2003 Systeme Evolutif Ltd Slide 5we’ll change the way you think about testing
Let’s Play Marketing
We need to consider:
– Who are the customers of testing?
– What exactly does the customer value?
– How can we increase the value of our testing?
– How can we increase their appreciation of our testing?
– How do we select an “improvement” to make?
– How do we judge the value of an “improvement”?

5. ©2003 Systeme Evolutif Ltd Slide 6we’ll change the way you think about testing
Quantity, Quality, Process or
Product – Where is the Value?

6. ©2003 Systeme Evolutif Ltd Slide 7we’ll change the way you think about testing
Mine clearing
According to captured records, there are
100 mines in that minefield
We have identified and removed seven
mines
I crossed it once but don’t remember how
Is it safe?

7. ©2003 Systeme Evolutif Ltd Slide 8we’ll change the way you think about testing
Test activity and information
I ran some tests
– But I can’t remember what happened or how I could
repeat those tests
– Was this testing valuable?
I ran a hundred tests and the system failed 23
times
– I can’t remember how I did it or where the system
failed
– Was this testing valuable?
The ACTIVITY of testing only has VALUE if it
generates INFORMATION (this is the deliverable).

8. ©2003 Systeme Evolutif Ltd Slide 9we’ll change the way you think about testing
How much detail?
Which is best?
– “We ran lots of tests”
– “We found lots of faults”
– “10% of our tests caused the system to fail”
– “This test case caused a failure”
– “This set of test steps caused the system to fail at
precisely this point, in precisely this way…”
Test records should be detailed enough to plan,
design, implement, execute tests and interpret test
results and reproduce failure
– AND NO MORE.

9. ©2003 Systeme Evolutif Ltd Slide 10we’ll change the way you think about testing
Quantity or quality?
Which would you pay more for?
– 100 page test plan or 200 page test plan?
– 100 incident reports or 200 incident reports?
– 100 requirements faults or 100 coding faults?
– 100 random tests or 100 ‘designed’ tests?
Quantity is one thing, but the nature and quality of
the information and the way it was prepared
makes a huge difference to its value.

10. ©2003 Systeme Evolutif Ltd Slide 11we’ll change the way you think about testing
Testing generates “project
intelligence”
If you don’t know where you are, a map won’t
help
If you don’t know the status of the deliverables,
the best plan won’t help you manage your project
A project is like driving off-road in the dark
– Testing provides the headlights for the journey
Testing provides data on the status of deliverables
and generates “PROJECT INTELLIGENCE”
Stakeholders and management need intelligence to
make decisions, command and control the project.

11. ©2003 Systeme Evolutif Ltd Slide 12we’ll change the way you think about testing
Does testing ‘change’ anything?
Product Testing
Product
(unchanged)
Valuable
Stuff
Testing doesn’t
change the product in
any way. We do it to
get the ‘valuable stuff’.

12. ©2003 Systeme Evolutif Ltd Slide 13we’ll change the way you think about testing
What is the valuable stuff?
Test deliverables (infrastructure, test plans,
specifications, scripts, data, results etc…)
TEST RECORDS – test documentation (but
do only testers read it?)
PROJECT INTELLIGENCE – rarely
documented as such though.

13. ©2003 Systeme Evolutif Ltd Slide 14we’ll change the way you think about testing
Military intelligence?

14. ©2003 Systeme Evolutif Ltd Slide 15we’ll change the way you think about testing
The Most Valuable Product of
Testing is Project Intelligence

15. ©2003 Systeme Evolutif Ltd Slide 16we’ll change the way you think about testing
Documentation
PRODUCTS
– Test Strategy
– Test plans, designs, specifications
– Procedures/scripts, expected results
– Actual results
– Incident reports
– Log
– End of Phase Report
– Etc.
TANGIBLE, BUT LESS VALUABLE.

16. ©2003 Systeme Evolutif Ltd Slide 17we’ll change the way you think about testing
Test infrastructure
PRODUCTS
– Test process
– Configured test hardware
– Test databases
– Tools
– Manual and automated tests
– Skilled resources
– Etc…
TANGIBLE, USUALLY AN OVERHEAD
– (unless these are required deliverables).

17. ©2003 Systeme Evolutif Ltd Slide 18we’ll change the way you think about testing
TEST RECORDS
DATA
– How the product SHOULD work (test specs)
– How the product ACTUALLY works (test results)
– Where the product is STABLE (incident trends)
– Where the product is UNSTABLE (incident trends)
– What they have COVERED in the product (test log)
– What they have NOT COVERED (test plan – test log)
– How (and where) the product FAILS (incidents)
TANGIBLE AND VALUABLE, BUT…
– (valuable to testers, developers, NOT management).

18. ©2003 Systeme Evolutif Ltd Slide 19we’ll change the way you think about testing
PROJECT INTELLIGENCE
KNOWLEDGE OF PROJECT RISKS/BENEFITS
– RISKS
» Requirements are poor and/or unstable
» Less of the product is ready, product is not as good as planned
» They should have been less ambitious, done it differently
» The project will slip
– BENEFITS
» Some features/benefits are available
» Other features/benefits are blocked by risk uncertainty
INTANGIBLE AND VALUABLE
– Crucial to management and stakeholders
– Intelligence often arrives TOO LATE TO BE OF USE.

19. ©2003 Systeme Evolutif Ltd Slide 20we’ll change the way you think about testing
Where the value of testing lies
LOW VALUE
– Documentation
– Test infrastructure
**UNLESS these are
specific objectives or
deliverables of project
HIGH VALUE
– Test Records
– Project Intelligence
We can increase the value of testing MOST by:
• Improved INTELLIGENCE
• Providing INTELLIGENCE EARLIER.

20. ©2003 Systeme Evolutif Ltd Slide 21we’ll change the way you think about testing
Influences on the Value of
Testing

21. ©2003 Systeme Evolutif Ltd Slide 22we’ll change the way you think about testing
Intangible value, tangible cost
Testers think testing is good and has undeniable
value, but we are not independent!
Not everyone thinks that (software suppliers, in-
house developers?)
If you are not a tester, the value seems intangible
– But the cost IS tangible
– Exercise, flossing, house maintenance, life insurance,
testing can all be put off till another day
– If you don’t do it now you store up problems for later
A bit like a lifestyle choice – it’s human nature to
put things off.

22. ©2003 Systeme Evolutif Ltd Slide 23we’ll change the way you think about testing
Key factors in the value of testing
Adding value depends on how well we achieve
and communicate:
1. Coverage
2. Incidents
3. Risk-based testing
4. Benefits-based testing
Time to deliver that value depends on the test
process (more later)
Willingness of the organisation to accept and act
upon project intelligence is key (more later).

23. ©2003 Systeme Evolutif Ltd Slide 24we’ll change the way you think about testing
Coverage
Breadth of coverage
– Are all aspects of the product tested?
Depth of coverage
– Are critical aspects of the product tested more
thoroughly?
Can the measure be applied systematically?
Is the measure relevant to the faults to be
detected or the risks of concern?
Can we communicate coverage in a meaningful,
understandable way?

24. ©2003 Systeme Evolutif Ltd Slide 25we’ll change the way you think about testing
Incidents
Are the failures of most concern being
found and logged as incidents?
Can these failures be reproduced easily?
Are the incidents being addressed in the
right order?
Are the incidents of most concern being
resolved quickly?

25. ©2003 Systeme Evolutif Ltd Slide 26we’ll change the way you think about testing
Risk-based testing
Are product risks identified early in the project?
Are stakeholders involved in the decision to rank
these risks?
Is the decision to include risks in scope for testing
done by consensus?
Is the decision to de-scope risks done by
consensus?
Is test design driven by the need to address risk?
Is test progress reporting based on risks
addressed?

26. ©2003 Systeme Evolutif Ltd Slide 27we’ll change the way you think about testing
Benefit-based testing
Does the project have cardinal objectives
(benefits) defined?
Are all risks related to the benefits?
Are tests related to the risks that block the
benefits?
Is test progress reporting based on the
benefits that are threatened or available?

27. ©2003 Systeme Evolutif Ltd Slide 28we’ll change the way you think about testing
Increasing the Value of Testing

28. ©2003 Systeme Evolutif Ltd Slide 29we’ll change the way you think about testing
Contribution of the test process
Does testing generating valuable intelligence?
Does testing generate intelligence early enough?
Is testing flexible enough to react to new risks?
Are the test communication channels clear?
Improvement types:
– Effectiveness
– Efficiency
– Perceived value of testing.

29. ©2003 Systeme Evolutif Ltd Slide 30we’ll change the way you think about testing
Effectiveness improvements
Improvements to “intellectual” test activities
enhance the quality of intelligence
Implementing these activities earlier make
intelligence available earlier
Examples:
– Formal test design improves coverage, and improves the
quality of test records
– Improvements that promote earlier test activities tend
to make the value available earlier (e.g. more reviews).

30. ©2003 Systeme Evolutif Ltd Slide 31we’ll change the way you think about testing
Efficiency improvements
These reduce the cost of doing testing
Improvements to the mechanical or clerical
aspects of our test process make us more efficient
(but don't affect the value of testing)
For example
– Using a tool to automate tests might make it cheaper to
do a task, but does it add value?
– It might save time, allowing us to do other useful
activities
Ultimately, effectiveness improvements are more
valuable.

31. ©2003 Systeme Evolutif Ltd Slide 32we’ll change the way you think about testing
Organisational culture
Willingness to accept and act on the
intelligence that testers generate is key
Primarily a cultural issue
If the intelligence isn’t valued by the culture
- it doesn't matter what value it has
We need to EDUCATE our customers.

32. ©2003 Systeme Evolutif Ltd Slide 33we’ll change the way you think about testing
Increasing the perceived value of
testing
No such thing as “absolute value” of testing
We must work on the PERCEIVED VALUE to
the customers of our testing:
1. Improve the value in line with their expectations
– Provide richer, deeper, focused, relevant information
– “Create more complex, subtle, interesting wine”
2. Increase their appreciation of what we do
– Educate them to understand testing products
– “Train their palate to detect the complex flavours”
Improvements must address both issues.

33. ©2003 Systeme Evolutif Ltd Slide 34we’ll change the way you think about testing
Process “Improvements” and the
Value of Testing

34. ©2003 Systeme Evolutif Ltd Slide 35we’ll change the way you think about testing
Test process or test product
assessment?
Assessments of the “Test Process” may miss the
point
“Process healthchecks” are useless because they
don’t focus on the VALUE of test deliverables
Assessment of the perceived value of Test
Products must be a key driver
We must focus on process changes that:
– add value to the test products
– enhance the perceived value of test products
– make test products available earlier.

35. ©2003 Systeme Evolutif Ltd Slide 36we’ll change the way you think about testing
The challenges of testing
improvement
Barriers are organisational, personal, not technical:
– changing management perceptions so they support
testing and improvement
– overcoming management and practitioner resistance to
change
– design and implementation of workable processes and
management controls
People, not tools, implement test strategies
Change is easier if people see VALUE being added.

36. ©2003 Systeme Evolutif Ltd Slide 37we’ll change the way you think about testing
Problems or symptoms?
Need to separate problems from symptoms:
– “Management doesn't understand the value of testing”
– “The cost of testing is high but difficult to pin down”
– “Developers, testers, users never trained”
– “The quality of the product delivered into testing is
poor, so takes longer to system test”
– “The perceived value of testing is low”
Need to understand symptoms to identify the
source of problem and fix the problem
Improvements often tackle the symptoms only.

37. ©2003 Systeme Evolutif Ltd Slide 38we’ll change the way you think about testing
Right improvement mix is key
A mix of improvements is most likely to be
required:
– management awareness
– tester training
– developer training
– improved definition of the test stages and their
objectives
– measurement of the quality of the product at each stage
– etc. etc.

38. ©2003 Systeme Evolutif Ltd Slide 39we’ll change the way you think about testing
Constraints and expectations
Not all improvements a good idea straight away
– some improvements are expensive
– some save time, but bring dramatic change
– some improve the quality of the testing, but take longer
to implement
– some tidy up ‘loose ends’ but have negligible value
Very few improvements save time, improve quality,
cause minimal change and pay back after two
weeks
Need to set/agree clear management expectations.

39. ©2003 Systeme Evolutif Ltd Slide 40we’ll change the way you think about testing
Do tools increase the value of
testing?
With tool vendors present, I’d better be careful!
Need to distinguish between
– Quantity, quality and the information provided
A tool might allow you do to testing faster, but it
probably won’t increase the value of tests
It might make a task interesting, so a valuable task
gets done
It might give you more time to do other valuable
tasks.

40. ©2003 Systeme Evolutif Ltd Slide 41we’ll change the way you think about testing
To Summarise…

41. ©2003 Systeme Evolutif Ltd Slide 42we’ll change the way you think about testing
How to increase the value of
testing
The value of testing lies in the project intelligence
it generates
Produce documentation with purpose:
– To allow testers to do their job (THE MEANS)
– To communicate intelligence (THE END)
Process ‘improvements’ selected to deliver better
intelligence, earlier, faster and cheaper
You need to train your customers to appreciate
the project intelligence that testers produce.

42. ©2003 Systeme Evolutif Ltd Slide 43we’ll change the way you think about testing
Themes revisited
“A poor tester knows the challenge and cost of testing but not
its value”
– Don’t be cynical – testing isn’t just a costly activity to be
gotten through, testers should aim to deliver value
“To win without risk is to triumph without glory”
– Risks are inevitable - the testers’ role isn’t to avoid them,
but inform management and reduce their uncertainty
– Satisfaction (and recognition) comes from overcoming
risks, not doing the mundane tasks of development
– Make your management aware of software product risks
and report progress towards their goal, not just incidents.
Testers deal in success, not just failure.

43. ©2003 Systeme Evolutif Ltd Slide 44we’ll change the way you think about testing
www.evolutif.co.uk
www.riskbasedtesting.com
What Is the Value of Testing
and How Can We Increase It?
Paul Gerrard