How CXO's can reduce IT Project risk by improving software testing

1. info@kjross.com.au
www.kjross.com.au
1300 854 063
VERTICALS
MANAGEDSERVICES
TRAINING
RESOURCE
PLACEMENT
CONSULTING
“Less than 40% of all
respondees involve
testing at the concept
phase of a project”
WHITE
PAPER
High profile IT projects have been making headlines
withdisturbingregularityinrecentmonths,prompting
the question: who’s next? Responsible CEOs and CIOs
needtoreducerisk,negativePublicRelationsexposure
and project costs. So, is there anything that they can
do to reduce the IT project risk?
K.J. Ross & Associates (KJ Ross) are an independent
software testing organisation that often deals with
the end result of poor software developments or
application implementations. Following are five key
pointers for risk reduction in IT projects from a testing
perspective. These are compiled from the experience
of KJ Ross and from the annual Ross Report on the
Australian Software Testing Industry.
BEFORE YOU DO ANYTHING, START TESTING!
The 2010 Ross Report shows less than 40% of all
respondees involve testing at the concept phase of
a project, but over 70% aim to achieve this by 2012.
Why does early engagement matter? At the vendor or
product selection phase of an IT acquisition project,
setting measurable criteria for acceptance will assist in
providing a more objective basis for selection.
Before you sign a deal, get input from the testing team
on contracts, asking the question, does the vendor
have internal quality processes? Are key software
quality deliverables included? A typical issue is that
the requirement to provide evidence of system
testing is frequently left out of contracts. This results
in the acquirer having to engage in a very heavy user
acceptance test phase, which is really a duplication of
system testing, simply because the coverage of the
system by the vendor is unknown.
K. J. Ross & Associates
Software Testing & ICT Risk Mitigation
As the graph above illustrates, there is a significant shift over the
last 2 years getting testers involved earlier in the process. This has
been successful at reducing project costs and future plans involve
getting testers involved even earlier.
HOWCXO'SCANREDUCEITPROJECTRISK
BYIMPROVINGSOFTWARETESTING
DR. MARK PEDERSEN &
JOE GRIFFITHS 05/01/2012

2. info@kjross.com.au
www.kjross.com.au
1300 854 063
K. J. Ross & Associates
Software Testing & ICT Risk Mitigation
WHITEPAPER |
MEASURE TWICE, CUT ONCE: TESTING YOUR REQUIREMENTS
The Standish Group’s CHAOS report identifies the
top three factors determining project failure are lack
of user input, incomplete or incorrect requirements,
and changing requirements. Testing requirements for
correctness, consistency, and completeness before
going to development or to purchase a solution,
improves software quality through setting clearer
goals for developers, technical support and testers
later in the lifecycle. This reduces waste by avoiding
rework and extra redesign, development and testing
cycles. Requirements evaluation has been consistently
demonstrated to be 5 times more effective as a defect
discovery technique compared to typical black box
testing. The cost of defect detection and removal rises
exponentially throughout the software lifecycle: hence
early error detection through Requirements Evaluation
delivers a strong ROI compared to wearing the cost of
defect discovery and repair downstream.
The secret is to get the requirements right!
The graph below illustrates that 46% of the defects are injected before
coding begins
HOWCXO'SCANREDUCEITPROJECTRISK
BYIMPROVINGSOFTWARETESTING
The graph to the left shows that currently only 9% of defects are detected at the
requirements stage with the majority detected in the classical testing phases.
Defect Discovery Paradox
The later in the process that errors occur the more costly they are to fix.
Page 2
Economic Benefit of Early Defect Detection
As seen to the left there was a 40% saving in
costs through using early error detection.

3. Typically the real reason for implementing a change
in IT lies in the perceived risk of delivering the change
being less than the risk of not changing. It is therefore
essential to identify risks arising from the deployment
of an IT solution and adopt a proactive approach to
reducing these risks:
»» Establish a risk-based test strategy which identifies
where the greatest technical and operational risks
lie.
»» Incorporate risk-based test activities into the
overall project delivery methodology.
»» Incentivise quality by setting up vendor payments
according to measurable quality milestones.
»» Identify the shortfalls and provide feedback to
the vendor and business for review as the project
progresses.
Testing typically suffers from development slippages
and immovable delivery dates, resulting in reduced
test coverage and inappropriate sequencing of tests
(e.g. running user acceptance testing in parallel with
system testing).
Organisations can protect the quality of their IT
projects by ensuring independent test process
governance as well as providing adequate funding.
The testing process can be compromised when
vendors are allowed to “mark their own homework”,
or even“setting their own exams”. Problems also occur
when testing reports go via project management,
rather than directly to key stakeholders. Make testing
status reports a regular feature of steering committee
meetings and consider conducting independent test
process audits.
If you follow an agile methodology, ensure there are
dedicatedtestersembeddedwithineachdevelopment
team.
In terms of budget, estimate testing as a percentage of
the total project cost, rather a malleable phase of the
project which can be impacted by schedule changes.
The average spend on software testing in Australia
runs to around 25% of the total project budget (p.
15, Ross Report 2010 see below). Even with the right
budget, tight timelines can prevent adequate testing,
in which case you need to look at improving efficiency
through test automation and test case optimisation.
The chance of hitting the headlines after Go Live is
greatly reduced by protecting the testing phase from
being squeezed by upstream slippages, and insisting
on achieving meaningful test coverage of the system
according to the risk profile of the project.
TAILORING TESTING: ASSESS RISK AND TEST ACCORDINGLY
Page 3
K. J. Ross & Associates
Software Testing & ICT Risk Mitigation
WHITEPAPER | HOWCXO'SCANREDUCEITPROJECTRISK
BYIMPROVINGSOFTWARETESTING
PROTECT THE TESTING CYCLE
info@kjross.com.au
www.kjross.com.au
1300 854 063
The Ross Report shows that on average, 25%
of budgeted and actual costs of a successful
project are spent on testing.
This graph shows that some projects will spend up to
40% of project costs on testing. This will either be due
to poor quality or a need for mission critical systems.
The average project will spend 20 - 25% of project
costs on testing

4. VERTICALS
MANAGEDSERVICES
TRAINING
RESOURCE
PLACEMENT
CONSULTING
K. J. Ross & Associates
Software Testing & ICT Risk Mitigation
WHITEPAPER | HOWCXO'SCANREDUCEITPROJECTRISK
BYIMPROVINGSOFTWARETESTING
Page 4
info@kjross.com.au
www.kjross.com.au
1300 854 063
Ensure there is a separate channel of reporting for
Validation and Verification to avoid a single point of
pressureforGoLive.Validation,e.g.viaUserAcceptance
Testing (UAT), needs to focus on the solution being fit
for purpose, and that the business is operationally
ready to adopt the solution.
The Validation step is not a rubber stamp. Steering
committees need to be able to stop a Go Live if the risk
is too great for the business. Ensure that:
»» UAT phases are coordinated by experienced test
managers. Just throwing end users at the new
system will not deliver systematic coverage of
essential risks.
»» Non-functional requirements (performance,
usability, etc.) are covered adequately. On
average, only 20% of the total software testing
effort gets directed at non-functional testing, and
yet the success of a new IT solution is primarily
assessed on its ability to deliver improvements to
both efficiency and ease of use
»» Operational work-arounds are in place. Should
the worst happen, make sure you can still run
your business.
KJRosswillbeissuingfurtherWhitepapersonthetopic
of reducing IT risk from a testing perspective that will
cover these topics as well as “Testing in Production”
and“So you think you can test?”.
This will include commentary from major clients and
industry and will be available from our web site or by
request from marketing@kjross.com.au at a later date.
If you are interested in having input to or receiving the
Whitepaper please contact the team at marketing@
kjross.com.au
A free copy of “The Ross Report” a survey of testing in
Australia can be requested from marketing@kjross.
com.au
This report was compiled by Dr. Mark Pedesen,
Innovation Manager and Joe Griffiths, National
Engagement Manager at KJ Ross. This included input
from the 100+ testing and industry specialists at KJ
Ross including Dr. Kelvin J Ross himself.
Information from the 2010 Ross report was also used
in the article.
GOING LIVE WITH CONFIDENCE FURTHER INFORMATION
AUTHORS
“STEERING COMMITTEES NEED TO
BE ABLE TO STOP A GO-LIVE IF
THE RISK IS TOO GREAT FOR THE
BUSINESS”