Better Managed Outsourced Testing by Nathan Shearer Introduction This guide focuses on three key failure points: 1) misunderstanding of the cost of quality; 2) misalignment of contractual arrangements to the required testing/delivery outcomes; and 3) lack of effective metrics. Conclusion Organisations failing to fully understand their outsourced contracts and not including effective metrics and role accountabilities to monitor these often see cost overruns and project delays. There can also be reduced ROI against the outsourced agreement. Although the main driver for outsourced testing is often cost reduction, relying on a contractual arrangement to drive or manage this outcome often leads to a unit cost for testing KPI, instead of a metric based on total cost for quality. As KPIs drive behaviour, the organisation will be left with statistics to prove the KPI was met, but a less than optimal quality outcome. The outsourced testing provider’s approach to this situation would be to involve a professional, external organisation to perform a capability review. Recommended improvements or enhancements to the organisation’s governance of the commercial arrangement and the testing outcomes would be suggested, together with changes to the metrics used to report against the testing outcomes. Outsourced testing can be successful. It can speed up the test execution, improve quality and reduce costs. It is important to ensure the total cost of quality is understood, the quality outcomes are defined and aligned to the commercial agreement. Appropriate metrics should also be created, recorded and measured by both the provider and the client.

1. Better Outsourced Testing
Creating and managing agreements with offshore vendors

2. Introduction
This presentation will highlight three key failure points to outsourced testing:
1. misunderstanding of the cost of quality;
2. misalignment of contractual arrangements to the required testing/delivery
outcomes; and
3. lack of effective metrics.

3. Traditional outsourcing
agreements..

4. ..focus on the test execution
statistics and not quality as an
objective.

5. Is there a solution?
It is important to understand these issues can be mitigated and rectified by
examining current testing capability and improving the testing processes, not
by just policing the fine print in the outsourcing contract.

6. Defining the problem

7. Symptoms
Along with the typical less than
optimal outcomes (e.g. poor quality,
cost overruns and late delivery), other
potential symptoms of poorly
managed outsourced testing include:

8. ● poor or missing test planning documentation;

9. ● minimal test results;

10. ● deployment issues to test environments;

11. ● reactive focus on testing;

12. ● high severity issues upon deployment;

13. ● increases over time to the length of the test phase; and

14. ● little to none build or test automation.

15. Organisations should “Understand that testing is required to verify and validate that a business objective is being
realised... throughout the life cycle, not only at a user acceptance phase. Therefore, clear functional and non-functional
acceptance criteria should be defined for each phase of the life cycle.”
Gartner: ID:G00225707, ID:G00209445

16. Misunderstanding
Cost of Quality
Different outcomes from a unit cost
agreement and one where total cost
of quality is an objective
Unit Cost vs. Total Cost of Quality

17. Misaligned
contractual
arrangements
Often the commercial agreement is
the only governance provided to
measure, manage, report and maintain
the output of the outsourced testing
provider.
Different Outsourced Engagement Models

18. The true value of an outsource
provider is meeting the client’s
delivery need with the capability
(and ability) to deliver

19. Lack of Effective Metrics
If an outsource testing provider is bound by unit cost measures like total
defects found, the onus is on finding the maximum number of defects
(regardless of quality of defects, analysis, triage, allocation, etc.) in the allotted
period of time.

20. Common Results
Typically, the outcome will be an unquantified list of unqualified defects, test
results with a high failure or blockage rate, and no working solution.
The outsource provider may then be rewarded with more resources to come up
with another outcome (reduce the number of defects or pass tests), giving the
provider more revenue and leaving the client with cost and time overruns.

21. Organisations “have limited internal baseline metrics regarding quality and efficiency of their testing capabilities.”
Gartner: ID:G00225707, ID:G00209445

22. What’s the solution?

23. 1: A Capability Review
Assessing the current testing capability may be as simple as reviewing past
test results against estimated and planned outcomes. It could also involve a
professional external organisation performing a capability review and auditing
policies, processes, procedures and skills.

24. What is a Capability Review
This review can be used to recommend
improvements to specific areas. These could
include a new or updated test strategy (or
master test plan), to ensure the alignment to
desired outcomes, the introduction of metrics to
align to KPIs in the outsource contract, and
defining effective (light touch) governance
around roles and responsibilities to reach the
desired outcomes. Capability Review

25. How and What to Measure?
relative test effort;
test coverage of requirements;
percentage of tests automated;
defects remaining upon completion, and
defect analysis.
These metrics should be holistically called the cost of quality.

26. The cost of defects increases
throughout the testing lifecycle
and the longer a defect stays
within the process, the more
costly it is to fix.

27. Cost of Defects
To avoid cost overruns, consider using
defect classification metrics to more
easily identify trends in your test
process
Cost of Fixing Software Defects

28. To avoid cost overruns, consider
using defect classification
metrics to more easily identify
trends in your test process.

29. Defect reporting vs. defect analysis
Clients may already have effective defect reporting which can be invaluable for
providing a clear view of the current status of quality.
The move to defect analysis begins with reporting as an input, combined with
additional quantitative information to allow analysis of what is called phase
effectiveness.

30. There are two key metrics that
assist with understanding trends
in quality:

31. (PCE) Phase Containment Effectiveness
PCE is the ratio of faults captured in a phase and signifies how effective the
testing process is at preventing these defects in the first place.

32. (PCE) Phase Containment Effectiveness
In Figure 4, the PCE is calculated as follows:
Requirements captured 10 of 22 defects = 45%
Design contained seven of 22 defects = 32%
Build contained three of 22 defects = 14%
Test contained two of 22 defects = 9%
Phase Containment Effectiveness

33. (PSE) Phase Screening Effectiveness
PSE is the ratio of previously missed (not
contained) defects being captured in each
phase.
For the same example, the PSE is calculated as:
Requirements missed 12 of 22 defects
Design screened seven of these 12 defects = 58%
Build screened three of these 5 defects = 60%
Test screened two of the remaining two defects = 100%

34. Create or update a testing
framework, which will fully align
these metrics to the total cost of
quality.

35. Organisations failing to fully
understand their outsourced
contracts and not including
effective metrics and role
accountabilities to monitor these
often see cost overruns and
project delays.

36. KPIs drive
behaviour
Organisations will be left with statistics to prove the KPI was met, but a less than optimal quality outcome

37. 3 Simple Steps to Rectify
Outsourced testing can be successful.
It can speed up the test execution, improve quality and reduce costs.
It is important to ensure the total cost of quality is understood, the quality
outcomes are defined and aligned to the commercial agreement.

38. 1: Capability Review
Involve a professional, external organisation to perform a capability review

39. 2: Outsourced Agreement Review
Recommend improvements or enhancements to the organisation’s governance
of the commercial arrangement and the expected testing outcomes

40. 3: Improved Metrics
Together with changes to the metrics for both the outsourced provider and the
client organisation should be used to report against the testing outcomes

41. Nathan Shearer - Want to work with me? rapidexecutive.com/coaching
Like This?
Feel Free to Share it with
someone who could benefit
→
Connect with me on LinkedIn
Connect with me on Twitter