Webinar recording from February 6 with Shaun Bradshaw and Philip Lew covering using the S-curve to manage a test effort. Some insights into metrics in general, how to make interpretations and how not to.

1. Shaun Bradshaw & Philip
Lew
Managing with Metrics The
Saga of a Test Effort!

2. XBOSoft info
• Founded in 2006
• Dedicated to software quality
• Software QA consulting
• Software testing services
• Offices in Santa Clara,CA
Beijing, Oslo and Amsterdam

3. About the Speakers
Shaun Bradshaw
VP Consulting Services
Zenergy Technologies
Philip Lew
CEO and Founder
XBOSoft

4. The Metric

5. What is Measurement?
• Measurement is the process by which numbers
or symbols are assigned to attributes of entities
in the real world in such a way as to characterize
them according to clearly defined rules. [1]
• Measurement is the empirical, objective
assignment of numbers, according to a rule
derived from a model or theory, to attributes of
objects or events with the intent of describing
them. [2]

6. Rayleigh Cumulative Distribution
• In probability theory and statistics, the Rayleigh
distribution is a continuous probability distribution. As an
example of how it arises, the wind speed will have a
Rayleigh distribution if the components of the two-
dimensional wind velocity vector are uncorrelated and
normally distributed with equal variance. The distribution
is named after Lord Rayleigh. [1]
What is the Rayleigh Cumulative Distribution?

7. Rayleigh Cumulative Distribution
• Test execution starts slowly as the team works
through configuration issues and major blocking
defects.
• Once the initial issues are resolved a larger
variety of tests can be executed, increasing
execution velocity.
• As testing nears release, there are fewer tests to
be executed and only a few defects remain
outstanding, leveling out the speed of execution.
Why does it have an S shape?[2]

8. Data Requirements
• The curve can be tracked for two main purposes
with similar data requirements:
– Test Execution Progress
• Total tests to be executed
• Total number of tests in a Passed state
• Total number of days in the test effort
– Application Stability Tracking
• Total tests to be executed
• Historical fail rates (to derive the anticipated number of
failures)
• Total number of failures
• Total number of days in the test effort
What data is required to generate the curve?

9. The Graph

10. Actual vs Theoretical
By plotting the actual cumulative number of
passed tests and comparing it to the
theoretical curve we are able to identify
potential issues and make adjustments to
the effort
ensuring
testing is as
successful
as possible.

11. The Saga

12. • Team
– No professional testers
– 30-40 SMEs in the areas of:
• Supply Chain Planning, Supply Chain Management,
Manufacturing, Operations, Sales (divided by product
category), Accounting, etc.
• Test effort
– Complete manual test execution of 370 E2E, 390
functional, and 600 process flow test cases in 15
days
– Daily stand-up held to review metrics and adjust test
execution strategy
Interpreting the Curve

13. Day 1
Interpreting the Curve
Good start, as expected. Since this is a regression test effort
most environmental issues should be taken care of, as well as
the few, if any major blocking issues

14. Pass rate at this point indicates a possible early completion.
Process flow execution and passes have been high since the
accounting group shifted resources to test execution. The best
news is that the
team is about 2
days ahead of
schedule.
Manager verifies
with team that
they are execut-
ing high priority
tests first.
Interpreting the Curve
Day 5

15. Execution has become anemic. Accounting is not completing
process flow validations because of issues with taxes and a
known problem with VAT. Next week is focused on defect
correction and
re-testing. Devel-
opment indicates
the custom manu-
facturing issue
will be corrected
by the end of the
week and VAT
should be correct-
ed as well.
Interpreting the Curve
Day 10

16. Although the team did not achieve 100% completion, fewer
than 40 tests were outstanding. Furthermore, specific
acceptance criteria were established prior to test execution and
all criteria were
met or exceeded.
Interpreting the Curve
Day 15

17. Questions

18. Follow: @XBOSoft
Questions: services@xbosoft.com
408-350-0508
Follow: @Zenergytechnologies
Questions: 877.375.7041