The document discusses metrics for supporting release decisions based on model-based testing. It describes using an operational profile to generate test cases, calculating model coverage metrics, using a reliability demonstration chart to assess risk, and measuring relative proximity to compare expected and actual failure rates. A case study applies these methods to a word processing app and missile defense system. Key observations are that model coverage ensures sufficient testing, reliability demonstration charts assume flat profiles which may be optimistic, and relative proximity indicates when failure intensities match expectations.

1. The Tester’s Dashboard:
Release Decision Support
Robert V. Binder
System Verification Associates, LLC
rbinder@ieee.org
Peter B. Lakey
Cognitive Concepts, Inc.
peterlakey@sbcglobal.net

2. Overview
• Complementary metrics for release decision-
support
– Model-based testing
• Operational profile
• Model coverage metrics
– Reliability Demonstration Chart
– Relative Proximity
• Case Study
• Observations

3. Release Decision Support

5. Model-based Reliability Estimation
• Test suites must be
– Proportional to operational profile
– Sequentially feasible
– Input feasible
• Approach
– Markov model
– Monte Carlo simulation
– Post run analytics

6. Model Coverage Metrics
• % States
Usage Profile
Reached
S T
• % State-
Transitions
Observe
System
Reached
Failure
Trigger
Latent
Defect Software System
Process Space
Observe fault
System activated
Failure
Data Space

7. Reliability Demonstration Chart
• Sequential
Sampling
• Risk-
Adjusted
• Musa
equations
http://sourceforge.net/projects/rdc/

8. Relative Proximity
• Kullback-Lieber Distance
– Information theoretic
– Characterizes difference in variation of message population E
(expected) and sample A (actual) as “relative entropy”
KLD = ∑ 𝐴 𝑖 (𝑙𝑜𝑔2 (𝐴 𝑖 / E 𝑖 ))
• Relative Proximity
– KLD math doesn’t work unless failures modeled (sum of the
actuals must be 1.0)
– Assume the target failure rate is aggregate
– Allocate failure rate in proportion to each operation

9. Profile Explicit Failure Modes
• Assume maximum acceptable failure rate intensity of 1 in 10,000
Operation Mode Standard Explicit Failure Expected Number,
Profile Profile 10000 Tests
A Pass 0.7 0.6993 6993
B Pass 0.2 0.1998 1998
C Pass 0.1 0.0999 999

10. Profile Explicit Failure Modes
Mode Expected Actual KL Distance Actual KL Distance
A Pass 6993 7000 10.104 6990 -4.327
Fail 7 0 0.000 10 5.146
B Pass 1998 1990 -11.518 2000 2.887
Fail 2 10 23.219 0 0.000
C Pass 999 980 -27.149 994 -7.195
Fail 1 20 86.439 6 15.510
10000 10000 81.094 10000 12.020
• Relative Proximity indicates the difference
between actual and observed failure rates
• Many possible operation failure rates with better
or worse fidelity
• RDC based on aggregate FIO, not sensitive to
operation variance

11. Case Studies
• Stochastic Models
• Assumed Failure Rates
• Word Processing Application
• Ground-Based Midcourse Missile Defense

12. GBMD Test Run, 0-100

13. GBMD Test Run, 1K, 5K

14. GMBD Test Run, 10K

15. GBMD Relative Proximity Trend
1600.00
1484.00
1400.00
1200.00
1000.00
800.00
600.00
418.20
400.00
200.00 67.90
12.00 6.30
0.00
10 100 1000 10000

16. Observations
• Model coverage indicates minimal sufficiency
– Wouldn’t release without all state-xtn pairs covered
– Stochastic can take a long time to do this
– Cover with N+ first
• RDC assumes “flat” profile
– With sequential constraints, may be optimistic
– Strength is explicit risk-adjustment
• Relative Proximity will indicate when operation-
specific Failure Intensity is as expected (or not)

17. Q&A