ISTQB Technical Test Analyst 2012 Training - Structure-Based Testing

ISTQB Advanced Level 2012 –
Technical Test Analyst

Objectives
 Build on ISTQB CTFL in the area of technical test analysis
 Prepare for the ISTQB CTAL – Technical Test Analyst exam
31/08/20122
ISTQB Advanced Level 2012 - Technical Test
Analyst

Prerequisites
 ISTQB CTFL or equivalent
 Practical experience in SW testing
31/08/20123
Analyst

Notes
 Ask any time.
 Turn your cell silent.
31/08/20124
Analyst

References
 ISTQB CTAL – TTA syllabus version 2012
31/08/20125
Analyst

Outline
 The Technical Test Analyst's Tasks in Risk-Based Testing
 Structure-Based Testing
 Analytical Techniques
 Quality Characteristics for Technical Testing
 Reviews
 Test Tools and Automation
31/08/20126
Analyst

Outline
 The Technical Test Analyst's Tasks in Risk-Based Testing
 Structure-Based Testing
 Analytical Techniques
 Quality Characteristics for Technical Testing
 Reviews
 Test Tools and Automation
31/08/20127
Analyst

Structure-Based Testing
 Introduction
 Condition Testing
 Decision Condition Testing
 Modified Condition/Decision Coverage (MC/DC) Testing
 Multiple Condition Testing
 Path Testing
 API Testing
 Selecting a Structure-Based Technique
31/08/2012
Analyst
8

 Introduction
 Path Testing
 API Testing
31/08/2012
Analyst
9

What is Structure-Based Testing?
 AKA white box or code-based
testing
 Uses the code, the data and the
architecture and/or system flow as
the basis for test design
 Techniques used in structure-
based testing:
 Derive test cases systematically
 Focus on a particular aspect of
the structure
31/08/2012
Analyst
10

Examples of Structure-Based Testing
Test Basis
 Component level
 Code structure
 Statements
 Decisions
 Conditions
 Integration level
 Call tree
 System level
 Menus structure
 Business process
 Web page structure
31/08/2012
Analyst
11

Why Structure-Based Testing?
 Structure-based testing provides a coverage criteria.
 A criteria can be measured and associated with an objective.
 Achieving full coverage != Complete test
 It only means technique being used no longer suggests any useful tests for
the structure under consideration.
 Complete test is project context dependent.
31/08/2012
Analyst
12

When Structure-Based Testing should
be Used?
 Specification-based testing will never test all the code.
 Structure-based testing should be used on top of specification based
testing to measure the coverage and hence add extra tests to increase the
coverage.
31/08/2012
Analyst
13

Atomic Conditions vs. Decisions
 A is an atomic condition.
 B is a atomic condition.
 C is a atomic condition.
 (A &&B) || C is a decision.
 Logical combination of atomic conditions makes a decision predicate.
31/08/2012
Analyst
14
if ((A && B) || C){
...
}

Examples of Structure-Based Testing
Techniques
1. Statement testing
2. Decision/Branch testing
3. Condition testing
4. Decision Condition testing
5. Modified Condition/Decision Coverage (MC/DC) testing
6. Multiple Condition testing
7. Path testing
8. API testing
31/08/2012
Analyst
15

 Introduction
 Path Testing
 API Testing
31/08/2012
Analyst
16

What is Condition Testing?
 Considers how decisions are made rather than decision predicates
 Answers the question “Could there be bugs hiding in the condition itself?”
 Makes sure that each atomic condition is tested both ways, TRUE and
FALSE
31/08/2012
Analyst
17

Condition Testing Coverage
 % of conditions possibilities exercised by a test suite
 Condition coverage % = (# of conditions possibilities executed/total # of
conditions possibilities ) * 100
 Example:
 Program has 100 conditions, thus 200 conditions possibilities
 Tests exercise 112 conditions possibilities
 Condition coverage = 56%
31/08/2012
Analyst
18

How is Condition Testing Done?
1. Identify atomic conditions.
2. Find the minimal test cases to achieve 100% condition coverage for each
atomic condition.
3. Combine test cases of the atomic conditions to find the minimal test
cases to test the decision. Make sure all test cases identified in step 2 are
executed.
31/08/2012
Analyst
19

Exercise: Identify Min # of Test Cases
to Achieve 100% Condition Coverage
1. X
2. D && F
3. (A || B) && (C == D)
4. (A > B) || ( X + Y == - 1) && (! (D) == TRUE)
31/08/2012
Analyst
20

Condition Testing Applicability
 Probably interesting only in the abstract
 Understanding it is necessary to achieving greater levels of coverage that
build upon it.
31/08/2012
Analyst
21

Condition Testing Limitations
 Achieving 100% condition coverage does not necessitate 100% decision
coverage.
 100% condition coverage necessitates 100% decision coverage iff
decisions are consisted of single atomic expressions.
31/08/2012
Analyst
22
if (A && B){
...
} else {
...
}
if (X){
...
} else {
...
}
TC1: A = T and B = F
TC2: A = F and B = T
TC1: X = T
TC2: X = F

 Introduction
 Path Testing
 API Testing
31/08/2012
Analyst
23

What is Decision Condition Testing?
 Testing must achieve condition coverage and decision coverage.
 100% decision condition coverage guarantees 100% statement
coverage, 100% decision coverage, and 100% condition coverage.
 A thoughtful choice of test data values for atomic conditions may result in
achieving this level of coverage without adding extra test cases beyond
that needed to achieve condition coverage.
31/08/2012
Analyst
24
if (A && B){
...
} else {
...
}
TC1: A = T and B = T
TC2: A = F and B = F

Decision Condition Testing Coverage
 100% decision condition coverage = 100% decision coverage + 100%
condition coverage
31/08/2012
Analyst
25

How is Decision Condition Testing
Done?
1. Identify atomic conditions.
2. Find the minimal test cases to achieve 100% condition coverage for each
atomic condition.
3. Combine test cases of the atomic conditions to find the minimal test
cases to test the decision keeping in mind 100% decision coverage is
achieved. Make sure all test cases identified in step 2 are executed.
4. Add/Modify any test cases needed if 100% decision coverage is not
achieved while keeping the 100% condition coverage
31/08/2012
Analyst
26

Exercise: Identify Min # of Test Cases to Achieve
100% Decision Condition Coverage
1. X
2. D && F
3. (A || B) && (C == D)
4. (A > B) || ( X + Y == - 1) && (! (D) == TRUE)
31/08/2012
Analyst
27

Decision Condition Testing
Applicability
 Code is important but not critical.
 What is important and what is critical?!!!
31/08/2012
Analyst
28

Decision Condition Testing Limitations
 May require more test cases compared to previous techniques
 May consume more time compared to previous techniques
31/08/2012
Analyst
29

 Introduction
 Path Testing
 API Testing
31/08/2012
Analyst
30

What is MC/DC Testing?
 Created by Boeing for use when specific languages were to be used in
safety-critical programming
 Provides a stronger level of control flow coverage
 Guarantees 100% decision condition coverage and adds to it
31/08/2012
Analyst
31

MC/DC Coverage
1. At least one test where the decision outcome would change if the atomic
condition X were TRUE
2. At least one test where the decision outcome would change if the atomic
condition X were FALSE
3. Each different atomic condition has tests that meet requirements 1 and
2.
31/08/2012
Analyst
32

MC/DC Coverage in Other Words
1. 100% condition coverage
2. 100% decision coverage
3. Each atomic condition must affect the outcome decision independently
while the other atomic conditions are held fixed.
31/08/2012
Analyst
33

MC/DC Testing by Example
 (A || B) && C
31/08/2012
Analyst
34
A B C
(A or B)
and C
Test 1 T F T T
Test 2 F T T T
Test 3 F F T F
Test 4 T F F F

C/C’s of MC/DC Testing
 Number of test cases for a decision is unique.
 There may be more than a test group fulfilling the MC/DC coverage.
 Causes a lot of testing but not an exhaustive testing
 Assuming N unique atomic conditions, MC/DC can usually be achieved in
N+1 unique test cases
31/08/2012
Analyst
35

Logical Expressions and their Truth
Tables
31/08/2012
Analyst
36

MC/DC Testing for an And Gate
 N inputs and gate requires n+1 test cases.
 All inputs are true.
 Each input is set exclusively to false.
 Exercise: Find test cases for (A && B && C)
31/08/2012
Analyst
37

MC/DC Testing for an Or Gate
 N inputs or gate requires n+1 test cases.
 All inputs are false.
 Each input is set exclusively to true.
 Exercise: Find test cases for (A || B || C)
31/08/2012
Analyst
38

MC/DC Testing for an Xor Gate
 No rule
 For a 2-input xor gate, any 3 out of the 4 possible combinations can
achieve the MC/DC coverage.
 Only 1 combination can detect an OR mistyped as XOR.
31/08/2012
Analyst
39

MC/DC Testing for a Not Gate
 Only 2 test cases 
31/08/2012
Analyst
40

MC/DC Testing for Comparators
 <, >, <=, >=, ==, !=
 Needs 3 test cases:
 Output is false and inputs are close to the boundary.
 Output is true and inputs are close the boundary.
 The inputs are at the boundary.
 MC/DC + discovering typing mistakes
 Exercise: Find test cases for x >= 25000
31/08/2012
Analyst
41

MC/DC Testing for an If-Else
Statement
 Needs the following test cases:
 Inputs to make decision true.
 Inputs to force decision false.
 Inputs to exercise any of the previous items in the decision.
 Exercise: if(Z) statement 1; else statement 2;
31/08/2012
Analyst
42

MC/DC Testing for a While Loop
 Inputs to force decision false.
 What about the for loop?
 Exercise: while(Z) statement 1;
31/08/2012
Analyst
43

MC/DC Testing for a Do-While Loop
 Inputs to force decision false
 Exercise: do statement 1; while(Z);
31/08/2012
Analyst
44

How is MC/DC Testing Done?
1. Represent source code graphically in terms of logic gates.
2. Identify test inputs and output relation.
1. Draw truth table
3. Identify observation point and control input(s), then eliminate masked
test cases.
4. Determine MC/DC for each logic gate starting from the input moving
towards the output.
 Exercise: Determine MC/DC for (B && C) || D.
31/08/2012
Analyst
45

1- Representing the Code Graphically
31/08/2012
Analyst
46

2- I/O Relation
ID B C D A
1 F F F F
2 F F T T
3 F T F F
4 F T T T
5 T F F F
6 T F T T
7 T T F T
8 T T T T
31/08/2012
Analyst
47

3- Observation Point, Control Input,
Masked Test Case
 An observation point is the point at which we observe the output of
concern.
 A control input is when its value is known, the value at the observation
point can be calculated regardless of the other inputs.
 Masked test cases are test cases where the we care only for the control
input and others inputs are do not care. They are in the essence
equivalent.
31/08/2012
Analyst
48

3- Observation Point, Control Input,
Masked Test Case cont’d
 Masking tests rules:
 W and true = W
 W or false = W
 W xor false = W
 W xor true = !W
31/08/2012
Analyst
49

3- Observation Point, Control
Input, Masked Test Case cont’d
ID B C D A
1 F F F F
2 F F T T
3 F T F F
4 F T T T
5 T F F F
6 T F T T
7 T T F T
8 T T T T
31/08/2012
Analyst
50
Observation Point
Control Input when =
T
Masked Test Cases

4- Determining MC/DC for the Gates
 By combining the results in the table, we can use any of the following tests
:
 2, 3, 5, 7
 3, 4, 5, 7
 3, 5, 6, 7
31/08/2012
Analyst
51
Valid Inputs Missing Test Cases
And gate TT: row 7
TF: row 5
FT: row 3
Or gate FF: rows 1, 3, or 5
TF: row 7
FT: rows 2, 4, or 6

Exercise: Identify Min # of Test Cases
to Achieve 100% MC/DC Coverage
1. Z = (A || B) && (C || D)
2. Z = (A && !B) || (C || D)
3. A = (B || C) && D; E = (X && Y) || C; Z = A&& E
4. O = (O || I) && !R
5. Z = X & Y
31/08/2012
Analyst
52

MC/DC Testing Applicability
 Aerospace SW industry
 Safety-critical systems
 In general, it is used when dealing with safety critical SW where any failure
may cause a catastrophe.
31/08/2012
Analyst
53

MC/DC Testing Limitations
 Short-circuiting may affect the ability to attain MC/DC coverage since
some required tests may not be achievable.
 Achieving MC/DC coverage may be complicated when the atomic
conditions are coupled in the expression.
 Depending on the decision statement in the code, it may not be possible to
vary the value of the coupled term such that it alone causes the decision
outcome to change.
31/08/2012
Analyst
54

Short Circuiting
 In some programming languages or some compiler options, the right hand
operand is evaluated iff it is needed to determine the expression value.
 Q: If func() is never evaluated, the question then becomes, can we still
achieve MC/DC coverage?
 A: May be as some tests may be not achievable at all.
31/08/2012
Analyst
55
if (A || func()){
...
}

Short Circuiting cont’d
 Many organizations evaluated it in different ways.
 If your project is subject to regulatory statutes, consult how that
regulatory body wants you to deal with the short circuiting.
 For example, NASA claims that short circuit logic can be dealt with by
applying the masking MC/DC approach explained earlier, where a case-by-
case basis analysis of the decision is done.
31/08/2012
Analyst
56

Coupled Atomic Conditions
 Multiple occurrences of a condition in an expression is called coupling.
 A and !A can not be varied independently as required by MC/DC.
 If your project is subject to regulatory statutes, consult how that
regulatory body wants you to deal with the short circuiting.
 There are 2 approaches for facing such situation:
1. Unique Cause MC/DC approach, where the term condition in the definition of
MC/DC is deemed to mean uncoupled condition.
2. Masking MC/DC approach explained earlier, where a case-by-case basis
analysis of the decision is done.
31/08/2012
Analyst
57
if (A || (! A && B)){
...
}

 Introduction
 Path Testing
 API Testing
31/08/2012
Analyst
58

What is Multiple Condition Testing?
 Test every possible combination of atomic conditions in a decision!
 Truly exhaustive testing
31/08/2012
Analyst
59

How is Multiple Condition Testing
Done?
 Create a truth table with every combination.
 Does not take much analysis
 # of test cases = 2# of atomic conditions
31/08/2012
Analyst
60

Multiple Condition Testing by
Example
 (A || B) && C
31/08/2012
Analyst
61
A B C
(A or B)
and C
Test 1 F F F F
Test 2 F F T F
Test 3 F T F F
Test 4 F T T T
Test 5 T F F F
Test 6 T F T T
Test 7 T T F F
Test 8 T T T T

Is it Really a Truth Table?
 Short circuiting and coupling are still applicable.
 If the language uses short-circuiting, the number of actual test cases will
often be reduced, depending on the order and grouping of logical
operations that are performed on the atomic conditions.
31/08/2012
Analyst
62

100% Multiple Condition Coverage
 Assume we have short circuit logic compiler
1. A && B && (C || (D && E))
2. ((A || B) && (C || D)) && E
31/08/2012
Analyst
63

Multiple Condition Testing
Applicability
 Testing embedded software which was expected to run reliably without
crashing for long periods of time
 Telephone switches that were expected to last 30 years
 This type of testing is likely to be replaced by MC/DC testing in most
critical applications.
31/08/2012
Analyst
64

Multiple Condition Testing Limitations
 Because the number of test cases can be derived directly from a truth
table containing all of the atomic conditions, this level of coverage can
easily be determined.
 However, the sheer number of test cases required makes MC/DC coverage
more applicable to most situations.
31/08/2012
Analyst
65

 Introduction
 Path Testing
 API Testing
31/08/2012
Analyst
66

What is Path Testing?
 Identifying paths through the code and then creating tests to cover them
 In some cases we will get some of the same tests we got through control-
flow testing.
 On the other hand, we might come up with some other interesting tests.
31/08/2012
Analyst
67

Brute Force Path Testing
 Test every independent path through the system.
 Loops are the killers.
 Needs infinite time and infinite resources!
 However, it is realistic to perform some path testing.
 Exercise: How many paths do exist in the shown figure?
31/08/2012
Analyst
68

How can Loops be Tested in Path
Testing?
 A significant and meaningful reduction is needed.
1. Execute the loop zero times.
2. Execute the loop one time.
3. Execute the loop n times.
 n is very small and typical loop value.
4. Execute the loop m times
 m is the maximum number of loop times.
5. Test m-1, and m+1 times
31/08/2012
Analyst
69

Path Testing Coverage
 % of paths exercised by a test suite
 Path coverage % = (# of paths executed/total # of paths ) * 100
 Example:
 Program has 74 paths
 Tests exercise 30 paths
 Path coverage = 40%
 Notes:
 100% path coverage (disregarding loops) = 100 % statement coverage + 100%
decision coverage
 Path testing provides more thorough testing than decision coverage, with a
relatively small increase in the number of tests .
31/08/2012
Analyst
70

How is Path Testing Done?
1. Beizer’s Method of Path Testing
2. Linear Code Sequence And Jump Testing – LCSAJ Testing
3. Basis Path/Cyclomatic Complexity Testing
31/08/2012
Analyst
71

Beizer’s Method
1. Pick the first path as the simplest, functionally sensible path from entry
to exit.
2. Pick each additional path as a small variation on the previous path.
 Try to change only one branch in the path.
 Favor short paths over long paths when possible.
 Favor paths that make functional sense over ones that do not.
3. Pick paths that don't make functional sense only when required for
coverage.
 Such paths might be extraneous and should be questioned.
4. Use intuition when choosing paths.
31/08/2012
Analyst
72

Notes about Beizer’s Method
 Some path segments are likely to be executed more than once.
 The key point is to test every possible branch through the code at least
once and possibly many times.
31/08/2012
Analyst
73

100% Path Coverage Using Beizer’s Method
31/08/2012
Analyst
74
A = getchar();
B = getchar();
C = getchar();
if(A > 10) {
printf(“Case 1n”);
}
if((B > 1) || (C > 2)){
printf(“Case 1n”);
}

What is a LCSAJ Testing?
 AKA as decision-to-decision testing
 Concerned with testing a linear sequence of executable statements of
code followed by a jump to another executable statement of code. This
might happen after a decision or a goto statement in the code.
 Devised by Professor Michael Hennell of the University of Liverpool in
1976
 Relatively high overhead methodology for testing
31/08/2012
Analyst
75

What is a LCSAJ?
 Small block of code that:
1. Has a start which can be a start of a module or a line of code jumped to from
another LCSAJ
2. Has an end which can be a module end or a line of code where a jump occurs
3. Has a target line of the jump
31/08/2012
Analyst
76

LCSAJ Testing Coverage
 % of LCSAJs exercised by a test suite
 LCSAJ coverage % = (# of LCSAJs executed/total # of LCSAJs ) * 100
 Example:
 Program has 42 LCSAJs
 Tests exercise 30 LCSAJs
 LCSAJ coverage = 71 %
31/08/2012
Analyst
77

LCSAJ Example Code and Table
31/08/2012
Analyst
78
1:#include <stdlib.h>
2:#include <string.h>
3:#include <math.h>
4:#define MAXCOLUMNS 26
5:#define MAXROWS 20
6:#define MAXCOUNT 90
7:#define ITERATIONS 750
8:int main(void){
9: int count = 0, totals[MAXCOLUMNS], val = 0;
10: memset(totals, 0, MAXCOLUMNS * sizeof(int));
11: while(count < ITERATIONS){
12: val = abs(rand()) % MAXCOLUMNS;
13: totals[val] += 1;
14: if(totals[val] > MAXCOUNT){
15: totals[val] = MAXCOUNT;
16: }
17: count++;
18: }
19: return(0);
20:}
LCSAJ # Start End
Jump
To
1 8 11 19
2 8 14 17
3 8 18 11
4 11 11 19
5 11 14 17
6 11 18 11
7 17 18 11
8 19 19 -1

Exercise: Identify Min # of Test Cases to
Achieve 100% LCSAJ Coverage
 For the previous example, identify the min # of test cases to achieve 100%
LCSAJ coverage.
31/08/2012
Analyst
79

Notes About LCSAJ Testing
 Not all LCSAJs are really executable.
 They can be executed by special intervention using debuggers for example.
 A lot of time may be spent to achieve 100% LCSAJ coverage.
 LCSAJ testing is really brittle to code and requires expensive maintenance.
 LCSAJs testing in most cases leads to a partial path coverage.
31/08/2012
Analyst
80

What is Basis Path/Cyclomatic
Complexity Testing?
 Suggestion is to test the structure of the code to a reasonable amount
 Devised by Thomas McCabe in 1976
 Any SW module has a small # of unique, independent paths (excluding
iterations) called basis paths.
 Code structure can be tested by testing the basis paths because all the
infinite different paths are just using/reusing the basis paths.
 AKA minimal path testing
31/08/2012
Analyst
81

Basis Path and Basis Path Set
 Basis path is a unique independent path through the module with no
iterations or loops allowed.
 A basis path traverses at least one edge that no other path does.
 Basis path set is the smallest # of basis paths that cover the structure of
the code.
 100% basis path coverage = 100% statement coverage + 100% decision
coverage
31/08/2012
Analyst
82

Cyclomatic Complexity
 Cyclomatic refers to an imaginary loop from the end of a module of code
back to the beginning of it.
 Cyclomatic complexity # is the # of loops needed to cycle through the loop
until the structure of the code has been completely covered.
 Depends on the decisions in the module not on the module size
 Cyclomatic complexity # = # of test cases we need to test the set of basis
paths = # of basis paths
 The higher the complexity, the more likely there will be a higher bug count
31/08/2012
Analyst
83

Example: Calculating Cyclomatic
Complexity
31/08/2012
Analyst
84
EuclidGCD (int a, int b){
int t;
if (a <= 0)
return -1;
if (b <= 0)
return -1;
if (b > a) {
t = a;
a = b;
b = t;
}
t = a % b;
while (t != 0) {
a = b;
b = t;
t = a % b;
}
return b;
}
End
Start a <= 0
b <= 0
b > a
t != 0
A B
C
D
E F

Example: Calculating Cyclomatic
Complexity cont’d
 C = # of Regions + 1 = 4 + 1 = 5
 C = # of Edges - # of Nodes + 2 x # of
Connected Components = 9 – 6 + 2 =
5
 C = # of branching + # of looping + 1
= 3 + 1 + 1
 Used for large code constructs and
does not need CFGs
 if, for, while, or do-while constructs
count as 1.
 case block counts as 1.
 else and default block count as 0.
31/08/2012
Analyst
85
End
Start a <= 0
b <= 0
b > a
t != 0
A
C
D
E F
B

Example: Deriving Basis Paths and
Test Cases
 Basis paths are:
 ABF
 ABCF
 ABCDEF
 ABCDEF
 ABCDEEF
 Basis tests are:
 -5, 2  -1
 2, -5  -1
 16, 8  8
 4, 8  4
 20, 8  4
31/08/2012
Analyst
86
End
Start a <= 0
b <= 0
b > a
t != 0
A
C
D
E F
B

Exercise: Identify Min # of Test Cases to
Achieve 100% Basis Path Testing Coverage
31/08/2012
Analyst
87
int main(int MaxCols, int Iterations, int MaxCount){
int count = 0, totals[MaxCols], val = 0;
memset(totals, 0, MaxCols * sizeof(int));
if (MaxCount > Iterations){
while(count < Iterations){
val = abs(rand()) % MaxCols;
totals[val] += 1;
if(totals[val] > MaxCount){
totals[val] = MAXCOUNT;
}
count++;
}
}
return(0);
}

Path Testing Applicability
 Partial path testing is often performed in safety critical SW.
 It is a good addition to the other methods covered earlier because it looks
at paths through the SW rather than just at the way decisions are made.
31/08/2012
Analyst
88

Path Testing Limitations
 While it is possible to use a control flow graph to determine the
paths, realistically a tool is required to calculate them for complex
modules.
31/08/2012
Analyst
89

 Introduction
 Path Testing
 API Testing
31/08/2012
Analyst
90

What is an API?
 An Application Programming
Interface (API) is code which
enables communication between
different processes, programs
and/or systems.
 APIs are often utilized in a
client/server relationship where
one process supplies some kind of
functionality to other processes.
31/08/2012
Analyst
91

What is API Testing?
 In certain respects API testing is quite similar to testing a GUI.
 Though GUI is rarely involved.
 The focus is on the evaluation of input values and returned data.
 Setting (complex) initial environment as GUI is mostly not involved is very
common.
31/08/2012
Analyst
92

Aspects of API Testing
Aspect Description
-ve testing
Trying to use API interfaces in ways for which they
were not intended
Testing APIs robust error handling to avoid incorrect
operation
Combinatorial testing
Combining the APIs as well as their parameters in
different ways because APIs are often used in
conjunction with other APIs
Availability/Reliability testing
APIs are loosely coupled. Transactions can be lost or
timed-out.
Thorough testing of the recovery and retry
mechanisms to ensure that all services have a very
high availability
Requires reliability testing by the API author as well as
infrastructure support
31/08/2012
Analyst
93

API Testing Coverage
 API testing is a type of testing rather than a measurement/coverage
technique.
 No specific coverage level
 Coverage example could be input/output coverage.
1. Use equivalence partitioning to determine valid/invalid input/output
partitions.
2. Use equivalence partitioning testing and boundary value testing to design test
cases that cover all valid/invalid input/output partitions.
31/08/2012
Analyst
94

API Testing Applicability
 In distributed systems or remote processing
 OS calls
 SOA
 RPC
 Web services
 Testing systems of systems
31/08/2012
Analyst
95

API Testing Limitations
 Usage of specialized tools as there is no GUI involvement
 Tools are needed for:
 Initial environment setup
 Data marshaling
 API invocation
 Result determination
31/08/2012
Analyst
96

 Introduction
 Path Testing
 API Testing
31/08/2012
Analyst
97

Testing Principle #6: Testing is Context
Dependent
31/08/2012
Analyst
98
 The context of the software system
under test should guide the technical
test analyst.
 Code criticality α Level of coverage α
Time & resources
 Sometimes the level of coverage
required may be derived from applicable
standards that apply to the software
system.
 DO-178 B in USA (ED-12B in Europe)
 IEC-61508
Code
Criticality
Level of
Coverage
Time &
Resources

Comparisons can Help the Selection
 There are 3 ways of comparisons:
 Pros and cons general summary
 Subsumes
 Check-list based
 Note that API testing is not included as it is not relate to code.
31/08/2012
Analyst
99

Techniques Pros & Cons
Technique Pros Cons
Statement
Easy to understand & apply
Can isolate dead code
Low maintenance required
Limited value for decisions & loops
Many paths untested
Ad-hoc testing can achieve ~70% statement
coverage
Decision
Easy to understand & apply
Good cost-benefit ratio
Does not account for condition combinations
Condition No real advantage over decision testing May omit decisions if not carefully done
Decision/Condition
Easy to understand
Good cost-benefit ratio
Needs careful choice of test cases
MC/DC
Less test cases compared to multiple
condition
More difficult to understand
Multiple Condition High thoroughness
High # of test cases
Multiple conditions could have been coded as
consecutive single conditions
LCSAJ Gives different angle of path coverage
Difficult to apply
Highly brittle to code
Path
Very throughout
Good for procedure testing
Practically impossible to achieve high level of
coverage
31/08/2012
Analyst
100

Subsumes - Which Technique
Implicitly Includes Others?
31/08/2012
Analyst
101
Multiple
Condition
MC/DC
Decision
Condition
Decision
Condition
Statement
LCSAJ
Path

Check-List Comparison
Technique Thorough Understandable Tool Support Maintainability
Statement 1 5 5 5
Decision 2 5 5 5
Condition 2 5 4 5
Decision Condition 2 5 4.5 5
MC/DC 4 3 3 5
Multiple Condition 3 4 4 5
LCSAJ 3 1 1 2
Path 5 2 4 2
31/08/2012
Analyst
102

DO-178 B
 Used in an airborne environment
31/08/2012
Analyst
103
Failure Level Description Testing Technique
A: Catastrophic
Failure may cause lack of critical function
needed to safely fly or land the plane
MC/DC
B: Hazardous
Failure may have a large –ve impact on safety
or performance
Decision Testing
and MC/DC
optional
C: Major
Failure is significant, but less serious than A or
B
Statement testing
@ minimum
D: Minor
Failure is noticeable, but with less impact than
C
E: No effect failure has no impact on safety

IEC-61508
 For the functional safety of programmable, electronic,
safety-related systems (automotive, rail, manufacturing
process, nuclear power plants, and machinery)
31/08/2012
Analyst
104
SIL Testing Technique
1 (Least critical) Statement and branch coverage recommended
2
Statement coverage highly recommended, branch
coverage recommended
3 Statement and branch coverage highly recommended
4 (most critical) MC/DC highly recommended

Testing Multi-Systems
 In modern systems, it is rare that all processing will be done on a single
system.
 API testing should be instituted anytime some of the processing is going to
be done remotely.
 The criticality of the system should determine how much effort should be
invested in API testing.
31/08/2012
Analyst
105

One Final Word
 Dynamic techniques should not be used in isolation.
 They should be mixed and combined.
 Some techniques can be useful in identifying the test conditions, others
more suited for minimizing test conditions, while others are more suited
for deriving test cases from test conditions.
31/08/2012
Analyst
106

Combining Techniques Example
1. Using decision tables to identify test conditions from requirements.
 Test conditions will be every column in this case.
2. Transform the decision table into a binary decision table.
 This may result in the # of test conditions.
3. Identify a binary equation to relate actions to decisions.
4. Apply MC/DC to reduce the # of test conditions.
5. Identify valid/invalid ranges for every condition and action using equivalence
portioning.
6. Apply equivalence partitioning and boundary value analysis to derive test
cases to cover the test conditions.
31/08/2012
Analyst
107

ISTQB Technical Test Analyst 2012 Training - Structure-Based Testing

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie ISTQB Technical Test Analyst 2012 Training - Structure-Based Testing

Ähnlich wie ISTQB Technical Test Analyst 2012 Training - Structure-Based Testing (20)

Mehr von Amr Ali (ISTQB CTAL Full, CSM, ITIL Foundation)

Mehr von Amr Ali (ISTQB CTAL Full, CSM, ITIL Foundation) (12)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

ISTQB Technical Test Analyst 2012 Training - Structure-Based Testing

Hinweis der Redaktion