2. public class Message {
private long id = -1;
private String author;
private String message;
public Message(String author, String message) {
this.author = author;
this.message = message;
}
// Gets and Sets ...
/**
* Appends the given text at the end of this' message.
* A space is added before the given text.
*
* @param text The test to append.
*/
public void compose(String text) {
this.message = message + " " + text;
}
}
3. @Test
public void testComposeString() {
Message msg = new Message("Me", "despicable");
msg.compose("me");
}
4. Code coverage
Line coverage: percentage of lines covered by tests
https://github.com/cobertura/cobertura
Achieving line coverage
=
executing each line at least once
5. @Test
public void testComposeString() {
Message msg = new Message("Me", "despicable");
msg.compose("me");
}
Is the compose method covered by the test case?
6. @Test
public void testComposeString() {
Message msg = new Message("Me", "despicable");
msg.compose("me");
}
Is the compose method covered by the test case?
Yes
7. @Test
public void testComposeString() {
Message msg = new Message("Me", "despicable");
msg.compose("me");
}
Is the compose method covered by the test case?
Is it enough?
Yes
8. @Test
public void testComposeString() {
Message msg = new Message("Me", "despicable");
msg.compose("me");
}
Is the compose method covered by the test case?
Is it enough?
Yes
No
9. @Test
public void testComposeString() {
Message msg = new Message("Me", "despicable");
msg.compose("me");
}
Is the compose method covered by the test case?
Is it enough?
Yes
No
There are no assertions on the result/effect of the method
12. Input/output domain coverage
public void compose(String text);
• Equivalence partitioning
Normal case: "Despicable you"
(based on pre/post conditions)
13. Input/output domain coverage
public void compose(String text);
• Equivalence partitioning
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
(based on pre/post conditions)
14. Input/output domain coverage
public void compose(String text);
public void compose(String t1, String t2);
• Equivalence partitioning
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
(based on pre/post conditions)
15. Input/output domain coverage
public void compose(String text);
public void compose(String t1, String t2);
• Equivalence partitioning
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
• t1
• t2
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
(based on pre/post conditions)
16. Input/output domain coverage
public void compose(String text);
public void compose(String t1, String t2);
• Equivalence partitioning
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
• t1
• t2
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
(based on pre/post conditions)
public void compose(String t1, String t2, String t3);
17. Input/output domain coverage
public void compose(String text);
public void compose(String t1, String t2);
• Equivalence partitioning
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
• t1
• t2
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
☛ Combinatorial explosion with the number of parameters
(based on pre/post conditions)
public void compose(String t1, String t2, String t3);
18. Combinatorial Interaction Testing (CIT)
Hyp.: Most of the faults are due to undesired
interactions of a limited number (N) of factors
(here, equivalent classes of parameter values).
D. Kuhn et al, Software fault interactions and implications for software testing. TSE 2004
• In practice: N ≤ 4
• Usually: N = 2 (pairwise)
19. Combinatorial Interaction Testing (CIT)
Hyp.: Most of the faults are due to undesired
interactions of a limited number (N) of factors
(here, equivalent classes of parameter values).
D. Kuhn et al, Software fault interactions and implications for software testing. TSE 2004
• In practice: N ≤ 4
• Usually: N = 2 (pairwise)
Idea: Generate test cases such that all
combinations of N values are covered at least
once.
20. Combinatorial Interaction Testing (CIT)
t1 t2 t3
"Despicable you" "Despicable you" "Despicable you"
"" "" ""
null null null
"azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù"
Hyp.: most faults are caused by interactions of at most two factors
Pairwise testing: cover all pairs of values
21. Combinatorial Interaction Testing (CIT)
t1 t2 t3
"Despicable you" "Despicable you" "Despicable you"
"" "" ""
null null null
"azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù"
Hyp.: most faults are caused by interactions of at most two factors
Pairwise testing: cover all pairs of values
"Despicable you" "Despicable you" "Despicable you"
"Despicable you" "" ""
"Despicable you" null null
"Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù"
"" "Despicable you" ""
"" "" "Despicable you"
"" null "azrtyuiop^$ù"
"" "azrtyuiop^$ù" null
null "Despicable you" null
null "" "azrtyuiop^$ù"
null null "Despicable you"
null "azrtyuiop^$ù" ""
"azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù"
"azrtyuiop^$ù" "" null
"azrtyuiop^$ù" null ""
"azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you"
16 test cases
instead of 64
22. Combinatorial Interaction Testing (CIT)
t1 t2 t3
"Despicable you" "Despicable you" "Despicable you"
"" "" ""
null null null
"azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù"
Hyp.: most faults are caused by interactions of at most two factors
Pairwise testing: cover all pairs of values
"Despicable you" "Despicable you" "Despicable you"
"Despicable you" "" ""
"Despicable you" null null
"Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù"
"" "Despicable you" ""
"" "" "Despicable you"
"" null "azrtyuiop^$ù"
"" "azrtyuiop^$ù" null
null "Despicable you" null
null "" "azrtyuiop^$ù"
null null "Despicable you"
null "azrtyuiop^$ù" ""
"azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù"
"azrtyuiop^$ù" "" null
"azrtyuiop^$ù" null ""
"azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you"
16 test cases
instead of 64
23. Combinatorial Interaction Testing (CIT)
t1 t2 t3
"Despicable you" "Despicable you" "Despicable you"
"" "" ""
null null null
"azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù"
Hyp.: most faults are caused by interactions of at most two factors
Pairwise testing: cover all pairs of values
"Despicable you" "Despicable you" "Despicable you"
"Despicable you" "" ""
"Despicable you" null null
"Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù"
"" "Despicable you" ""
"" "" "Despicable you"
"" null "azrtyuiop^$ù"
"" "azrtyuiop^$ù" null
null "Despicable you" null
null "" "azrtyuiop^$ù"
null null "Despicable you"
null "azrtyuiop^$ù" ""
"azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù"
"azrtyuiop^$ù" "" null
"azrtyuiop^$ù" null ""
"azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you"
16 test cases
instead of 64
24. Combinatorial Interaction Testing (CIT)
t1 t2 t3
"Despicable you" "Despicable you" "Despicable you"
"" "" ""
null null null
"azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù"
Hyp.: most faults are caused by interactions of at most two factors
Pairwise testing: cover all pairs of values
"Despicable you" "Despicable you" "Despicable you"
"Despicable you" "" ""
"Despicable you" null null
"Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù"
"" "Despicable you" ""
"" "" "Despicable you"
"" null "azrtyuiop^$ù"
"" "azrtyuiop^$ù" null
null "Despicable you" null
null "" "azrtyuiop^$ù"
null null "Despicable you"
null "azrtyuiop^$ù" ""
"azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù"
"azrtyuiop^$ù" "" null
"azrtyuiop^$ù" null ""
"azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you"
16 test cases
instead of 64
25. Combinatorial Interaction Testing (CIT)
t1 t2 t3
"Despicable you" "Despicable you" "Despicable you"
"" "" ""
null null null
"azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù"
Hyp.: most faults are caused by interactions of at most two factors
Pairwise testing: cover all pairs of values
"Despicable you" "Despicable you" "Despicable you"
"Despicable you" "" ""
"Despicable you" null null
"Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù"
"" "Despicable you" ""
"" "" "Despicable you"
"" null "azrtyuiop^$ù"
"" "azrtyuiop^$ù" null
null "Despicable you" null
null "" "azrtyuiop^$ù"
null null "Despicable you"
null "azrtyuiop^$ù" ""
"azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù"
"azrtyuiop^$ù" "" null
"azrtyuiop^$ù" null ""
"azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you"
16 test cases
instead of 64
26. Combinatorial Interaction Testing (CIT)
t1 t2 t3
"Despicable you" "Despicable you" "Despicable you"
"" "" ""
null null null
"azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù"
Hyp.: most faults are caused by interactions of at most two factors
Pairwise testing: cover all pairs of values
"Despicable you" "Despicable you" "Despicable you"
"Despicable you" "" ""
"Despicable you" null null
"Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù"
"" "Despicable you" ""
"" "" "Despicable you"
"" null "azrtyuiop^$ù"
"" "azrtyuiop^$ù" null
null "Despicable you" null
null "" "azrtyuiop^$ù"
null null "Despicable you"
null "azrtyuiop^$ù" ""
"azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù"
"azrtyuiop^$ù" "" null
"azrtyuiop^$ù" null ""
"azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you"
16 test cases
instead of 64
27. Combinatorial Interaction Testing (CIT)
t1 t2 t3
"Despicable you" "Despicable you" "Despicable you"
"" "" ""
null null null
"azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù"
Hyp.: most faults are caused by interactions of at most two factors
Pairwise testing: cover all pairs of values
"Despicable you" "Despicable you" "Despicable you"
"Despicable you" "" ""
"Despicable you" null null
"Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù"
"" "Despicable you" ""
"" "" "Despicable you"
"" null "azrtyuiop^$ù"
"" "azrtyuiop^$ù" null
null "Despicable you" null
null "" "azrtyuiop^$ù"
null null "Despicable you"
null "azrtyuiop^$ù" ""
"azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù"
"azrtyuiop^$ù" "" null
"azrtyuiop^$ù" null ""
"azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you"
16 test cases
instead of 64
34. type
Android
version
Screen Processor
tablet 4.0.4 XLarge ARM Cortex
smartphone 4.1 Large Intel x86
phablet 4.1.1 Normal
4.1.2 Small
… or using a feature model
http://familiar-project.github.io
https://featureide.github.io
http://www.skalup.com/
http://research.henard.net/SPL/PLEDGE/
http://martinfjohansen.com/models2011/spltool/
http://cse.unl.edu/~citportal/
35. app.
Hyp.: most faults are caused by interactions of at most t factors
T-wise testing: cover all t-uples of values
Integration and System testing
Which
configuration
should I use
for my tests?
Under …
• tests requirements
• $$$ budget
• test execution time
budget
… constraints