In the context of testing of Object-Oriented (OO) software systems, researchers have recently proposed search based approaches to automatically generate whole test suites by considering simultaneously all targets (e.g., branches) defined by the coverage criterion (multi-target approach). The goal of whole suite approaches is to overcome the problem of wasting search budget that iterative single-target approaches (which iteratively generate test cases for each target) can encounter in case of infeasible targets. However, whole suite approaches have not been implemented and experimented in the context of procedural programs. In this paper we present OCELOT (Optimal Coverage sEarch-based tooL for sOftware Testing), a test data generation tool for C programs which implements both a state-of-the-art whole suite approach and an iterative single-target approach designed for a parsimonious use of the search budget. We also present an empirical study conducted on 35 open-source C programs to compare the two approaches implemented in OCELOT. The results indicate that the iterative single-target approach provides a higher efficiency while achieving the same or an even higher level of coverage than the whole suite approach.

1. Search-based Testing of Procedural Programs:
Iterative Single-Target or Multi-Target Approach?
Andrea De LuciaRocco Oliveto
Dario Di NucciSimone Scalabrino Giovanni Grano

2. “
The overall cost of testing has
been estimated at being at least
half of the entire development
cost, if not more.
Boris Beizer
Beizer. Software testing techniques. 2003. Dreamtech Press.

3. “
Software developers only spend a
quarter of their work time engineering
tests, whereas they think they test half of
their time.
Beller et al. When, how, and why developers (do not) test in their IDEs.
ESEC/FSE 2015
Beller et al.

4. Test case generation

5. Test case generation
SBST Methodologies

6. Test case generation
SBST Tools
AUSTIN
eToc - evolutionary Testing of classes

7. OCELOT
Optimal Coverage sEarch-based
tooL for sOftware Testing

8. Test Case Generation for C
Fully Implemented in Java and C (through JNI)
Based on JMetal Framework
Structs and Pointers Handling
Check Unit Testing Framework
OCELOT
Features

9. OCELOT
Why C?

10. Code
Execution
CE
SS
CI
TS
PB
Makefile
Generation
Program
Building
Target
Selection
Solution
Search
Code
Instrumentation
OCELOT
Process Overview
MG

11. MOSA
Multi-Objective Sorting Algorithm*
Panichella et al.
LIPS
Linearly Independent Path based Search
Scalabrino et al.
OCELOT
Target Selection
Algorithms

12. Many-Objective
Sorting Algorithm
Fitness function reformulation
Find a set of test cases that optimizes the branch coverage of each branch
Dominance and Pareto optimality
Each solution is evaluated in terms of Pareto dominance and optimality,
Preference Sorting
A new ranking algorithm for sorting the solutions

13. Linearly
Independent
Path based Search
Inspired by McCabe baseline method and
Dynamic Symbolic Execution
● Starts with random test data (t0
)
● Selects a new target at each iteration
a > 1
a++ a--
End
Start
Single target selection algorithm
Collateral coverage
Considers the targets serendipitously
achieved
Search budget optimization
Allocates SBi
/ni
evaluations for each target
Seeding
Final population of the previous iteration is
reused
Independent of the search algorithm
Current implementation based on GA

14. EMPIRICAL EVALUATION

15. Research Questions
RQ1
Effectiveness
Branch
Coverage
RQ3
Oracle Coast
Test Suite
Size
RQ2
Efficiency
Execution
Time
MOSA LIPS

16. Design
Experiment Details
Settings
Context
Population Size 100
Crossover Rate 0.90
Mutation Rate 1/#variables
Search Budget 200.000
35 C functions
605 branches
Gimp: Gnu Image Manipulation Program
GSL: Gnu Scientific Library
SGLIB: a generic library for C
spice: analogue circuit simulator
30 runs
Average Values
Wilcoxon’s Test (p-value 0.05)
Vargha-Delaney Test

17. Results RQ1:
Effectiveness
MOSA LIPS
84,73% 86.29%
Overall Branch
Coverage
Cases in which
is better
21
102
1
1 case with large effect size
2
8 cases with medium/large effect size

18. MOSA LIPS
14.80s 5.03s
Average
Execution Time
Cases in which
is better
0 351
Results RQ2:
Efficiency
1
with large effect size

19. MOSA LIPS
14.80s 5.03s
Average
Execution Time
Cases in which
is better
0 351
Results RQ2:
Efficiency
1
with large effect size
Too much time for
ranking the
Pareto Fronts!

20. Results RQ3:
Oracle Cost
MOSA LIPS
4.4 6.1
Average
# Test Cases
Cases in which
is better
351
0
1
33 cases with large effect size

21. Results RQ3:
Oracle Cost
MOSA LIPS
4.4 6.1
Average
# Test Cases
Cases in which
is better
351
0
1
33 cases with large effect size
LIPS does not directly
handle the oracle
problem!

22. LIPS*
no collateral coverage
MOSA LIPS*
4.4 4.25
Average
# Test Cases
Cases in which
is better
6 7
1
Although better than MOSA
Worse than LIPS in terms of branch coverage
Worse than LIPS in terms of execution time1

23. LIPS + minimization
MOSA1
LIPS
3.61 3.66
Average
# Test Cases
Cases in which
is better
62
23
1
For a fair comparison the minimization was applied also to MOSA suites
2
4 cases with medium/large effect size
3
1 case with large effect size

24. LIPS + minimization
1
For a fair comparison the minimization was applied also to MOSA suites
2
4 cases with medium/large effect size
3
1 case with large effect size
Minimization
execution
time < 1s
MOSA1
LIPS
3.61 3.66
Average
# Test Cases
Cases in which
is better
62
23

25. Conclusions

26. Conclusions

27. Conclusions

28. Conclusions

29. Conclusions

30. Conclusions

31. Conclusions

32. Conclusions

33. Conclusions

34. Future works

35. Future works
Replicate on larger dataset

36. Future works
Replicate on larger dataset
Study C landscapes

37. Future works
Replicate on larger dataset
Study C landscapes
Add LIPS to Evosuite

38. Future works
Replicate on larger dataset
Study C landscapes
Add LIPS to Evosuite
Go beyond
Genetic Algorithm

39. Dario Di Nucci
University of Salerno
ddinucci@unisa.it
http://www.sesa.unisa.it/people/ddinucci/
Thanks for your attention!
Questions?