1. A Tool to Support Testing Activities
Gilson Doi Junior Adilson Luiz Bonif´cio
a
November 7th, 2011
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 1 / 15

2. Summary
1 Concepts
2 Motivation
3 Tool
Test case generation methods
FSM generation
4 Similar Tools
5 Experiment
6 Results
7 Future works
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 2 / 15

3. Concepts
Concepts
Testing
Check whether an implementation behaves according to the
specification
Guarantee reliable systems
Find faults (avoiding financial damage or human losses)
Model-based testing
Provide a rigorous testing framework
Black-box approach
Reduce ambiguities with formalisms
Support automated testing activities
Finite State Machine (FSM)
Specify input and outputs
It can be used to specify systems’ behavior
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 3 / 15

4. Motivation
Motivation
Few model-based testing tools support FSM models
Lack of a pool of models to support practical experiments
Lack of an standard to describe FSM
Tools that only implements one test case generation method
Compare some test case generation methods
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 4 / 15

5. Tool
Tool
Test suite Oracle
Specification Verdict
generator execution
Implementation
Pool FSM generator Pool of
size models
Figure: Tool’s architecture
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 5 / 15

6. Tool
Tool(2)
Specification and IUT
Describe FSM models
Test suite generator
Need the Specification model
Apply one of test case generation methods
Oracle execution
Apply the test suite to the specification and implementation model
Compare both behaviors and gives a verdict
FSM generator
Generate FSM models to provide a pool to the testing framework
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 6 / 15

7. Tool Test case generation methods
Test case generation methods
We implemented three methods to FSM test case generation closely
related, but distinct in the following aspects:
W-method
Uses the Characterization sets that distinguish every pair of states
Wp-method
Uses Identification sets that distinguish every single state
G-method
Uses R sets that distinguish families of equivalent states
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 7 / 15

8. Tool FSM generation
FSM generation(1)
Obtain the pool of FSM
Define a standard grammar to describe FSM
Similar approaches
Sidhu and Leung (1989) apply changes on a model to derive distinct
IUT
Sim˜o and Petrenko (2010) present a similar algorithm that obtain only
a
restrict models
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 8 / 15

9. Tool FSM generation
FSM generation(2)
Specifications
Generates FSM according to test case generation methods
Implementation candidates (IUT)
Are unknown to the tester
Generates FSM with few restrictions: upper bound on the number of
states and input and output alphabet
Distinct transitions can be obtained with distinct combinations of input
and output symbol for each pair of states
Distinct FSM are derived by creating distinct transitions on the same
pair of states
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 9 / 15

10. Similar Tools
Similar tools
Similar Tools:
JTorX
Spec Explorer
TGV
None of them:
Deals with FSM models
Generates formal models
Implements the W-method, Wp-method or G-method
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 10 / 15

11. Experiment
Practical Experiments(1)
A pool of FSM models obtained through the FSM generator
The test suite were obtained by the Test Suite Generator
Each FSM of the pool is tested as a IUT candidate
The results are checked by the oracle
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 11 / 15

12. Experiment
Practical Experiments(2)
b/0 b/0 a/1
a/0 a/0
s0 s2 s3 b/1
W-method Wp-method G-method
ba, aba, bba, ba, aba, baba, a, aaa, aba, baa, bba,
baba, bbba, bbba baaa, baba, bbba, bbaaa,
bbaba, bbbba bbaba, bbbaa, bbbba
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 12 / 15

13. Results
Results
[1] [2] [3] [4] [5]
5 Method W 91 24 0
5 Method Wp 57 17 0
5 Method G 432 72 0
10 Method W 88 22 0
10 Method Wp 86 21 0
10 Method G 2112 224 0
1 Number of states
2 Method
3 Test Suite Length
4 Number of Test Sequences
5 Implementations in conformance
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 13 / 15

14. Results
Future works
Replace QT data structures by C++ STL to improve the performance
A graphical interface to aid the task of specifying formal models
based on the proposed grammar
Integrate test case generation methods for real-time systems and
systems with data flow transformations
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 14 / 15

15. Results
Thanks
Questions?
Gilson Doi Junior, Adilson Luiz Bonif´cio ()
a A Tool to Support Testing Activities November 7th, 2011 15 / 15