"Software Testing is the process of executing a program or system with the intent of finding errors.", Myers, 1979. The most important activity in this process is designing the required set of effective test cases. Thus, the problem is narrowed down to determining the exact number of required test cases and increasing their effectiveness. Project FoX is a production ready tool developed in Java, which offers Java developers the opportunity to leverage the proven theories and concepts of formal testing using generalized state automata (X-Machines) as a theoretical model of computation. The formal testing strategy FoX is applying, is proven to generate a complete test set that ensures the correctness of the implementation with respect to the specification. FoX enhances a novel testing process that is fully automated, ranging from complete test set generation, to test preparation and execution. This method can be applied to any Java based software system, regardless of its underlying technologies. Utilizing a formal approach will provide unambiguous test cases which are objective and not subjective to the tester’s experience and intuition. The formal testing strategy provides functional testing that tests not only for the desired system behaviour (the system does what it should) but also tests that the system has no undesired behaviour (the system does not do anything it should not do). This short presentation will strive to give the audience an overview of the formal testing methodology and a demonstration of the tool (FoX). It will also showcase a real-life demo of the project FoX, applied to a Java SE application and will discuss how the methodology can be applied to any Java EE or ME application. Anyone with a software engineering background will be able to easily follow the talk and understand the benefits which this process offers to modern day software engineering.

1. Project FoX: A Tool That Offers
Automated Testing Using a
Formal Approach
Ivo Neskovic

2. Agenda
• Software Engineering: What could go wrong?
• Formal Methods
• Project FoX
• Case Study: The buffer system
• Conclusions and Future Work
• Bibliography

3. The Problem of Software Engineering
• Faulty systems are a common notion nowadays.
• SE is an engineering discipline, yet lacking the
engineering formality.
• Subjective and informal testing.
• Impossible to prove that the system:
– Does what it is supposed to do.
– Does not do what it is not supposed to do.
• Needs structured and precise system designs.

4. Formal Methods
• The applied mathematics of computer systems engineering, used to
specify and model the behaviour of a system and mathematically verify
that the system design and implementation satisfy functional and safety
properties.
• Specification Languages:
– Abstract State Machines
– Generalized State Machines
– Communicating Sequential Processes
– Specification and Description Language
– Petri Nets
– Temporal Logic of Actions
– B and event – B method
– Z

5. Formal Methods at a Trial
• Benefits:
– Specification may be used as a basis for proving the presence or lack
of certain properties in the design and by inference in the developed
system.
– Mathematical proof of correctness (Theorem proving).
– Model checking (Proving desired properties in a design).
– Formal Testing.
• Used mainly for safety critical systems such as aerospace
engineering.
• Criticism:
– Expensive and time consuming approach (though questionable).
– Lack of tooling support.

6. Incorporating Formal Methods in the
Development Cycle

7. Project FoX
• Produce the complete set of test cases from a formal specification.
• Execute the tests on the systems implementation.
• Locate errors and non-equivalences and report them to the user.
• Developed in Java for Java.
• Compatible with Java Standard Edition, Enterprise Edition, Mobile
Edition.
• Can be extend to work in conjunction with popular Java frameworks.
• Operates on compiled bytecode with the addition of a few specific
annotations.
• Utilizes the test drivers of JUnit.
• FoX provides a bridge between regular Java developers and the benefits
of complete positive and negative testing, proven to find all faults.

8. Using Project FoX
• Two artefacts necessary:
– Formal specification of the system.
– The system’s implementation.

9. Buffer Case Study – Description
• Simple buffer in a factory.
• Accepts parts, any parts.
• Parts have a name and an ID.
• The buffer has a capacity of 2.
• The buffer can be empty, partially
full or completely full.
• Supports adding and removing
items.
• If the capacity is reached, no
additional items can be placed in
the buffer unless and item is
removed firsts.

10. Buffer Case Study – Formal
Specification
• Modelled as a Generalized State
Machine (stream X-Machine).
• A theoretical model of computing,
pioneered by Samuel Eilenberg
in1974 (X-Machine).
• Separates flow control from
processing.
• Flow control is abstracted to a level
suitable for representation as a finite
state machine.
• Complex data structures are modelled
as an infinite memory.
• Able to model both static (data) and
dynamic (control) parts of a system.

11. Buffer Case Study – Formal
Specification (cont.)
• Simple buffer in a factory.
< xMachine name = " Buffer " >
• The buffer can be empty, partially full or completely full.
< states >
< state initialState = " true " > empty </ state >
< state > non_empty </ state >
< state > full </ state >
</ states >

12. Buffer Case Study – Formal
Specification (cont.)
• Accepts parts, any parts.
< input name = " part " ref = " BufferObject " / >
• The buffer has a capacity of 2.
< types >
< builtInType name = " capacity " type = " integer " / >
< builtInType name = " buffer " type = " set: BufferObject " / >
</ types >
< memory >
< memoryBlock ref = " buffer " initialValue = " null " / >
< memoryBlock ref = " capacity " initialValue = " 2 " / >
</ memory >

13. Buffer Case Study – Formal
Specification (cont.)
• Parts have a name and an ID.
< types >
< complexType name = " ItemType " >
< attributes >
< builtInType name = " type " type = " string " / >
</ attributes >
</ complexType >
< complexType name = " BufferObject " >
< attributes >
< complexType name = " type " ref = " ItemType " / >
< builtInType name = " itemId " type = " integer " / >
</ attributes >
</ complexType >
< /type >

14. Buffer Case Study – Formal
Specification (cont.)
• Supports adding and removing items. < transitions >
< functions > < transition >
< function name = " add_part " > < startingState >
< guard > empty
!buffer. contains ( part ) && buffer . </ startingState >
size () + 1 < capacity . value () < appliedFunction >
</ guard > add_part
< body > buffer . add ( part ) ; </ body > </ appliedFunction >
< output > Part Added </ output > < endingState >
</ function > non_empty
... </ endingState >
</ functions > </ transition >
...
</ transitions>

15. Buffer Case Study – Implementation
public class BufferObject { public class ItemType {
private int itemId; private String type;
private ItemType type;
public ItemType(String type) {
public BufferObject(int itemId, this.type = type;
ItemType type) {
}
this.itemId = itemId;
}
this.type = type;
}
}

16. Buffer Case Study – Implementation
• @Xmachine - annotating the class representing the system modeled
with the specification.
• XMachineModel – a class representing the model, containing a number
of useful helper methods.
@XMachine(inputType = "BufferObject",
sampleInputs = {
"integer: 10, ItemType: (string:Box)",
"integer: 17, ItemType: (string:HeavyBox)",
"integer: 25, ItemType: (string:ReallyHeavyBox)"
})
public class Buffer extends XMachineModel {

17. Buffer Case Study – Implementation
• @XMMemoryBlock – a field level annotation, associating Java data
structures with their specification equivalents.
@XMMemoryBlock(name = "buffer")
private List<BufferObject> buffer;
@XMMemoryBlock(name = "capacity")
private int capacity;
public Buffer() {
super("Buffer");
buffer = new LinkedList<BufferObject>();
capacity = 2;
}

18. Buffer Case Study – Implementation
• @XMFunction – a method level annotation, referencing the
modeled functions implementations.
• reportOutcome( outcome: String) – one of the many helper
methods of the XMachineModel class.
@XMFunction(name = "add_part")
public void addPart(BufferObject part) {
if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
reportOutcome("Part Added");
}

19. Buffer Case Study – Executing Fox

20. Buffer Case Study – Executing FoX
(implanted error)
if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
capacity++;
reportOutcome("Part Added");
}

21. Buffer Case Study – Generated Test
Cases
• Tests report the sequence of inputs used for the specific scenario, the
sequence of expected outputs and the actual output.
• Outcome is reported to the user via the usual JUnit red / green
notifications.
<tests>
…
<test testID=”2”>
<input>[ itemId: 17 type: HeavyBox, itemId: 10 type: Box]</input>
<expectedOutput>
[ Part Added, Part Added – Become Full ]
</expectedOutput>
<output>[ Part Added, Part Added – Become Full ]</output>
</test>
…
</tests>

22. Conclusions and Future Work
• FoX enables developers to leverage the already
proven theories for formal testing.
• Provides a fully automated testing process, ranging
from complete test set generation (satisfying some
design for test conditions), to test preparation and
execution.
• Operates on any Java based software system,
being transparent to it's underlining technologies.
• Provides complete positive and complete negative
testing.

23. Conclusions and Future Work (cont.)
• Next steps:
– Thorough evaluation.
– An additional tool to make the specification step easier
and closer to the developer, aiming to “hide” the formality
as much as possible.
– NetBeans and Eclipse integration.
– A standalone X-Machine IDE providing additional related
functionalities.
– Branch out to other languages and frameworks (eg. C#
and .NET).

24. Bibliography
• S. Eilenberg, Automate, Languages and Machines, Vol. A. Academic Press,
London, 1974.
• M. Holcombe, “X-Machines as a basis for dynamic system specification,”
Software Engineering Journal, vol. 3(2), pp. 69-76, 1988.
• F. Ipate and M. Holcombe, “Specification and Testing using Generalized
Machines: a Presentation and a Case Study,” Softw. Test. Verif. Reliab, vol. 8,
pp. 61-81, 1998.
• M. Holcombe and F. Ipate, Correct Systems: Building a Business Process
Solution. Springer, Applied Computing Series, November 1998.
• G. Eleftherakis and A. Cowling, “An Agile Formal Development Methodology,” in
1st South Eastern European workshop on Formal Methods (SEEFM 03),
(Thessaloniki), pp. 36-47, Nov. 2002. Agile Formal Methods: Practical, Rigorous
Methods for a changing world.
• P. Kefalas, G. Eleftherakis, and E. Kehris, “Communicating X-Machines: a
practical approach for formal and modular specification of large systems,”
Information and Software Technology, vol. 45, pp. 269-280, Apr. 2003.

25. Thank you
• Contact:
– ivo.neskovic@gmail.com
– http://twitter.com/trumpets