Automatic Test Case Generation for Function Block Diagrams

1. Automatic
Test Case Generation
for Train Control Systems
Adnan Causevic
TOCSYC Industrial Workshop, 2014-11-13, Västerås

2. 2
Today’s talk is about…

3. ● Software Testing Laboratory @ MDH
● Research:
● Test Automation
● Quality of Tests
● Developers Testing
● AGENTS - KKS funded project
● Bombardier Transportation
● Maximatecc
● Involved in the TOCSYC
research environment
● Intelligent Embedded Systems
Master Program Coordinator
3
About me

4. ● ATAC research project
● 2011-2014
● Funded by ITEA2/Vinnova
● Advanced Test Automation for Complex Software-
Intensive System
4
Cooperation with Bombardier
“The project has brought test automation
on the agenda of European industry.”
- ITEA3 Final Review Report

5. ● Train Control Management System (TCMS)
● a high capacity, infrastructure backbone
● the center of the distributed system
● Functions controlled by TCMS include
● collecting line voltage,
● controlling the train engines,
● opening and closing the train doors,
● upload of diagnostic data.
5
About Bombardier

6. ● Vehicle
● Vehicle Integration testing
● System
● System Integration testing
● Software
● Modules integration testing
● Modules testing
● “Get to the driver’s perspective as soon as possible”
6
About Bombardier
… and their testing
Responsibility
of a Developer!

7. ● IEC 61131-3
● Function Block Diagram
● Graphical programming language
● Based on element composition
● Various predefined elements
7
Development Environment

8. ● Coverage
● Strict measurements mandated by safety standards
● Tooling
● No tool support to measure coverage on FBD
● Time
● Creating tests just to reach the coverage leaves very little
time for functional testing
8
Current challenges
… with module testing

9. 9
CompleteTest

10. 10
Research approach
FBD Program
Transformation 1 Annotation2
UPPAAL timed
automata
...
Reachability
properties
∃
♢
β
,
∃
♢
pi
,
∃
♢
c0
and
c1
Test traces Test cases
(Step) (1)
(Time) (20s)
(Inputs) (0 1 23)
(Outputs) (1 2 543,9)
UPPAAL
model checker
Test Generation
3
Logic-­‐‑based
coverage criteria
DC, CC,
MC/DC
”Automated Test Generation using Model-Checking: An Industrial Evaluation”, Eduard Paul Enoiu, Adnan
Causevic, Thomas J. Ostrand, Elaine J. Weyuker, Daniel Sundmark, Paul Pettersson. International Journal
on Software Tools for Technology Transfer, 2014, Springer.

11. ● Time required to generate tests
● satisfying the DC, CC and MC/DC logic coverage criteria
11
Case Study at BT

12. ● For 34 of the 157 programs, the tool did not terminate
after running for a substantial period of time.
● Cut-off time was set to 10 minutes.
● There is an indication that as the number of decisions
increases, the performance deteriorates and the cost of
using the tool may become prohibitive.
● But, if you have so many decisions, how would you test
it manually?
12
Case Study at BT (2)

13. ● Should a developer generate the tests from scratch?
● Should a developer provide manual tests first?
● How many (or how much in terms of coverage)?
● When should a developer use the Validate Test button?
● Should developers use this tool at all or should we
automate that as well (nightly build, etc.) ?
13
Best Practices?

14. ● Internal behavior of each function block has to be
modeled
● Currently, blocks from the standard language are modeled
● and a few custom made by Bombardier which are often used
● State-space explosion
● a known problem of model-checking
● large number of Boolean or Integer input values
● but, again, how would you do it manually ?
14
Limitations

15. ● Complementing model-checking with:
● Static analysis of FBD’s
● Search-based software testing
● Trying some new approaches:
● Different model-checkers, SMT solvers, …
● Measuring effectiveness of generated tests
● In terms of how good they are in finding faults
● Experimenting with FBD programs from
other organizations
15
On-going and Future work

16. ● Adnan Causevic
adnan.causevic@mdh.se
● Eduard Enoiu
eduard.paul.enoiu@mdh.se
● www.completetest.org
16
Questions?