FYP research

1. Splat: Simple Python
Lazy Automated Tester
tiny.cc/lwy08
Lee Wei Yeong
lwy08@doc.ic.ac.uk

2. Introduction
2

3. Motivation
• writing unit tests
‣ crucial
‣ costly but invisible
‣ can be automated
3

4. • dynamically typed
• clean, concise & elegant syntax
• builtin introspection & reflection facility
4

5. Why bytecode?
• “Growing numbers of... closed source applications...
using...Python...” (Smith, 2010)
• simple
• fast
5

6. Background
6

7. Existing tools
• Pythoscope
• Pytestsgenerator
• ...but neither works!
7

8. Existing tools
• Ruby Test case Generator (RuTeG)
‣ structural testing
‣ evolutionary search
‣ tests object-oriented programs
8

9. Software testing
9

10. Sample test target
10

11. Sample unit test
11

12. SPLAT
12

13. Lazy instantiation
• technique inspired by IRULAN
• initially: None proxy metaclass wrapper
• test data only generated for active variables
13

14. Lazy instantiation
14

15. Lazy instantiation
LOAD_GLOBAL (A)
CALL_FUNCTION 0
STORE_FAST (arg1)
LOAD_CONST (‘arg1’)
LOAD_FAST (arg1)
STORE_ATTR (attr1)
LOAD_FAST (arg1)
RETURN_VALUE
15

16. Lazy instantiation
function1
(
Param1(None),
Param2(None),
LOAD_GLOBAL (A)
Param3(None)
CALL_FUNCTION 0
)
STORE_FAST (arg1)
LOAD_CONST (‘arg1’)
LOAD_FAST (arg1)
STORE_ATTR (attr1)
LOAD_FAST (arg1)
RETURN_VALUE
16

17. Lazy instantiation
function1
(
Param1(None),
Param2(None),
LOAD_GLOBAL (A)
‘default’
CALL_FUNCTION 0
)
STORE_FAST (arg1)
LOAD_CONST (‘arg1’)
LOAD_FAST (arg1)
STORE_ATTR (attr1)
LOAD_FAST (arg1)
RETURN_VALUE
17

18. Lazy instantiation
function1
(
Param1(None),
Param2(None),
LOAD_GLOBAL (A)
‘default’
CALL_FUNCTION 0
)
STORE_FAST (arg1)
LOAD_CONST (‘arg1’)
LOAD_FAST (arg1)
STORE_ATTR (attr1)
LOAD_FAST (arg1)
RETURN_VALUE
18

19. Lazy instantiation
TypeError!
LOAD_GLOBAL (A)
CALL_FUNCTION 0
STORE_FAST (arg1)
LOAD_CONST (‘arg1’)
LOAD_FAST (arg1)
STORE_ATTR (attr1)
LOAD_FAST (arg1)
RETURN_VALUE
19

20. Lazy instantiation
function1
(
{
‘attr1’:
Param1_attr1(None)
LOAD_GLOBAL (A) },
CALL_FUNCTION 0 Param2(None),
STORE_FAST (arg1) ‘default’
LOAD_CONST (‘arg1’) )
LOAD_FAST (arg1)
STORE_ATTR (attr1)
LOAD_FAST (arg1)
RETURN_VALUE
20

21. Lazy instantiation
function1
(
{
‘attr1’:
Param1_attr1(None)
LOAD_GLOBAL (A) },
CALL_FUNCTION 0 None,
STORE_FAST (arg1) ‘default’
LOAD_CONST (‘arg1’) )
LOAD_FAST (arg1)
STORE_ATTR (attr1)
LOAD_FAST (arg1)
RETURN_VALUE
21

22. Demonstration
22

23. Architecture
23

24. Challenges
• Vague error messages
• Lack of existing tool support
• Test program contains imports
24

25. Challenges
• Relationship between input arguments
• Various programming constructs
• Range of test data values
25

26. Limitations
• Custom Exceptions, e.g. InvalidGraphType
• Random functions
• Generators
26

27. Evaluation
27

28. Criteria
• Quality
• Performance
• Generality
28

29. Results
Python package
#
29

30. Results
Mean number of iterations to achieve 100% coverage = 497
30

31. Conclusion
31

32. Future work
• improve tool sophistication
• optimise using PyPy
• more comprehensive benchmarks
32

33. evandrix.github.com/Splat