DevOps has emerged, as a major cultural movement to handle the need for increased agility in software development. While this movement is loosely bound to development methods, a number of practices have emerged to operationalize this extreme agility: loosely coupled software architectures meant to support incremental updates and build (services or micro services with clear APIs); a very high degree of automation at multiple stages of the development lifecycle. Quality assurance is a major challenge in this context. There is very little time for manual testing and the responsibility of bug detection is placed upon the automated test suites. STAMP (Software Testing AMPlification), is a new European R&D project, which aims to bolster automated test suites through the automatic transformation of test assets. The key technical challenge that STAMP aims at overcoming is to reduce the cost due to regression bugs that propagate to production, through advanced research in automatic test generation. The key novelty of our research agenda is to leverage existing assets (such as test cases or execution logs) in order to increase test effectiveness. This innovative research is at the crossroads of program analysis and transformation, software testing, automatic deployment and search-based software engineering.

1. STAMP
Software testing amplification to increase quality
in DevOps
1

2. 2
DevOps

3. 3
DevOps
Test assets are available
BUT
They are manually developed
and their quality is not sufficient

4. 4
DevOps
The main objective of STAMP is to
automatically transform existing
test assets in order to detect
regression bugs before
production and drive down the
cost of software testing.

5. 5
STAMP in DevOps

6. Continuous test amplification
6

7. Continuous test amplification
7
behav. diff.

8. Continuous test amplification
8
behav. diff.

9. Continuous test amplification
9
behav. diff. perf. diff.

10. Continuous test amplification
10
behav. diff. perf. diff.

11. Continuous test amplification
11
behav. diff. perf. diff.

12. Unit test cases amplification
12
Program
P
Input space Output space
state

13. Unit test cases amplification
13
Program
P
Input space Output space
state

14. DSpot: amplification of JUnit test suites
14
original test suite and
program under test
update in the program
under test
P
DSpot
TS
P'
TS++

15. DSpot
•First experiments with amplification
•Automatic transformation of Junit test cases
15
P
TS
P'
TS+
search-based
input space
amplification
observation
space
amplification
run and look
for differences
TS++

16. Search-based input space amplification
• Mutate existing test cases
• transform literal values
• replace literal values by others found in the scope
• add new invocations to methods not well covered
• create new objects
• Search-based
• ‘stack’ transformations iteratively
• select the ones that have a good fitness (e.g. cover new
branches)
16

17. Observation space amplification
• Remove all assertions from original test cases
• keep invocations that can be inside the assertion
• Add new observation points
• run all test cases on original program
• record all values that can be observed (getters, is_*, etc.)
• Filter new observation points
• several runs of all test cases with observation points on
the original program
• discard all points that vary (e.g., OS-dependent values,
IO, etc.)
17

18. Example
18
public void testSearch() {
final ArrayStack<E> stack = makeObject();
stack.push(((E)("First Item")));
stack.push(((E)("Second Item")));
assertEquals("Top item is 'Second Item'", 1, stack.search("Second Item"));
assertEquals("Next Item is 'First Item'", 2, stack.search("First Item"));
assertEquals("Cannot find 'Missing Item'", -1, stack.search("Missing Item"));}
public void testSearch_cf5() {
final ArrayStack<E> stack = makeObject();
stack.push(((E)("First Item")));
stack.push(((E)("Second Item")));
E o_5 = stack.peek();
assertFalse(((java.lang.String)o_5).isEmpty());
E o_6 = stack.pop();
assertFalse(((java.lang.String)o_6).isEmpty());
Object o_7_0 = stack.search("Second Item");
Object o_9_0 = stack.search("First Item");
assertEquals("Cannot find 'Missing Item'", -1, stack.search("Missing Item"));
assertEquals(((java.lang.String)o_5).length(), 11);
assertEquals(((java.lang.String)o_6).length(), 11);}
new calls and asserts
extracted from asserts
new asserts

19. Methodology
19
Unit test ampli
Config. test ampli
Online test ampli

20. Methodology
20
Micro service
architecture
Unit test ampli
Config. test ampli
Online test ampli

21. Methodology
21
Micro service
architecture
Unit test ampli
Config. test ampli
Online test ampli
Information
management
Software
quality
Cloud
computing
E-Health
Smart cities

22. STAMP
•3-year project
•9 EU partners
•Full OSS
•Starts 01/12/16
22