Que vous soyez développeur ou entrepreneur, découvrez le projet STAMP piloté par Inria, l'institut national de recherche dédié aux sciences du numérique.

2. Comment améliorer la qualité de vos logiciels avec
STAMP ?
2
27/02/2019
Benoit Baudry
Project Coordinator and Scientific Leader
KTH, Sweden
Caroline Landry
Project Technical Manager
INRIA, France

3. •4 research
institutions
•5 companies
•1 open source
consortium
•Automated Testing
in DevOps
This work was partially supported by the EU Project
STAMP ICT-16-10 No.731529

4. DevOps
4
•High degrees of automation in software
development, deployment and operations
•Objectives
• Better quality
• Shorter release cycles
• Continuous feedback from Ops to Dev

5. DevOps
5

6. DevOps
unit
perf.
fuzzing
loggingdep. inj.
UI
CI
pertur-
bation
fault
recov.
IDEs
libraries
container
IDS
VMs
cluster
config.
6

7. DevOps
unit
perf.
fuzzing
loggingdep. inj.
UI
CI
pertur-
bation
fault
recov.
IDEs
libraries
container
IDS
VMs
cluster
config.
7
Automation

8. DevOps
unit
perf.
fuzzing
loggingdep. inj.
UI
CI
pertur-
bation
fault
recov.
IDEs
libraries
container
IDS
VMs
cluster
config.
8
Step acceptance & Feedback
Automation

9. Continuous Integration
Continuous Delivery. Jez Humble and David Farley. 2010. 9

10. The pull request loop
Collaborative
platform
pull req.
10

11. Collaborative
platform
The pull request loop
pull req.
code
11

12. The pull request loop
Collaborative
platform
pull req.
code
analyses
12

13. The pull request loop
Collaborative
platform
pull req.
code
analyses
feedback
13

14. The pull request loop
pull req.
code
analyses
feedback
14

15. The pull request loop
Collaborative
platform
pull req.
code
analyses
feedback
15

16. The pull request loop
Collaborative
platform
pull req.
code
analyses
feedback
16
Novel analyses

17. The pull request loop
Collaborative
platform
pull req.
code
analyses
feedback
17
Novel analyses

18. The pull request loop
Collaborative
platform
pull req.
code
analyses
feedback
18
Novel analyses

19. DevOps
19

20. DevOps – Testing
unit
perf.
fuzzing
logging
UI
pertur-
bation
fault
recov.
IDS
cluster
config.
20
IDEs

21. DevOps – Testing in Dev
21
Continuous Integration
- Unit, integration testing
- Coverage, mutation
Automatic test improvement
- Test amplification
- Test fixing
IDE
- Linters
- Completion

22. DevOps – Testing in Ops
22
Crash analysis
- Reproduction
- Localization
Online experiments
- Chaos
- A/B testing

23. 23

24. DevOps – STAMP focus
24

25. DevOps – STAMP focus
25

26. DevOps – STAMP focus
26

27. DevOps – STAMP focus
27

28. STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
https://dictionary.cambridge.org/dictionary/english/amplify 28

29. STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
 Test amplification: Increase the effect of test
assets
29https://dictionary.cambridge.org/dictionary/english/amplify

30. STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
 Test amplification: Increase the effect of test
assets
 Test assets: test cases, configuration files, production logs
 Effect metrics: mutation score, feature interactions
 Automatic amplification
30https://dictionary.cambridge.org/dictionary/english/amplify

31. Unit test amplification
31
Descartes &
DSpot

32. Descartes
•Assess the ability of your tests to reveal bugs
•Find weaknesses
• weak tests
• pseudo-tested methods
• non covered code
32

33. Descartes
33Apache Commons Collections
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new UnsupportedOperationException();
}
...
}

34. Descartes
34Apache Commons Collections
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new UnsupportedOperationException();
}
...
}
class SingletonListIteratorTest {
...
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
...
}

35. Descartes
35Apache Commons Collections
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new UnsupportedOperationException();
}
...
}
class SingletonListIteratorTest {
...
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
...
}
Pseudo-tested

36. Descartes
36Apache Commons Collections
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new UnsupportedOperationException();
}
...
}
class SingletonListIteratorTest {
...
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
...
}
No exception is thrown
A fail is needed here
Pseudo-tested

37. Descartes
37

38. DSpot
•Automatically enhances existing JUnit test suites
•Generate new assertions or new test cases
38

39. 39
@Test
public void html() {
Attribute attr = new Attribute("key", "value &");
assertEquals("key="value &"", attr.html());
assertEquals(attr.html(), attr.toString());
}
DSpot
Original
test case

40. 40
@Test
public void html_add33() throws Exception {
Attribute attr = new Attribute("key", "value &");
Assert.assertEquals("key="value &"", attr.html());
Assert.assertEquals("key="value &"", attr.toString());
Assert.assertEquals("key", attr.getKey());
Assert.assertEquals("value &", attr.getValue());
}
@Test
public void html() {
Attribute attr = new Attribute("key", "value &");
assertEquals("key="value &"", attr.html());
assertEquals(attr.html(), attr.toString());
}
DSpot
Original
test case
Generated
test case

41. 41
DSpot
Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.

42. Configuration testing amplification
42
CAMP

43. CAMP
43
…features and
constraints
Initial
config file
reusable pieces
Config ampli Config deploy
Amplified
config file
Amplified
config testing

44. 44
CAMP
goals:
running:
- Registry
components:
registry:
provides_services: [ Registry ]
requires_services: [ DB ]
requires_features: [ Ruby ]
implementation:
docker:
file: registry/Dockerfile

45. 45
CAMP
ruby:
provides_features: [ Ruby ]
variables:
version:
type: Text
values: [ v2.4, v2.5 ]
realization:
- targets: [ ruby/Dockerfile ]
pattern: "FROM ruby:2.5.1-alpine3.7"
replacements:
[ "FROM ruby:2.4.1-alpine3.7",
"FROM ruby:2.4.5-alpine3.7" ]
max_alloc:
type: Integer
values: [ 8, 16, 32, 64 ]
realization:
- targets: [ ruby/config.rb ]
pattern: ":malloc_limit=>16777216"
replacements:
- ":malloc_limit=>8388608"
- ":malloc_limit=>16777216"
- ":malloc_limit=>33554432"
- ":malloc_limit=>67108864"
implementation:
docker:
file: ruby/Dockerfile

46. 46
CAMP
mysql:
provides_services: [ DB ]
variables:
version:
type: Text
values: [ v5.6, v8.2]
realization:
- targets: [ docker-compose.yml ]
pattern: "image:mysql:5.7"
replacements: [ "image:mysql:5.6",
"image:mysql:8.2" ]
implementation:
docker:
image: mysql:5.6
mariadb:
provides_services: [ DB ]
variables:
version:
type: Text
values: [ v5.5, v10.1 ]
realization:
- targets: [ docker-compose.yml ]
pattern: "image:mysql:5.7"
replacements: [ "image:mariadb:5.5",
"image:mariadb:10.1" ]
implementation:
docker:
image: mariadb:5.5

47. 47
CAMP
Configuration specific bugs

48. Online test amplification
48
Botsing

49. Online test amplification
49
Botsing
Crash!?!java.lang.ClassCastException: […]
at org…..SolrEntityReferenceResolver.getWikiReference(....java:93)
at org…..SolrEntityReferenceResolver.getEntityReference(….java:70)
at org…..SolrEntityReferenceResolver.resolve(….java:63)

50. java.lang.ClassCastException: […]
at org…..SolrEntityReferenceResolver.getWikiReference(....java:93)
at org…..SolrEntityReferenceResolver.getEntityReference(….java:70)
at org…..SolrEntityReferenceResolver.resolve(….java:63)
at org…..SolrDocumentReferenceResolver.resolve(….java:48)
at …
Exception
Frames
{Target ➞
50
Java Stack Trace (Issue XWIKI-13031)
Botsing

51. public void test0() throws Throwable {
…
SolrEntityReferenceResolver solrEntityReferenceResolver0 = new …();
EntityReferenceResolver entityReferenceResolver0 = … mock(…);
solrDocument0.put("wiki", (Object) entityType0);
Injector.inject(solrEntityReferenceResolver0, …);
Injector.validateBean(solrEntityReferenceResolver0, …);
…
// Undeclared exception!
solrEntityReferenceResolver0.resolve(solrDocument0, entityType0, objectArray0);
}
java.lang.ClassCastException: […]
at org…..SolrEntityReferenceResolver.getWikiReference(....java:93)
at org…..SolrEntityReferenceResolver.getEntityReference(….java:70)
at org…..SolrEntityReferenceResolver.resolve(….java:63)
51
Botsing

52. Botsing
52

53. Contacts
https://github.com/STAMP-project/
http://stamp-project.eu/
baudry@kth.se
caroline.landry@inria.fr
http://www.diverse-team.fr/
barais@irisa.fr
53

54. 54
MERCI !

55. STAMP Activeeon Use Case
Mohamed Boussaa, February 27, 2019

56. 2
Resource
Manager
Workflow
Schedule
r
</>
ProActive Workflows & Scheduling
LS
F
Clusters Cloud Local
Manage resources and schedule execution
Customers around the World
Machine Learning, Data Scientists, Statisticians
https://www.activeeon.com/

57. 3
Pull request
● Maven/Gradle
● Dspot
● Descartes
● CI System-tests
● CAMP
STAMP tools
DEV
Proactive
Release
User
● Create issues
● Botsing
STAMP will help us improve our CI cycle:
● more tests added to our test suite
● better bug/regression detection

58. 4
CAMP for configuration testing
Proactive
Release
User
● CI System-tests
● CAMP => Generate different Proactive configurations
for database support

59. 5
CAMP for configuration testing
System tests
Generate

61. MERCI !