Presentation at the Lorentz workshop on In-Vivo Analytics for Big Software Quality
https://www.lorentzcenter.nl/lc/web/2018/1008/info.php3?wsid=1008&venue=Oort
2. DevOps
•High degrees of automation in software
development, deployment and operations
•Objectives
•Better quality
•Shorter release cycles
•Continuous feedback from Ops to Dev
Automatic Testing in DevOps - Lorentz Workshop 2
11. Continuous integration
Continuous Integration
- Unit, integration testing
- Coverage, mutation
- Test generation and repair
Automatic test improvement
- Test refactoring
- Test amplification
- Test fixing
IDE
- Linters
- Completion
Automatic Testing in DevOps - Lorentz Workshop 11
13. The pull request loop
Collaboration
platform
pull req.
Automatic Testing in DevOps - Lorentz Workshop 13
14. The pull request loop
Collaboration
platform
pull req.
code
Automatic Testing in DevOps - Lorentz Workshop 14
15. The pull request loop
Collaboration
platform
pull req.
code
analyses
Automatic Testing in DevOps - Lorentz Workshop 15
16. The pull request loop
Collaboration
platform
pull req.
code
analyses
feedback
Automatic Testing in DevOps - Lorentz Workshop 16
17. The pull request loop
pull req.
code
analyses
feedback
Automatic Testing in DevOps - Lorentz Workshop 17
18. The pull request loop for research
Collaboration
platform
pull req.
code
analyses
feedback
Empirical analyses
Automatic Testing in DevOps - Lorentz Workshop 18
19. The pull request loop for research
Collaboration
platform
pull req.
code
analyses
feedback
Novel analysesAutomatic Testing in DevOps - Lorentz Workshop 19
20. The pull request loop for research
Collaboration
platform
pull req.
code
analyses
feedback
Novel analysesAutomatic Testing in DevOps - Lorentz Workshop 20
21. Revisit test prioritization in the CI
•Test priorization orders tests to detect
failures faster
•Non-CI approaches reorder tests cases
according to a change
•Proposal: in the CI, reorder commits to be
executed
Redefining Prioritization: Continuous Prioritization for Continuous Integration. J. Liang, S. Elbaum, G. Rothermel. ICSE 2018.
Automatic Testing in DevOps - Lorentz Workshop 21
22. Revisit test prioritization in the CI
Redefining Prioritization: Continuous Prioritization for Continuous Integration. J. Liang, S. Elbaum, G. Rothermel. ICSE 2018.
Automatic Testing in DevOps - Lorentz Workshop 22
23. Revisit test prioritization in the CI
Redefining Prioritization: Continuous Prioritization for Continuous Integration. J. Liang, S. Elbaum, G. Rothermel. ICSE 2018.
Automatic Testing in DevOps - Lorentz Workshop 23
24. Analysis of code coverage evolution
•“Statement coverage … reduces the quality
measure to a single ratio, making
developers potentially miss valuable
information about their test suite and its
limitations”
•7,816 revisions of 47 projects
•http://www.code-coverage.org
Automatic Testing in DevOps - Lorentz Workshop 24
A Large-Scale Study of Test Coverage Evolution. M. Hilton, J. Bell, D. Marinov. ASE 2018.
25. Analysis of code coverage evolution
A Large-Scale Study of Test Coverage Evolution. M. Hilton, J. Bell, D. Marinov. ASE 2018.
Automatic Testing in DevOps - Lorentz Workshop 25
26. Deflaker
•Flaky tests are tests which verdict changes
even if the code does not change
•Deflaker is a new bot to automate the
detection of flaky tests
•http://www.deflaker.org
DeFlaker: Automatically Detecting Flaky Tests. J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov2. ICSE 2018.
Automatic Testing in DevOps - Lorentz Workshop 26
27. Deflaker
DeFlaker: Automatically Detecting Flaky Tests. J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov2. ICSE 2018.
Automatic Testing in DevOps - Lorentz Workshop 27
28. DeFlaker: Automatically Detecting Flaky Tests. J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov2. ICSE 2018.
Automatic Testing in DevOps - Lorentz Workshop 28
29. Repairnator
•Automatic repair bot to target build failures
•Runs since February 2017
•ICSE SEIP paper reports on
•11 523 test failures over 1 609 open-source
software projects hosted on GitHub
•generated patches for 15 different bugs
How to Design a Program Repair Bot? Insights from the Repairnator Project. S. Urli, Z. Yu, L. Seinturier, M. Monperrus. ICSE SEIP 2018.
Automatic Testing in DevOps - Lorentz Workshop 29
30. Repairnator
How to Design a Program Repair Bot? Insights from the Repairnator Project. S. Urli,
Z. Yu, L. Seinturier, M. Monperrus. ICSE SEIP 2018.
30
31. Repairnator
How to Design a Program Repair Bot? Insights from the Repairnator Project. S. Urli, Z. Yu, L. Seinturier, M. Monperrus. ICSE SEIP 2018.
Automatic Testing in DevOps - Lorentz Workshop 31
32. DSpot
•Amplify existing unit test cases
•Start from developers’ test cases
•Automatically generate variants
•Submit pull requests
Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.
Automatic Testing in DevOps - Lorentz Workshop 32
33. DSpot
Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.
@Test
public void html() {
Attribute attr = new Attribute("key", "value &");
assertEquals("key="value &"", attr.html());
assertEquals(attr.html(), attr.toString());}
Automatic Testing in DevOps - Lorentz Workshop 33
34. DSpot
Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.
@Test
public void html() {
Attribute attr = new Attribute("key", "value &");
assertEquals("key="value &"", attr.html());
assertEquals(attr.html(), attr.toString());}
@Test
public void html_add33() throws Exception {
Attribute attr = new Attribute("key", "value &");
Assert.assertEquals("key="value &"", attr.html());
Assert.assertEquals("key="value &"", attr.toString());
Assert.assertEquals("key", attr.getKey());
Assert.assertEquals("value &", attr.getValue()); }
Automatic Testing in DevOps - Lorentz Workshop 34
35. DSpot
Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.
Automatic Testing in DevOps - Lorentz Workshop 35
36. DSpot
Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.
Automatic Testing in DevOps - Lorentz Workshop 36
37. Mutation at Google
•Mutation testing assesses the validity of test
cases
•Inject bugs in code
•Check that test cases detect the bugs
•Traditional mutation does not scale
•Google has developed a mutation approach
for their PR-loop
State of Mutation Testing at Google. G. Petrovic, M. Ivankovic. ICSE SEIP 2018.
Automatic Testing in DevOps - Lorentz Workshop 37
38. Mutation at Google
State of Mutation Testing at Google. G. Petrovic, M. Ivankovic. ICSE SEIP 2018.
Automatic Testing in DevOps - Lorentz Workshop 38
39. Mutation at Google
State of Mutation Testing at Google. G. Petrovic, M. Ivankovic. ICSE SEIP 2018.
Automatic Testing in DevOps - Lorentz Workshop 39
40. Mutation at Google
State of Mutation Testing at Google. G. Petrovic, M. Ivankovic. ICSE SEIP 2018.
40
41. Sapienz
•Automatic test generation and repair
•Black-box, system level
•Deployed in the Facebook PR-loop
•All tests, bugs and patches are reviewed by
developers
Automatic Testing in DevOps - Lorentz Workshop 41Deploying Search Based Software Engineering with Sapienz at Facebook. N. Alshahwan, X. Gao, M. Harman, Y. Jia, K. Mao, A. Mols, T. Tei, and I. Zorin. SSBSE 2018.
https://code.fb.com/developer-tools/finding-and-fixing-software-bugs-automatically-with-sapfix-and-sapienz/
42. Sapienz
Automatic Testing in DevOps - Lorentz Workshop 42Deploying Search Based Software Engineering with Sapienz at Facebook. N. Alshahwan, X. Gao, M. Harman, Y. Jia, K. Mao, A. Mols, T. Tei, and I. Zorin. SSBSE 2018.
https://code.fb.com/developer-tools/finding-and-fixing-software-bugs-automatically-with-sapfix-and-sapienz/
43. Sapienz
•In production since September 2017
•75% of bugs reported have been fixed
•Sapfix deployed in August 2018
•Some automatic patches accepted
Automatic Testing in DevOps - Lorentz Workshop 43Deploying Search Based Software Engineering with Sapienz at Facebook. N. Alshahwan, X. Gao, M. Harman, Y. Jia, K. Mao, A. Mols, T. Tei, and I. Zorin. SSBSE 2018.
https://code.fb.com/developer-tools/finding-and-fixing-software-bugs-automatically-with-sapfix-and-sapienz/
45. Crash analysis
- Reproduction
- Localization
Online experiments
- Chaos
- A/B testing
Testing in Ops
•Test in production
•Requires production
•Industry-driven
•Test in the feedback
•Little research
Automatic Testing in DevOps - Lorentz Workshop 45
46. Online experiments
•Software testing is all about experiments
•Express expected behavior (oracle)
•Run system to check actual vs. Expected (test case)
•On the Ops side, one can perform
experiments with the real system
•Express hypothesis
•Run experiments to check validity
Automatic Testing in DevOps - Lorentz Workshop 46
49. A/B testing
•Modular features, feature toggles
•Graphical elements
•Game levels
•Sample and track a population
of users
•Precise metrics about the value of a
feature
Automatic Testing in DevOps - Lorentz Workshop 49
50. Chaos engineering
•Breaking things on purpose in order to build
more resilient systems!
https://principlesofchaos.org/
Automatic Testing in DevOps - Lorentz Workshop 50
51. Principles of chaos engineering
•Build a Hypothesis around Steady State
Behavior
•Vary Real-world Events
•Run Experiments in Production
•Automate Experiments to Run Continuously
•Minimize Blast Radius
https://principlesofchaos.org/
Automatic Testing in DevOps - Lorentz Workshop 51
52. Netflix’s simian army
•Induce failure regularly
• ‘break’ production code to check the
system’s ability to react
• Chaos monkey: randomly terminates an
instance in production
• Chaos kong: take an entire region offline
• Latency monkey: artificial delay in
RESTful clients
52Automatic Testing in DevOps - Lorentz Workshop
53. Chaos engineering
•Growing adoption: SDN, CDN, JVM
•Resilience of large distributed systems
•Online perturbation
•Monitor and report
•Loosely coupled architectures
•Macro-level health metric
Automatic Testing in DevOps - Lorentz Workshop 53
54. Crash reproduction
54
java.util.NoSuchElementException
at org.xwiki.rendering.listener.chaining.EmptyBlockChainingListener.stopContainerBlock(EmptyBlockChainingListener.java:458)
at org.xwiki.rendering.listener.chaining.EmptyBlockChainingListener.endFormat(EmptyBlockChainingListener.java:263)
at org.xwiki.rendering.listener.chaining.AbstractChainingListener.endFormat(AbstractChainingListener.java:290)
at org.xwiki.rendering.listener.chaining.BlockStateChainingListener.endFormat(BlockStateChainingListener.java:439)
at org.xwiki.rendering.listener.chaining.AbstractChainingListener.endFormat(AbstractChainingListener.java:290)
at org.xwiki.rendering.listener.CompositeListener.endFormat(CompositeListener.java:253)
at org.xwiki.rendering.internal.parser.wikimodel.DefaultXWikiGeneratorListener.flushFormat(DefaultXWikiGeneratorListener.java:325)
at org.xwiki.rendering.internal.parser.wikimodel.DefaultXWikiGeneratorListener.flushFormat(DefaultXWikiGeneratorListener.java:273)
at org.xwiki.rendering.internal.parser.wikimodel.DefaultXWikiGeneratorListener.flushFormat(DefaultXWikiGeneratorListener.java:267)
at org.xwiki.rendering.internal.parser.wikimodel.DefaultXWikiGeneratorListener.onWord(DefaultXWikiGeneratorListener.java:906)
at org.xwiki.rendering.wikimodel.impl.InternalWikiScannerContext.onWord(InternalWikiScannerContext.java:1147)
at org.xwiki.rendering.wikimodel.impl.WikiScannerContext.onWord(WikiScannerContext.java:597)
at org.xwiki.rendering.wikimodel.xhtml.impl.TagStack.flushStack(TagStack.java:204)
at org.xwiki.rendering.wikimodel.xhtml.impl.TagStack.onCharacters(TagStack.java:227)
at org.xwiki.rendering.wikimodel.xhtml.impl.XhtmlHandler.characters(XhtmlHandler.java:180)
at org.xml.sax.helpers.XMLFilterImpl.characters(XMLFilterImpl.java:588)
at org.xwiki.rendering.wikimodel.xhtml.filter.XHTMLWhitespaceXMLFilter.sendCharacters(XHTMLWhitespaceXMLFilter.java:487)
at org.xwiki.rendering.wikimodel.xhtml.filter.XHTMLWhitespaceXMLFilter.sendCharacters(XHTMLWhitespaceXMLFilter.java:480)
at org.xwiki.rendering.wikimodel.xhtml.filter.XHTMLWhitespaceXMLFilter.flushContent(XHTMLWhitespaceXMLFilter.java:357)
at org.xwiki.rendering.wikimodel.xhtml.filter.XHTMLWhitespaceXMLFilter.flushContent(XHTMLWhitespaceXMLFilter.java:335)
at org.xwiki.rendering.wikimodel.xhtml.filter.XHTMLWhitespaceXMLFilter.endElement(XHTMLWhitespaceXMLFilter.java:200)
at org.xml.sax.helpers.XMLFilterImpl.endElement(XMLFilterImpl.java:570)
at org.xwiki.rendering.wikimodel.xhtml.filter.AccumulationXMLFilter.endElement(AccumulationXMLFilter.java:86)
at org.xml.sax.helpers.XMLFilterImpl.endElement(XMLFilterImpl.java:570)
at org.xwiki.rendering.wikimodel.xhtml.filter.DTDXMLFilter.endElement(DTDXMLFilter.java:86)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11NonValidatingConfiguration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11NonValidatingConfiguration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.xml.sax.helpers.XMLFilterImpl.parse(XMLFilterImpl.java:357)
at org.xwiki.rendering.wikimodel.xhtml.filter.DefaultXMLFilter.parse(DefaultXMLFilter.java:58)
at org.xml.sax.helpers.XMLFilterImpl.parse(XMLFilterImpl.java:357)
at org.xwiki.rendering.wikimodel.xhtml.filter.DefaultXMLFilter.parse(DefaultXMLFilter.java:58)
at org.xml.sax.helpers.XMLFilterImpl.parse(XMLFilterImpl.java:357)
at org.xwiki.rendering.wikimodel.xhtml.filter.DefaultXMLFilter.parse(DefaultXMLFilter.java:58)
at org.xwiki.rendering.wikimodel.xhtml.XhtmlParser.parse(XhtmlParser.java:132)
Automatic Testing in DevOps - Lorentz Workshop
55. Crash-reproducing Test Case
55
public void test0() throws Throwable {
…
SolrEntityReferenceResolver solrEntityReferenceResolver0 = new …();
EntityReferenceResolver entityReferenceResolver0 = … mock(…);
solrDocument0.put("wiki", (Object) entityType0);
Injector.inject(solrEntityReferenceResolver0, …);
Injector.validateBean(solrEntityReferenceResolver0, …);
…
// Undeclared exception!
solrEntityReferenceResolver0.resolve(solrDocument0, entityType0, objectArray0);
}
java.lang.ClassCastException: […]
at org…..SolrEntityReferenceResolver.getWikiReference(....java:93)
at org…..SolrEntityReferenceResolver.getEntityReference(….java:70)
at org…..SolrEntityReferenceResolver.resolve(….java:63)
Automatic Testing in DevOps - Lorentz Workshop
56. Crash-Guided
Genetic Algorithm
• EvoCrash
• Implemented on top of EvoSuite
• Requires
• Stack trace
• Binaries
• .jar files
• Time budget
• Set by the user
56
Initialize population
Evaluate fitness
Next generation
Selection
Crossover
Mutation
Reinsertion
[fitness == 0 or
budget exhausted]
Automatic Testing in DevOps - Lorentz Workshop
58. @Test
testDecodeQueryString() {
Map<String, String> p = newHashMap();
String uri = "something?test=value";
RestUtils.decodeQueryString
(uri, uri.indexOf(’?’) + 1,p);
assertThat(p.size(), equalTo(1));
assertThat(p.get("test"), equalTo("value"));
}
Flaky tests
Does Refactoring of Test Smells Induce Fixing Flaky Tests? Fabio
Palomba and Andy Zaidman. ICSME 2017.
Automatic Testing in DevOps - Lorentz Workshop 58
59. Complex builds
How to Design a Program Repair Bot? Insights from the Repairnator Project. S. Urli, Z. Yu, L. Seinturier, M. Monperrus. ICSE SEIP 2018.
Automatic Testing in DevOps - Lorentz Workshop 59
65. Ecosystems of
dependencies
•Only 1% of Maven
Central
•31877 artefacts
•57227
dependencies
Collected and visualized by Amine Benelallam and Cesar Soto
Automatic Testing in DevOps - Lorentz Workshop 65
66. Get Ops conditions
•Record production traffic
•Replay traffic
•Shadow traffic
Automatic Testing in DevOps - Lorentz Workshop 66
69. Crash analysis
- Reproduction
- Localization
Online experiments
- Chaos
- A/B testing
Automatic Testing in DevOps - Lorentz Workshop 69
Continuous Integration
- Unit, integration testing
- Coverage, mutation
- Test generation and repair
Test improvement
- Test refactoring
- Test amplification
- Test fixing
IDE
- Linters
- Completion
70. Automatic testing in DevOps
•Vibrant research and development topic
•The PR loop is an pportunity for research
•Incremental analyses and testing
•Developer in the loop
•Challenges
•Going in the Ops
•Build complexity
Automatic Testing in DevOps - Lorentz Workshop 70
71. Acknowledgements
•Oscar Vera-Pérez, Benjamin Danglot, César
Soto, Amine Benelalam, Nicolas Harrand,
Martin Monperrus
•https://stamp.ow2.org
Automatic Testing in DevOps - Lorentz Workshop 71