SlideShare ist ein Scribd-Unternehmen logo

Test Dependencies and the Future of Build Acceleration

•
•
New York City College of Technology Computer Systems Technology Colloquium
New York City College of Technology Computer Systems Technology Colloquium

With the proliferation of testing culture, many developers are facing new challenges. As projects are getting started, the focus may be on developing enough tests to maintain confidence that the code is correct. However, as developers write more and more tests, performance and repeatability become growing concerns for test suites. In our study of large open source software, we found that running tests took on average 41% of the total time needed to build each project – over 90% in those that took the longest to build. Unfortunately, typical techniques for accelerating test suites from literature (like running only a subset of tests, or running them in parallel) can’t be applied in practice safely, since tests may depend on each other. These dependencies are very hard to find and detect, posing a serious challenge to test and build acceleration. In this talk, I will present my recent research in automatically detecting and isolating these dependencies, enabling for significant, safe and sound build acceleration of up to 16x.

Technologie
1 von 73
Downloaden Sie, um offline zu lesen
Test Dependencies and the Future of Build Acceleration Jonathan Bell (@_jon_bell_) Columbia University
@_jon_bell_Future of Build Acceleration Simplified Software Lifecycle Make changes to code Build & test Commit How long is too long of a build? 1 day? 6 hours? 10 minutes?
@_jon_bell_Future of Build Acceleration Simplified Software Lifecycle • Compile sources • Generate documentation • Run tests • Package Make changes to code Build & test Commit
@_jon_bell_Future of Build Acceleration Testing Dominates Build Times 20% 38% 41% 351 projects from GitHub Testing Other Compiling
@_jon_bell_Future of Build Acceleration Testing Dominates Build Times 14% 26% 60% Projects taking > 10 minutes to build (69) Testing Other Compiling
@_jon_bell_Future of Build Acceleration Testing Dominates Build Times 2%8% 90% Projects taking > 1 hour to build (8) Testing OtherCompiling
Faster tests = Faster builds
@_jon_bell_Future of Build Acceleration JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined) For EVERY test!Up to 4,153%, avg 618% Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec) *From our study of 20 popular FOSS apps
@_jon_bell_Future of Build Acceleration Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests? • Dangerous: If wrong, can have false positives or false negatives (Muşlu [FSE ’11], Zhang [ISSTA ’14])
@_jon_bell_Future of Build Acceleration Test Independence /**  If  true,  cookie  values  are  allowed  to  contain  an  equals   character  without  being  quoted.  */   public  static  boolean  ALLOW_EQUALS_IN_VALUE  =          Boolean.valueOf(System.getProperty("org.apache.tomcat.          util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))                  .booleanValue(); This field is set once, when the class that owns it is initialized This field’s value is dependent on an external property
@_jon_bell_Future of Build Acceleration A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public  static  boolean  ALLOW_EQUALS_IN_VALUE  =  Boolean.valueOf(     System.getProperty(“org.apache.tomcat.util.http.ServerCookie.     ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test But our static field is stuck! TestAllowEqualsInValue TestDontAllowEqualsInValue
@_jon_bell_ Smarter Test Isolation for Faster Testing “Unit Test Virtualization with VMVM” [Bell and Kaiser at ICSE ’14; Distinguished Paper Award] Forkm e on Github
@_jon_bell_Future of Build Acceleration How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible Objects references Accessible Objects references Accessible Objects references Test Case n references We think in terms of object graphs No cross-talk No cross-talk
@_jon_bell_Future of Build Acceleration How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs
@_jon_bell_Future of Build Acceleration How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class  A Static   Fields Class  B Static   Fields Static fields: owned by a class, NOT by an instance These are leakage points references references
@_jon_bell_Future of Build Acceleration Isolating Side Effects Class  A Static   Fields Class  B Static   Fields Class  C Static   Fields Test 1 Test 2 Writes Reads Reads Static   Fields Writes
@_jon_bell_Future of Build Acceleration Isolating Side Effects Class  A Static   Fields Class  B Static   Fields Class  C Static   Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* Static   Fields So, don’t touch them! These classes had no possible conflicts Key Insight: No need to re-initialize the entire application in order to isolate tests
@_jon_bell_Future of Build Acceleration VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation • No changes to JVM, no access to source code required
@_jon_bell_Future of Build Acceleration Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process • Register a hook with test runner to tell us when a new test starts
@_jon_bell_Future of Build Acceleration VMVM: Unit Test Virtualization if(CookiesSupport.ALLOW_EQUALS_IN_VALUE) //... else //... if(CookiesSupport.ALLOW_EQUALS_IN_VALUE) //... else //... VMVM adds guards to reinitialize classes if(ShouldReInit(CookiesSupport.class) CookiesSupport.REINIT();
@_jon_bell_Future of Build Acceleration Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM? • RQ3: Does VMVM impact fault finding ability?
@_jon_bell_Future of Build Acceleration RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique • Study performed on the popular Software Infrastructure Repository dataset
@_jon_bell_Future of Build Acceleration 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% 49% RQ1: VMVM vs Test Minimization Larger is better
@_jon_bell_Future of Build Acceleration RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes • Range from 5,000 LoC - 5,692,450 LoC; 3 - 292 test classes; 3.5-15 years in age
@_jon_bell_Future of Build Acceleration RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Max: 97% Average: 62% Larger is better
@_jon_bell_Future of Build Acceleration Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup • Result: Only average time per test is correlated with VMVM’s speedup (in fact, quite strongly; p < 0.0001)
@_jon_bell_Future of Build Acceleration RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications • Result: False positives occur when not isolated. VMVM shows no false positives or false negatives.
@_jon_bell_ How do we make it faster? Java VMVM Unit Tests
@_jon_bell_ How do we make it faster? Java VMVM Unit Tests
@_jon_bell_Future of Build Acceleration Testing is Embarrassingly Parallel Project Raw  ,me   (minutes) 8  Worker   Speedup 24  Worker   Speedup Internal  CI 20.50 2.5x 1.8x Mule  ESB 150.92 6.4x 10.9x Jenkins 2.33 2.2x 2.3x OpenWebBeans 0.54 1.9x 2.1x Cut from 2.5 hours to 14 minutes
@_jon_bell_Future of Build Acceleration Feedback from Developers about VMVM • “It’s great! It cuts our 45 minute tests in half!” • “It’s useless! We don’t isolate our tests! Our tests take 24 hours so isolating them would make them take days!” • Remember: Although our study showed many isolate their tests, not all do!
What happens if you don’t isolate?
@_jon_bell_Future of Build Acceleration Regression Test Selection Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Gligoric et al. [ISSTA ’15], Orso et al. [FSE ’04], Harrold et al. [OOPSLA ’01] Changeset Tests not relevant to changeset: skipped
@_jon_bell_Future of Build Acceleration Test Suite Minimization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 < /> Code Hao et al. [ICSE ’12]; Orso et al. [ICSE ’09]; Jeffrey et al. [TSE ’07]; Tallam et al. [PASTE ’05]; Jones et al. [TOSEM ’03]; Harrold et al. [TOSEM ’93]; Chen et al. [IST ’98]; Wong et al. [ICSE ’95] and more Redundant tests: removed
@_jon_bell_Future of Build Acceleration Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14
@_jon_bell_Future of Build Acceleration Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14
@_jon_bell_Future of Build Acceleration Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External FactorsExternal Factors External Factors External Factors External Factors Test 1Test 2Test 3
Not sound in practice
@_jon_bell_Future of Build Acceleration Test Dependencies Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Shared File Value: A Write, Value “A” Test 4 Read Write, Value “B” Value: B Test 3 Read
@_jon_bell_Future of Build Acceleration Test Dependencies Test 1 Test 2 Test 3Test 4Test 1 Test 2 Test 3 Shared File Value: A Write, Value “A” Test 4 Write, Value “B” Read, Expect Value “A” Value: B A manifest test dependency Read
@_jon_bell_Future of Build Acceleration Test Dependencies: A Clear and Present Danger • Really exist in practice (Zhang et al. found 96, Luo et al. found 14) • Hard to specify - if we could specify, would be safe to accelerate • Can’t arbitrarily isolate (and it adds overhead!) • Existing technique to detect: combinatorially run tests [Zhang, et al ’14]
@_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Test 4Test 3
@_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 1 Test 2 Test 4 Test 3Test 1 Test 2 Test 3Test 4
@_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 2 Test 1 Test 3 Test 4Test 2 Test 1 Test 4Test 3
@_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 4 Test 2 Test 3 Test 1Test 4 Test 2 Test 1Test 3
@_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 1 Test 3 Test 2 Test 4Test 1 Test 3 Test 4Test 2
@_jon_bell_Future of Build Acceleration Brute Force Dependency Detection • Looked at feasibility on 10 large open source test suites • Exhaustive approach: > 10300 years to find all dependencies • Pairwise approach: Average 31,882 executions of the entire test suite to find (incomplete) dependencies • Problem: How do we safely accelerate test suites in the presence of unknown dependencies?
@_jon_bell_Future of Build Acceleration Manifest Test Dependencies • Definition: a data dependence between tests T1, T2 that results in the outcome of T2 changing • All manifest dependencies are data dependencies • Not all data dependencies are manifest dependencies
@_jon_bell_Future of Build Acceleration Data Dependencies Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Shared File Write, Value “A” Test 4 Read Write, Value “B” Test 3 Read Present Dependencies: Test 1 must run before 2 and 3 Test 4 must run after 2 and 3
Key Insight: Dependencies don’t need to be precise, but must be sound
@_jon_bell_Future of Build Acceleration Intuition Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test 11 Test 12 Test 13 Test 14 Test 15 Idle extra capacity
@_jon_bell_Future of Build Acceleration Intuition Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Idle extra capacity A lot of dependencies, but still a 2x speedup
@_jon_bell_Future of Build Acceleration Efficient Dependency Detection for Safe Java Test Acceleration Jonathan Bell, Gail Kaiser, Eric Melski and Mohan Dattatreya Columbia University & Electric Cloud, Inc
@_jon_bell_Future of Build Acceleration ElectricTest - Detecting Data Dependencies in Java • Tracks in-memory dependencies (JVMTI plugin) • Tracks file and network dependencies (IO-Trace agent) • Implemented entirely within the Oracle or OpenJDK JVM, no specialized drivers, etc required • Captures stack traces when dependencies occur to support debugging • Generates dependency trees to enable sound test acceleration
@_jon_bell_Future of Build Acceleration Identifying Heap Dependencies After each test, garbage collect; traverse heap to map objects back to static fields. Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield End of test 1
@_jon_bell_Future of Build Acceleration Identifying Heap Dependencies During test execution, monitor accesses to existing objects Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield W2 W2 W1 Write! Write! Read! During Test 2 Dependency!
@_jon_bell_Future of Build Acceleration Identifying External Dependencies Application under test Network Filesystem Log remote host address Log path
ElectricTest enables sound exploitation of existing test acceleration techniques
@_jon_bell_Future of Build Acceleration Safe Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test 11 Test 12 Test 13 Test 14 Test 15
@_jon_bell_Future of Build Acceleration Safe Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13Test 14 Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13Test 14
@_jon_bell_Future of Build Acceleration Safe Test Selection Test 15 Single test selected to be executed
@_jon_bell_Future of Build Acceleration Safe Test Selection Test 15Test 1 Test 2 Test 3 Single test selected to be executed with its dependencies
@_jon_bell_Future of Build Acceleration Understanding Dependencies • What should a developer do about test dependencies? • Might be intentional (e.g. cache shared state) • Might be unintentional but OK (e.g. loggers) • Might be unintentional and bad (e.g. bug)
@_jon_bell_Future of Build Acceleration Assisting Debugging Debugging information reported by the previous technique Test 3 Test 1 Depends on
@_jon_bell_Future of Build Acceleration Assisting Debugging Exception  in  thread  "main"   edu.columbia.cs.psl.testdepends.DependencyException:  Static  Field   ClassA.FieldA  member  was  previously  written  by  Test  1,  read  here.     at  edu.columbia.cs.psl.testdepends.test.Example$NestedExample.dragons(Example.java:20)     at  edu.columbia.cs.psl.testdepends.test.Example.moreMagic(Example.java:12)     at  edu.columbia.cs.psl.testdepends.test.Example.magic(Example.java:8)     at  edu.columbia.cs.psl.testdepends.test.Example.main(Example.java:15)   Really helpful Test that wrote value Stack trace shows use Value that is read
@_jon_bell_Future of Build Acceleration Evaluation • RQ1: Recall (accuracy) • RQ2: Runtime overhead • RQ3: Impact on acceleration
@_jon_bell_Future of Build Acceleration RQ1: Recall Dependencies Detected ElectricTest Shared Ground Truth ElectricTest Resource Locations Project Writers Readers App Library Joda 2 15 121 39 12 XMLSecurity 4 3 103 3 15 Crystal 18 15 39 4 19 Synoptic 1 10 117 3 14
@_jon_bell_Future of Build Acceleration RQ2: Overhead • Selected 10 projects with > 10 minutes of tests • Also included projects studied by Zhang et al, averaging < 10 seconds of testing • Previous exhaustive approach slowdown: >10300X • Previous heuristic approach slowdown: 31,882X • ElectricTest slowdown: 36X (885X faster than previous approach)
@_jon_bell_Future of Build Acceleration 0X 1,000X 2,000X 3,000X 4,000X 5,000X 6,000X 7,000X 8,000X 9,000X 10,000X mongo%java%driver- tachyon- spring%data%mongodb- xml-security- ne8y- je8y.project- crystal- crunch- camel- :tan- synop:c- hazelcast- mule- joda%:me- ElectricTest Slowdown Pairwise Slowdown *418,000X RQ2: Overhead On average, ElectricTest is 885X faster than running all tests pairwise Slowdown relative to a single test suite execution (lower is better)
@_jon_bell_Future of Build Acceleration 0X 50X 100X 150X 200X 250X 300X mongo%java%driver- tachyon- spring%data%mongodb- xml-security- ne8y- je8y.project- crystal- crunch- camel- :tan- synop:c- hazelcast- mule- joda%:me- RQ2: Overhead Average 36X A lot of fast running tests: Runtime dominated by pauses between tests (gc) Slowdown relative to a single test suite execution (lower is better)
@_jon_bell_Future of Build Acceleration 0X 5X 10X 15X 20X 25X 30X camel& crunch& hazelcast& je/y.project& mongo5java5driver& mule& ne/y& spring5data5mongodb& tachyon& :tan& Safe Unsafe Speedup (higher is better) RQ3: Impact on Acceleration Average (Unsafe) 19x Average (Safe) 7x
@_jon_bell_Future of Build Acceleration
Test Dependencies and the Future of Build Acceleration Jonathan Bell Columbia University jbell@cs.columbia.edu http://jonbell.net/

Empfohlen

Arquillian : An introduction
Arquillian : An introductionArquillian : An introduction
Arquillian : An introductionVineet Reynolds
 
Testing JSF with Arquillian and Selenium
Testing JSF with Arquillian and SeleniumTesting JSF with Arquillian and Selenium
Testing JSF with Arquillian and SeleniumLukáš Fryč
 
Real Java EE Testing with Arquillian and ShrinkWrap
Real Java EE Testing with Arquillian and ShrinkWrapReal Java EE Testing with Arquillian and ShrinkWrap
Real Java EE Testing with Arquillian and ShrinkWrapDan Allen
 
DevOps - Boldly Go for Distro
DevOps - Boldly Go for DistroDevOps - Boldly Go for Distro
DevOps - Boldly Go for DistroPaul Boos
 
Need forbuildspeed agile2012
Need forbuildspeed agile2012Need forbuildspeed agile2012
Need forbuildspeed agile2012drewz lin
 
Test it! Unit, mocking and in-container Meet Arquillian!
Test it! Unit, mocking and in-container Meet Arquillian!Test it! Unit, mocking and in-container Meet Arquillian!
Test it! Unit, mocking and in-container Meet Arquillian!Ivan Ivanov
 
Mutation Testing DevoxxUK 2021
Mutation Testing DevoxxUK 2021Mutation Testing DevoxxUK 2021
Mutation Testing DevoxxUK 2021Paco van Beckhoven
 
Arquillian - Integration Testing Made Easy
Arquillian - Integration Testing Made EasyArquillian - Integration Testing Made Easy
Arquillian - Integration Testing Made EasyJBUG London
 

Empfohlen

Arquillian : An introduction
Arquillian : An introductionArquillian : An introduction
Arquillian : An introductionVineet Reynolds
 
Testing JSF with Arquillian and Selenium
Testing JSF with Arquillian and SeleniumTesting JSF with Arquillian and Selenium
Testing JSF with Arquillian and SeleniumLukáš Fryč
 
Real Java EE Testing with Arquillian and ShrinkWrap
Real Java EE Testing with Arquillian and ShrinkWrapReal Java EE Testing with Arquillian and ShrinkWrap
Real Java EE Testing with Arquillian and ShrinkWrapDan Allen
 
DevOps - Boldly Go for Distro
DevOps - Boldly Go for DistroDevOps - Boldly Go for Distro
DevOps - Boldly Go for DistroPaul Boos
 
Need forbuildspeed agile2012
Need forbuildspeed agile2012Need forbuildspeed agile2012
Need forbuildspeed agile2012drewz lin
 
Test it! Unit, mocking and in-container Meet Arquillian!
Test it! Unit, mocking and in-container Meet Arquillian!Test it! Unit, mocking and in-container Meet Arquillian!
Test it! Unit, mocking and in-container Meet Arquillian!Ivan Ivanov
 
Mutation Testing DevoxxUK 2021
Mutation Testing DevoxxUK 2021Mutation Testing DevoxxUK 2021
Mutation Testing DevoxxUK 2021Paco van Beckhoven
 
Arquillian - Integration Testing Made Easy
Arquillian - Integration Testing Made EasyArquillian - Integration Testing Made Easy
Arquillian - Integration Testing Made EasyJBUG London
 
Integration Testing in Python
Integration Testing in PythonIntegration Testing in Python
Integration Testing in PythonPanoptic Development, Inc.
 
Arquillian: Helping web developers and QA get along
Arquillian: Helping web developers and QA get alongArquillian: Helping web developers and QA get along
Arquillian: Helping web developers and QA get alongLukáš Fryč
 
Escaping Automated Test Hell - One Year Later
Escaping Automated Test Hell - One Year LaterEscaping Automated Test Hell - One Year Later
Escaping Automated Test Hell - One Year LaterWojciech Seliga
 
Continuous Integration, Continuous Quality, Continuous Delivery
Continuous Integration, Continuous Quality, Continuous DeliveryContinuous Integration, Continuous Quality, Continuous Delivery
Continuous Integration, Continuous Quality, Continuous DeliveryJohn Ferguson Smart Limited
 
Writing useful automated tests for the single page applications you build
Writing useful automated tests for the single page applications you buildWriting useful automated tests for the single page applications you build
Writing useful automated tests for the single page applications you buildAndrei Sebastian Cîmpean
 
Testing 101
Testing 101Testing 101
Testing 101Noam Barkai
 
Jenkins Evolutions - JEEConf 2012
Jenkins Evolutions - JEEConf 2012Jenkins Evolutions - JEEConf 2012
Jenkins Evolutions - JEEConf 2012Anton Arhipov
 
Windmill Testing certification
Windmill Testing certificationWindmill Testing certification
Windmill Testing certificationVskills
 
Implementing Quality on a Java Project
Implementing Quality on a Java ProjectImplementing Quality on a Java Project
Implementing Quality on a Java ProjectVincent Massol
 
Open stack qa and tempest
Open stack qa and tempestOpen stack qa and tempest
Open stack qa and tempestKamesh Pemmaraju
 
Testing: ¿what, how, why?
Testing: ¿what, how, why?Testing: ¿what, how, why?
Testing: ¿what, how, why?David Rodenas
 
Integration testing - A&BP CC
Integration testing - A&BP CCIntegration testing - A&BP CC
Integration testing - A&BP CCJWORKS powered by Ordina
 
Automated Application Tests For Lotus Notes Uklug 2009
Automated Application Tests For Lotus Notes Uklug 2009Automated Application Tests For Lotus Notes Uklug 2009
Automated Application Tests For Lotus Notes Uklug 2009maxistar
 
Testing in FrontEnd World by Nikita Galkin
Testing in FrontEnd World by Nikita GalkinTesting in FrontEnd World by Nikita Galkin
Testing in FrontEnd World by Nikita GalkinSigma Software
 
Android Test Driven Development
Android Test Driven DevelopmentAndroid Test Driven Development
Android Test Driven DevelopmentArif Huda
 
Understanding Layers of Testing
Understanding Layers of TestingUnderstanding Layers of Testing
Understanding Layers of TestingChristopher Rex
 
Arquillian: Effective tests from the client to the server
Arquillian: Effective tests from the client to the serverArquillian: Effective tests from the client to the server
Arquillian: Effective tests from the client to the serverLukáš Fryč
 
Jest: Frontend Testing leicht gemacht @EnterJS2018
Jest: Frontend Testing leicht gemacht @EnterJS2018Jest: Frontend Testing leicht gemacht @EnterJS2018
Jest: Frontend Testing leicht gemacht @EnterJS2018Holger Grosse-Plankermann
 
An introduction to unit testing
An introduction to unit testingAn introduction to unit testing
An introduction to unit testingAdam Stephensen
 
Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Babul Mirdha
 
Android Apps the Right Way
Android Apps the Right WayAndroid Apps the Right Way
Android Apps the Right WayNew York City College of Technology Computer Systems Technology Colloquium
 
Static Analysis and Verification of C Programs
Static Analysis and Verification of C ProgramsStatic Analysis and Verification of C Programs
Static Analysis and Verification of C ProgramsNew York City College of Technology Computer Systems Technology Colloquium
 

Weitere ähnliche Inhalte

Was ist angesagt?

Arquillian: Helping web developers and QA get along
Arquillian: Helping web developers and QA get alongArquillian: Helping web developers and QA get along
Arquillian: Helping web developers and QA get alongLukáš Fryč
 
Escaping Automated Test Hell - One Year Later
Escaping Automated Test Hell - One Year LaterEscaping Automated Test Hell - One Year Later
Escaping Automated Test Hell - One Year LaterWojciech Seliga
 
Continuous Integration, Continuous Quality, Continuous Delivery
Continuous Integration, Continuous Quality, Continuous DeliveryContinuous Integration, Continuous Quality, Continuous Delivery
Continuous Integration, Continuous Quality, Continuous DeliveryJohn Ferguson Smart Limited
 
Writing useful automated tests for the single page applications you build
Writing useful automated tests for the single page applications you buildWriting useful automated tests for the single page applications you build
Writing useful automated tests for the single page applications you buildAndrei Sebastian Cîmpean
 
Jenkins Evolutions - JEEConf 2012
Jenkins Evolutions - JEEConf 2012Jenkins Evolutions - JEEConf 2012
Jenkins Evolutions - JEEConf 2012Anton Arhipov
 
Windmill Testing certification
Windmill Testing certificationWindmill Testing certification
Windmill Testing certificationVskills
 
Implementing Quality on a Java Project
Implementing Quality on a Java ProjectImplementing Quality on a Java Project
Implementing Quality on a Java ProjectVincent Massol
 
Testing: ¿what, how, why?
Testing: ¿what, how, why?Testing: ¿what, how, why?
Testing: ¿what, how, why?David Rodenas
 
Automated Application Tests For Lotus Notes Uklug 2009
Automated Application Tests For Lotus Notes Uklug 2009Automated Application Tests For Lotus Notes Uklug 2009
Automated Application Tests For Lotus Notes Uklug 2009maxistar
 
Testing in FrontEnd World by Nikita Galkin
Testing in FrontEnd World by Nikita GalkinTesting in FrontEnd World by Nikita Galkin
Testing in FrontEnd World by Nikita GalkinSigma Software
 
Android Test Driven Development
Android Test Driven DevelopmentAndroid Test Driven Development
Android Test Driven DevelopmentArif Huda
 
Understanding Layers of Testing
Understanding Layers of TestingUnderstanding Layers of Testing
Understanding Layers of TestingChristopher Rex
 
Arquillian: Effective tests from the client to the server
Arquillian: Effective tests from the client to the serverArquillian: Effective tests from the client to the server
Arquillian: Effective tests from the client to the serverLukáš Fryč
 
Jest: Frontend Testing leicht gemacht @EnterJS2018
Jest: Frontend Testing leicht gemacht @EnterJS2018Jest: Frontend Testing leicht gemacht @EnterJS2018
Jest: Frontend Testing leicht gemacht @EnterJS2018Holger Grosse-Plankermann
 
An introduction to unit testing
An introduction to unit testingAn introduction to unit testing
An introduction to unit testingAdam Stephensen
 
Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Babul Mirdha
 

Was ist angesagt? (20)

Integration Testing in Python
Integration Testing in PythonIntegration Testing in Python
Integration Testing in Python
 
Arquillian: Helping web developers and QA get along
Arquillian: Helping web developers and QA get alongArquillian: Helping web developers and QA get along
Arquillian: Helping web developers and QA get along
 
Escaping Automated Test Hell - One Year Later
Escaping Automated Test Hell - One Year LaterEscaping Automated Test Hell - One Year Later
Escaping Automated Test Hell - One Year Later
 
Continuous Integration, Continuous Quality, Continuous Delivery
Continuous Integration, Continuous Quality, Continuous DeliveryContinuous Integration, Continuous Quality, Continuous Delivery
Continuous Integration, Continuous Quality, Continuous Delivery
 
Writing useful automated tests for the single page applications you build
Writing useful automated tests for the single page applications you buildWriting useful automated tests for the single page applications you build
Writing useful automated tests for the single page applications you build
 
Testing 101
Testing 101Testing 101
Testing 101
 
Jenkins Evolutions - JEEConf 2012
Jenkins Evolutions - JEEConf 2012Jenkins Evolutions - JEEConf 2012
Jenkins Evolutions - JEEConf 2012
 
Windmill Testing certification
Windmill Testing certificationWindmill Testing certification
Windmill Testing certification
 
Implementing Quality on a Java Project
Implementing Quality on a Java ProjectImplementing Quality on a Java Project
Implementing Quality on a Java Project
 
Open stack qa and tempest
Open stack qa and tempestOpen stack qa and tempest
Open stack qa and tempest
 
Testing: ¿what, how, why?
Testing: ¿what, how, why?Testing: ¿what, how, why?
Testing: ¿what, how, why?
 
Integration testing - A&BP CC
Integration testing - A&BP CCIntegration testing - A&BP CC
Integration testing - A&BP CC
 
Automated Application Tests For Lotus Notes Uklug 2009
Automated Application Tests For Lotus Notes Uklug 2009Automated Application Tests For Lotus Notes Uklug 2009
Automated Application Tests For Lotus Notes Uklug 2009
 
Testing in FrontEnd World by Nikita Galkin
Testing in FrontEnd World by Nikita GalkinTesting in FrontEnd World by Nikita Galkin
Testing in FrontEnd World by Nikita Galkin
 
Android Test Driven Development
Android Test Driven DevelopmentAndroid Test Driven Development
Android Test Driven Development
 
Understanding Layers of Testing
Understanding Layers of TestingUnderstanding Layers of Testing
Understanding Layers of Testing
 
Arquillian: Effective tests from the client to the server
Arquillian: Effective tests from the client to the serverArquillian: Effective tests from the client to the server
Arquillian: Effective tests from the client to the server
 
Jest: Frontend Testing leicht gemacht @EnterJS2018
Jest: Frontend Testing leicht gemacht @EnterJS2018Jest: Frontend Testing leicht gemacht @EnterJS2018
Jest: Frontend Testing leicht gemacht @EnterJS2018
 
An introduction to unit testing
An introduction to unit testingAn introduction to unit testing
An introduction to unit testing
 
Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)
 

Andere mochten auch

Rising Underwriting Capacity Driving The Asia Pacific Insurance Sector: Ken R...
Rising Underwriting Capacity Driving The Asia Pacific Insurance Sector: Ken R...Rising Underwriting Capacity Driving The Asia Pacific Insurance Sector: Ken R...
Rising Underwriting Capacity Driving The Asia Pacific Insurance Sector: Ken R...Ankur Gupta
 
Week 1 documentary
Week 1 documentaryWeek 1 documentary
Week 1 documentaryromanyangel4
 
Planeacion actividad semana # 1 emprendimiento
Planeacion actividad semana # 1 emprendimientoPlaneacion actividad semana # 1 emprendimiento
Planeacion actividad semana # 1 emprendimientomariazuluagaorozco
 
Презентация 1.14 - Гостевые эко-дома для учёных в Исландии
Презентация 1.14 - Гостевые эко-дома для учёных в ИсландииПрезентация 1.14 - Гостевые эко-дома для учёных в Исландии
Презентация 1.14 - Гостевые эко-дома для учёных в ИсландииПавел Ефимов
 
Intro to Programming for Communicators - Hacks/Hackers ATX
Intro to Programming for Communicators - Hacks/Hackers ATXIntro to Programming for Communicators - Hacks/Hackers ATX
Intro to Programming for Communicators - Hacks/Hackers ATXCindy Royal
 
Презентация 2.5.7.1 - "Технологические решения" (ИОС7) - текстовая часть
Презентация 2.5.7.1 - "Технологические решения" (ИОС7) - текстовая частьПрезентация 2.5.7.1 - "Технологические решения" (ИОС7) - текстовая часть
Презентация 2.5.7.1 - "Технологические решения" (ИОС7) - текстовая частьПавел Ефимов
 
Decalogo del Profesor de Hoy
Decalogo del Profesor de Hoy Decalogo del Profesor de Hoy
Decalogo del Profesor de Hoy Roberto Abarca
 
求人掲載マニュアル
求人掲載マニュアル求人掲載マニュアル
求人掲載マニュアルtomokiyo-n
 
Interaction Design L03 - Color
Interaction Design L03 - ColorInteraction Design L03 - Color
Interaction Design L03 - ColorMohammad Shaker
 
정품드래곤발기제♤w5.ow.to ☆톡 w2015♤ 드래곤발기제 종류, 드래곤발기제 구매,드래곤발기제 약효, 드래곤발기제 정품구입
정품드래곤발기제♤w5.ow.to ☆톡 w2015♤ 드래곤발기제 종류, 드래곤발기제 구매,드래곤발기제 약효, 드래곤발기제 정품구입정품드래곤발기제♤w5.ow.to ☆톡 w2015♤ 드래곤발기제 종류, 드래곤발기제 구매,드래곤발기제 약효, 드래곤발기제 정품구입
정품드래곤발기제♤w5.ow.to ☆톡 w2015♤ 드래곤발기제 종류, 드래곤발기제 구매,드래곤발기제 약효, 드래곤발기제 정품구입fdsx cvxx
 
Deja Las Lagrimas Rodar
Deja Las Lagrimas RodarDeja Las Lagrimas Rodar
Deja Las Lagrimas Rodarnimiaazucena
 

Andere mochten auch (20)

Android Apps the Right Way
Android Apps the Right WayAndroid Apps the Right Way
Android Apps the Right Way
 
Static Analysis and Verification of C Programs
Static Analysis and Verification of C ProgramsStatic Analysis and Verification of C Programs
Static Analysis and Verification of C Programs
 
Ontology-based Classification and Faceted Search Interface for APIs
Ontology-based Classification and Faceted Search Interface for APIsOntology-based Classification and Faceted Search Interface for APIs
Ontology-based Classification and Faceted Search Interface for APIs
 
2016 j
2016 j2016 j
2016 j
 
Rising Underwriting Capacity Driving The Asia Pacific Insurance Sector: Ken R...
Rising Underwriting Capacity Driving The Asia Pacific Insurance Sector: Ken R...Rising Underwriting Capacity Driving The Asia Pacific Insurance Sector: Ken R...
Rising Underwriting Capacity Driving The Asia Pacific Insurance Sector: Ken R...
 
Aliens
AliensAliens
Aliens
 
Week 1 documentary
Week 1 documentaryWeek 1 documentary
Week 1 documentary
 
Valuing Health at the End of Life
Valuing Health at the End of LifeValuing Health at the End of Life
Valuing Health at the End of Life
 
Planeacion actividad semana # 1 emprendimiento
Planeacion actividad semana # 1 emprendimientoPlaneacion actividad semana # 1 emprendimiento
Planeacion actividad semana # 1 emprendimiento
 
Презентация 1.14 - Гостевые эко-дома для учёных в Исландии
Презентация 1.14 - Гостевые эко-дома для учёных в ИсландииПрезентация 1.14 - Гостевые эко-дома для учёных в Исландии
Презентация 1.14 - Гостевые эко-дома для учёных в Исландии
 
How to Study New Ones
How to Study New OnesHow to Study New Ones
How to Study New Ones
 
Intro to Programming for Communicators - Hacks/Hackers ATX
Intro to Programming for Communicators - Hacks/Hackers ATXIntro to Programming for Communicators - Hacks/Hackers ATX
Intro to Programming for Communicators - Hacks/Hackers ATX
 
Презентация 2.5.7.1 - "Технологические решения" (ИОС7) - текстовая часть
Презентация 2.5.7.1 - "Технологические решения" (ИОС7) - текстовая частьПрезентация 2.5.7.1 - "Технологические решения" (ИОС7) - текстовая часть
Презентация 2.5.7.1 - "Технологические решения" (ИОС7) - текстовая часть
 
Decalogo del Profesor de Hoy
Decalogo del Profesor de Hoy Decalogo del Profesor de Hoy
Decalogo del Profesor de Hoy
 
求人掲載マニュアル
求人掲載マニュアル求人掲載マニュアル
求人掲載マニュアル
 
Interaction Design L03 - Color
Interaction Design L03 - ColorInteraction Design L03 - Color
Interaction Design L03 - Color
 
294 2015 infortunio escavatore
294 2015 infortunio escavatore294 2015 infortunio escavatore
294 2015 infortunio escavatore
 
정품드래곤발기제♤w5.ow.to ☆톡 w2015♤ 드래곤발기제 종류, 드래곤발기제 구매,드래곤발기제 약효, 드래곤발기제 정품구입
정품드래곤발기제♤w5.ow.to ☆톡 w2015♤ 드래곤발기제 종류, 드래곤발기제 구매,드래곤발기제 약효, 드래곤발기제 정품구입정품드래곤발기제♤w5.ow.to ☆톡 w2015♤ 드래곤발기제 종류, 드래곤발기제 구매,드래곤발기제 약효, 드래곤발기제 정품구입
정품드래곤발기제♤w5.ow.to ☆톡 w2015♤ 드래곤발기제 종류, 드래곤발기제 구매,드래곤발기제 약효, 드래곤발기제 정품구입
 
Deja Las Lagrimas Rodar
Deja Las Lagrimas RodarDeja Las Lagrimas Rodar
Deja Las Lagrimas Rodar
 
Company Profile - SWD
Company Profile - SWDCompany Profile - SWD
Company Profile - SWD
 

Ähnlich wie Test Dependencies and the Future of Build Acceleration

Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckDeliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckKevin Brockhoff
 
谷歌 Scott-lessons learned in testability
谷歌 Scott-lessons learned in testability谷歌 Scott-lessons learned in testability
谷歌 Scott-lessons learned in testabilitydrewz lin
 
We Are All Testers Now: The Testing Pyramid and Front-End Development
We Are All Testers Now: The Testing Pyramid and Front-End DevelopmentWe Are All Testers Now: The Testing Pyramid and Front-End Development
We Are All Testers Now: The Testing Pyramid and Front-End DevelopmentAll Things Open
 
Unit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing TimeUnit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing Timejon_bell
 
Unit testing with Spock Framework
Unit testing with Spock FrameworkUnit testing with Spock Framework
Unit testing with Spock FrameworkEugene Dvorkin
 
Enabling Agile Testing Through Continuous Integration Agile2009
Enabling Agile Testing Through Continuous Integration Agile2009Enabling Agile Testing Through Continuous Integration Agile2009
Enabling Agile Testing Through Continuous Integration Agile2009sstolberg
 
Into The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applicationsInto The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applicationsOrtus Solutions, Corp
 
Qt test framework
Qt test frameworkQt test framework
Qt test frameworkICS
 
JUnit 5 - The Next Generation
JUnit 5 - The Next GenerationJUnit 5 - The Next Generation
JUnit 5 - The Next GenerationKostadin Golev
 
New types of tests for Java projects
New types of tests for Java projectsNew types of tests for Java projects
New types of tests for Java projectsVincent Massol
 
Lightweight Java EE with MicroProfile
Lightweight Java EE with MicroProfileLightweight Java EE with MicroProfile
Lightweight Java EE with MicroProfileJosh Juneau
 
Release Automation: Better Quality, Faster Deployment, Amazing ROI
Release Automation: Better Quality, Faster Deployment, Amazing ROIRelease Automation: Better Quality, Faster Deployment, Amazing ROI
Release Automation: Better Quality, Faster Deployment, Amazing ROITechWell
 
Google test training
Google test trainingGoogle test training
Google test trainingThierry Gayet
 

Ähnlich wie Test Dependencies and the Future of Build Acceleration (20)

Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckDeliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
 
Advanced Java Testing
Advanced Java TestingAdvanced Java Testing
Advanced Java Testing
 
Testing w-mocks
Testing w-mocksTesting w-mocks
Testing w-mocks
 
Testacular
TestacularTestacular
Testacular
 
谷歌 Scott-lessons learned in testability
谷歌 Scott-lessons learned in testability谷歌 Scott-lessons learned in testability
谷歌 Scott-lessons learned in testability
 
We Are All Testers Now: The Testing Pyramid and Front-End Development
We Are All Testers Now: The Testing Pyramid and Front-End DevelopmentWe Are All Testers Now: The Testing Pyramid and Front-End Development
We Are All Testers Now: The Testing Pyramid and Front-End Development
 
Unit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing TimeUnit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing Time
 
Testing Angular
Testing AngularTesting Angular
Testing Angular
 
Introduction to JUnit
Introduction to JUnitIntroduction to JUnit
Introduction to JUnit
 
Unit testing with Spock Framework
Unit testing with Spock FrameworkUnit testing with Spock Framework
Unit testing with Spock Framework
 
Enabling Agile Testing Through Continuous Integration Agile2009
Enabling Agile Testing Through Continuous Integration Agile2009Enabling Agile Testing Through Continuous Integration Agile2009
Enabling Agile Testing Through Continuous Integration Agile2009
 
Into The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applicationsInto The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applications
 
Qt test framework
Qt test frameworkQt test framework
Qt test framework
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 
JUnit 5 - The Next Generation
JUnit 5 - The Next GenerationJUnit 5 - The Next Generation
JUnit 5 - The Next Generation
 
New types of tests for Java projects
New types of tests for Java projectsNew types of tests for Java projects
New types of tests for Java projects
 
Junit_.pptx
Junit_.pptxJunit_.pptx
Junit_.pptx
 
Lightweight Java EE with MicroProfile
Lightweight Java EE with MicroProfileLightweight Java EE with MicroProfile
Lightweight Java EE with MicroProfile
 
Release Automation: Better Quality, Faster Deployment, Amazing ROI
Release Automation: Better Quality, Faster Deployment, Amazing ROIRelease Automation: Better Quality, Faster Deployment, Amazing ROI
Release Automation: Better Quality, Faster Deployment, Amazing ROI
 
Google test training
Google test trainingGoogle test training
Google test training
 

Mehr von New York City College of Technology Computer Systems Technology Colloquium

Mehr von New York City College of Technology Computer Systems Technology Colloquium (9)

Towards Improving Interface Modularity in Legacy Java Software Through Automa...
Towards Improving Interface Modularity in Legacy Java Software Through Automa...Towards Improving Interface Modularity in Legacy Java Software Through Automa...
Towards Improving Interface Modularity in Legacy Java Software Through Automa...
 
Data-driven, Interactive Scientific Articles in a Collaborative Environment w...
Data-driven, Interactive Scientific Articles in a Collaborative Environment w...Data-driven, Interactive Scientific Articles in a Collaborative Environment w...
Data-driven, Interactive Scientific Articles in a Collaborative Environment w...
 
Cloud Technology: Virtualization
Cloud Technology: VirtualizationCloud Technology: Virtualization
Cloud Technology: Virtualization
 
Google BigTable
Google BigTableGoogle BigTable
Google BigTable
 
Pharmacology Powered by Computational Analysis: Predicting Cardiotoxicity of ...
Pharmacology Powered by Computational Analysis: Predicting Cardiotoxicity of ...Pharmacology Powered by Computational Analysis: Predicting Cardiotoxicity of ...
Pharmacology Powered by Computational Analysis: Predicting Cardiotoxicity of ...
 
How We Use Functional Programming to Find the Bad Guys
How We Use Functional Programming to Find the Bad GuysHow We Use Functional Programming to Find the Bad Guys
How We Use Functional Programming to Find the Bad Guys
 
Big Data Challenges and Solutions
Big Data Challenges and SolutionsBig Data Challenges and Solutions
Big Data Challenges and Solutions
 
Introduction to new features in java 8
Introduction to new features in java 8Introduction to new features in java 8
Introduction to new features in java 8
 
More than Words: Advancing Prosodic Analysis
More than Words: Advancing Prosodic AnalysisMore than Words: Advancing Prosodic Analysis
More than Words: Advancing Prosodic Analysis
 

Kürzlich hochgeladen

Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessPixlogix Infotech
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 

Kürzlich hochgeladen (20)

Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 

Test Dependencies and the Future of Build Acceleration

  • 1. Test Dependencies and the Future of Build Acceleration Jonathan Bell (@_jon_bell_) Columbia University
  • 2. @_jon_bell_Future of Build Acceleration Simplified Software Lifecycle Make changes to code Build & test Commit How long is too long of a build? 1 day? 6 hours? 10 minutes?
  • 3. @_jon_bell_Future of Build Acceleration Simplified Software Lifecycle • Compile sources • Generate documentation • Run tests • Package Make changes to code Build & test Commit
  • 4. @_jon_bell_Future of Build Acceleration Testing Dominates Build Times 20% 38% 41% 351 projects from GitHub Testing Other Compiling
  • 5. @_jon_bell_Future of Build Acceleration Testing Dominates Build Times 14% 26% 60% Projects taking > 10 minutes to build (69) Testing Other Compiling
  • 6. @_jon_bell_Future of Build Acceleration Testing Dominates Build Times 2%8% 90% Projects taking > 1 hour to build (8) Testing OtherCompiling
  • 7. Faster tests = Faster builds
  • 8. @_jon_bell_Future of Build Acceleration JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined) For EVERY test!Up to 4,153%, avg 618% Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec) *From our study of 20 popular FOSS apps
  • 9. @_jon_bell_Future of Build Acceleration Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests? • Dangerous: If wrong, can have false positives or false negatives (Muşlu [FSE ’11], Zhang [ISSTA ’14])
  • 10. @_jon_bell_Future of Build Acceleration Test Independence /**  If  true,  cookie  values  are  allowed  to  contain  an  equals   character  without  being  quoted.  */   public  static  boolean  ALLOW_EQUALS_IN_VALUE  =          Boolean.valueOf(System.getProperty("org.apache.tomcat.          util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))                  .booleanValue(); This field is set once, when the class that owns it is initialized This field’s value is dependent on an external property
  • 11. @_jon_bell_Future of Build Acceleration A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public  static  boolean  ALLOW_EQUALS_IN_VALUE  =  Boolean.valueOf(     System.getProperty(“org.apache.tomcat.util.http.ServerCookie.     ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test But our static field is stuck! TestAllowEqualsInValue TestDontAllowEqualsInValue
  • 12. @_jon_bell_ Smarter Test Isolation for Faster Testing “Unit Test Virtualization with VMVM” [Bell and Kaiser at ICSE ’14; Distinguished Paper Award] Forkm e on Github
  • 13. @_jon_bell_Future of Build Acceleration How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible Objects references Accessible Objects references Accessible Objects references Test Case n references We think in terms of object graphs No cross-talk No cross-talk
  • 14. @_jon_bell_Future of Build Acceleration How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs
  • 15. @_jon_bell_Future of Build Acceleration How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class  A Static   Fields Class  B Static   Fields Static fields: owned by a class, NOT by an instance These are leakage points references references
  • 16. @_jon_bell_Future of Build Acceleration Isolating Side Effects Class  A Static   Fields Class  B Static   Fields Class  C Static   Fields Test 1 Test 2 Writes Reads Reads Static   Fields Writes
  • 17. @_jon_bell_Future of Build Acceleration Isolating Side Effects Class  A Static   Fields Class  B Static   Fields Class  C Static   Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* Static   Fields So, don’t touch them! These classes had no possible conflicts Key Insight: No need to re-initialize the entire application in order to isolate tests
  • 18. @_jon_bell_Future of Build Acceleration VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation • No changes to JVM, no access to source code required
  • 19. @_jon_bell_Future of Build Acceleration Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process • Register a hook with test runner to tell us when a new test starts
  • 20. @_jon_bell_Future of Build Acceleration VMVM: Unit Test Virtualization if(CookiesSupport.ALLOW_EQUALS_IN_VALUE) //... else //... if(CookiesSupport.ALLOW_EQUALS_IN_VALUE) //... else //... VMVM adds guards to reinitialize classes if(ShouldReInit(CookiesSupport.class) CookiesSupport.REINIT();
  • 21. @_jon_bell_Future of Build Acceleration Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM? • RQ3: Does VMVM impact fault finding ability?
  • 22. @_jon_bell_Future of Build Acceleration RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique • Study performed on the popular Software Infrastructure Repository dataset
  • 23. @_jon_bell_Future of Build Acceleration 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% 49% RQ1: VMVM vs Test Minimization Larger is better
  • 24. @_jon_bell_Future of Build Acceleration RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes • Range from 5,000 LoC - 5,692,450 LoC; 3 - 292 test classes; 3.5-15 years in age
  • 25. @_jon_bell_Future of Build Acceleration RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Max: 97% Average: 62% Larger is better
  • 26. @_jon_bell_Future of Build Acceleration Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup • Result: Only average time per test is correlated with VMVM’s speedup (in fact, quite strongly; p < 0.0001)
  • 27. @_jon_bell_Future of Build Acceleration RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications • Result: False positives occur when not isolated. VMVM shows no false positives or false negatives.
  • 28. @_jon_bell_ How do we make it faster? Java VMVM Unit Tests
  • 29. @_jon_bell_ How do we make it faster? Java VMVM Unit Tests
  • 30. @_jon_bell_Future of Build Acceleration Testing is Embarrassingly Parallel Project Raw  ,me   (minutes) 8  Worker   Speedup 24  Worker   Speedup Internal  CI 20.50 2.5x 1.8x Mule  ESB 150.92 6.4x 10.9x Jenkins 2.33 2.2x 2.3x OpenWebBeans 0.54 1.9x 2.1x Cut from 2.5 hours to 14 minutes
  • 31. @_jon_bell_Future of Build Acceleration Feedback from Developers about VMVM • “It’s great! It cuts our 45 minute tests in half!” • “It’s useless! We don’t isolate our tests! Our tests take 24 hours so isolating them would make them take days!” • Remember: Although our study showed many isolate their tests, not all do!
  • 32. What happens if you don’t isolate?
  • 33. @_jon_bell_Future of Build Acceleration Regression Test Selection Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Gligoric et al. [ISSTA ’15], Orso et al. [FSE ’04], Harrold et al. [OOPSLA ’01] Changeset Tests not relevant to changeset: skipped
  • 34. @_jon_bell_Future of Build Acceleration Test Suite Minimization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 < /> Code Hao et al. [ICSE ’12]; Orso et al. [ICSE ’09]; Jeffrey et al. [TSE ’07]; Tallam et al. [PASTE ’05]; Jones et al. [TOSEM ’03]; Harrold et al. [TOSEM ’93]; Chen et al. [IST ’98]; Wong et al. [ICSE ’95] and more Redundant tests: removed
  • 35. @_jon_bell_Future of Build Acceleration Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14
  • 36. @_jon_bell_Future of Build Acceleration Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14
  • 37. @_jon_bell_Future of Build Acceleration Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External FactorsExternal Factors External Factors External Factors External Factors Test 1Test 2Test 3
  • 38. Not sound in practice
  • 39. @_jon_bell_Future of Build Acceleration Test Dependencies Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Shared File Value: A Write, Value “A” Test 4 Read Write, Value “B” Value: B Test 3 Read
  • 40. @_jon_bell_Future of Build Acceleration Test Dependencies Test 1 Test 2 Test 3Test 4Test 1 Test 2 Test 3 Shared File Value: A Write, Value “A” Test 4 Write, Value “B” Read, Expect Value “A” Value: B A manifest test dependency Read
  • 41. @_jon_bell_Future of Build Acceleration Test Dependencies: A Clear and Present Danger • Really exist in practice (Zhang et al. found 96, Luo et al. found 14) • Hard to specify - if we could specify, would be safe to accelerate • Can’t arbitrarily isolate (and it adds overhead!) • Existing technique to detect: combinatorially run tests [Zhang, et al ’14]
  • 42. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Test 4Test 3
  • 43. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 1 Test 2 Test 4 Test 3Test 1 Test 2 Test 3Test 4
  • 44. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 2 Test 1 Test 3 Test 4Test 2 Test 1 Test 4Test 3
  • 45. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 4 Test 2 Test 3 Test 1Test 4 Test 2 Test 1Test 3
  • 46. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 1 Test 3 Test 2 Test 4Test 1 Test 3 Test 4Test 2
  • 47. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection • Looked at feasibility on 10 large open source test suites • Exhaustive approach: > 10300 years to find all dependencies • Pairwise approach: Average 31,882 executions of the entire test suite to find (incomplete) dependencies • Problem: How do we safely accelerate test suites in the presence of unknown dependencies?
  • 48. @_jon_bell_Future of Build Acceleration Manifest Test Dependencies • Definition: a data dependence between tests T1, T2 that results in the outcome of T2 changing • All manifest dependencies are data dependencies • Not all data dependencies are manifest dependencies
  • 49. @_jon_bell_Future of Build Acceleration Data Dependencies Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Shared File Write, Value “A” Test 4 Read Write, Value “B” Test 3 Read Present Dependencies: Test 1 must run before 2 and 3 Test 4 must run after 2 and 3
  • 50. Key Insight: Dependencies don’t need to be precise, but must be sound
  • 51. @_jon_bell_Future of Build Acceleration Intuition Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test 11 Test 12 Test 13 Test 14 Test 15 Idle extra capacity
  • 52. @_jon_bell_Future of Build Acceleration Intuition Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Idle extra capacity A lot of dependencies, but still a 2x speedup
  • 53. @_jon_bell_Future of Build Acceleration Efficient Dependency Detection for Safe Java Test Acceleration Jonathan Bell, Gail Kaiser, Eric Melski and Mohan Dattatreya Columbia University & Electric Cloud, Inc
  • 54. @_jon_bell_Future of Build Acceleration ElectricTest - Detecting Data Dependencies in Java • Tracks in-memory dependencies (JVMTI plugin) • Tracks file and network dependencies (IO-Trace agent) • Implemented entirely within the Oracle or OpenJDK JVM, no specialized drivers, etc required • Captures stack traces when dependencies occur to support debugging • Generates dependency trees to enable sound test acceleration
  • 55. @_jon_bell_Future of Build Acceleration Identifying Heap Dependencies After each test, garbage collect; traverse heap to map objects back to static fields. Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield End of test 1
  • 56. @_jon_bell_Future of Build Acceleration Identifying Heap Dependencies During test execution, monitor accesses to existing objects Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield W2 W2 W1 Write! Write! Read! During Test 2 Dependency!
  • 57. @_jon_bell_Future of Build Acceleration Identifying External Dependencies Application under test Network Filesystem Log remote host address Log path
  • 58. ElectricTest enables sound exploitation of existing test acceleration techniques
  • 59. @_jon_bell_Future of Build Acceleration Safe Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test 11 Test 12 Test 13 Test 14 Test 15
  • 60. @_jon_bell_Future of Build Acceleration Safe Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13Test 14 Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13Test 14
  • 61. @_jon_bell_Future of Build Acceleration Safe Test Selection Test 15 Single test selected to be executed
  • 62. @_jon_bell_Future of Build Acceleration Safe Test Selection Test 15Test 1 Test 2 Test 3 Single test selected to be executed with its dependencies
  • 63. @_jon_bell_Future of Build Acceleration Understanding Dependencies • What should a developer do about test dependencies? • Might be intentional (e.g. cache shared state) • Might be unintentional but OK (e.g. loggers) • Might be unintentional and bad (e.g. bug)
  • 64. @_jon_bell_Future of Build Acceleration Assisting Debugging Debugging information reported by the previous technique Test 3 Test 1 Depends on
  • 65. @_jon_bell_Future of Build Acceleration Assisting Debugging Exception  in  thread  "main"   edu.columbia.cs.psl.testdepends.DependencyException:  Static  Field   ClassA.FieldA  member  was  previously  written  by  Test  1,  read  here.     at  edu.columbia.cs.psl.testdepends.test.Example$NestedExample.dragons(Example.java:20)     at  edu.columbia.cs.psl.testdepends.test.Example.moreMagic(Example.java:12)     at  edu.columbia.cs.psl.testdepends.test.Example.magic(Example.java:8)     at  edu.columbia.cs.psl.testdepends.test.Example.main(Example.java:15)   Really helpful Test that wrote value Stack trace shows use Value that is read
  • 66. @_jon_bell_Future of Build Acceleration Evaluation • RQ1: Recall (accuracy) • RQ2: Runtime overhead • RQ3: Impact on acceleration
  • 67. @_jon_bell_Future of Build Acceleration RQ1: Recall Dependencies Detected ElectricTest Shared Ground Truth ElectricTest Resource Locations Project Writers Readers App Library Joda 2 15 121 39 12 XMLSecurity 4 3 103 3 15 Crystal 18 15 39 4 19 Synoptic 1 10 117 3 14
  • 68. @_jon_bell_Future of Build Acceleration RQ2: Overhead • Selected 10 projects with > 10 minutes of tests • Also included projects studied by Zhang et al, averaging < 10 seconds of testing • Previous exhaustive approach slowdown: >10300X • Previous heuristic approach slowdown: 31,882X • ElectricTest slowdown: 36X (885X faster than previous approach)
  • 69. @_jon_bell_Future of Build Acceleration 0X 1,000X 2,000X 3,000X 4,000X 5,000X 6,000X 7,000X 8,000X 9,000X 10,000X mongo%java%driver- tachyon- spring%data%mongodb- xml-security- ne8y- je8y.project- crystal- crunch- camel- :tan- synop:c- hazelcast- mule- joda%:me- ElectricTest Slowdown Pairwise Slowdown *418,000X RQ2: Overhead On average, ElectricTest is 885X faster than running all tests pairwise Slowdown relative to a single test suite execution (lower is better)
  • 70. @_jon_bell_Future of Build Acceleration 0X 50X 100X 150X 200X 250X 300X mongo%java%driver- tachyon- spring%data%mongodb- xml-security- ne8y- je8y.project- crystal- crunch- camel- :tan- synop:c- hazelcast- mule- joda%:me- RQ2: Overhead Average 36X A lot of fast running tests: Runtime dominated by pauses between tests (gc) Slowdown relative to a single test suite execution (lower is better)
  • 71. @_jon_bell_Future of Build Acceleration 0X 5X 10X 15X 20X 25X 30X camel& crunch& hazelcast& je/y.project& mongo5java5driver& mule& ne/y& spring5data5mongodb& tachyon& :tan& Safe Unsafe Speedup (higher is better) RQ3: Impact on Acceleration Average (Unsafe) 19x Average (Safe) 7x
  • 73. Test Dependencies and the Future of Build Acceleration Jonathan Bell Columbia University jbell@cs.columbia.edu http://jonbell.net/