SlideShare ist ein Scribd-Unternehmen logo

ICSE 2014: Unit Test Virtualization with VMVM

•
•
J
jon_bell

Presentation of the ACM SigSoft Distinguished Paper, "Unit Test Virtualization with VMVM" given at ICSE 2014. Unit Test Virtualization significantly decreases the amount of time necessary to run unit tests.

WissenschaftTechnologieBusiness
1 von 97
Downloaden Sie, um offline zu lesen
@_jon_bell_ICSE 2014 June 5, 2014 Unit Test Virtualization with VMVM Jonathan Bell and Gail Kaiser Columbia University Forkm e on Github
@_jon_bell_ICSE 2014 June 5, 2014 Good news: We have tests! No judgement on whether they are complete or not, but we sure have a lot of them Number of tests Apache Tomcat 1,734 Closure Compiler 7,949 Commons I/O 1,022
@_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests!
@_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests! • Much work has focused on improving the situation: • Test Suite Prioritization • E.g. Wong [ISSRE ’97], Rothermel [ICSM ’99]; Elbaum [ICSE ’01]; Srivastava [ISSTA ’02] and more
@_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests! • Much work has focused on improving the situation: • Test Suite Prioritization • E.g. Wong [ISSRE ’97], Rothermel [ICSM ’99]; Elbaum [ICSE ’01]; Srivastava [ISSTA ’02] and more • Test Suite Minimization • E.g. Harrold [TOSEM ’93]; Wong [ICSE ’95]; Chen [IST ’98]; Jones [TOSEM ’03]; Tallam [PASTE ’05]; Jeffrey [TSE ’07]; Orso [ICSE ’09] Hao [ICSE ’12] and more
@_jon_bell_ICSE 2014 June 5, 2014 Testing still takes too long.
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62%
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Apache Tomcat: From 26 minutes to 18 minutes
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Integrates with JUnit, ant, and mvn on unmodied JVMs. Apache Tomcat: From 26 minutes to 18 minutes
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Integrates with JUnit, ant, and mvn on unmodied JVMs. Apache Tomcat: From 26 minutes to 18 minutes Available on GitHub
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test!
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM?
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM? Unit tests as fast as 3-5 ms
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec)
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test!Up to 4,153%, avg 618% Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec) *From our study of 20 popular FOSS apps
@_jon_bell_ICSE 2014 June 5, 2014 Do applications really use a new JVM for each test? • Checked out the 1,000 largest Java projects from Ohloh • 81% of projects with more than 1,000 tests do it • 71% of projects with more than 1 million LOC do it • Overall: 41% of all of the projects do
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests?
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests? • Dangerous: If wrong, can have false positives or false negatives (Muşlu [FSE ’11], Zhang [ISSTA ’14])
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue();
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue(); This eld is set once, when the class that owns it is initialized
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue(); This field is set once, when the class that owns it is initialized This field’s value is dependent on an external property
@_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue();
@_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue();
@_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test
@_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test But our static field is stuck! TestDontAllowEqualsInValue
@_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test TestAllowEqualsInValue
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach Unit Test Virtualization: Allow tests to leave side-effects. But efciently contain them.
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance We think in terms of object graphs
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Accessible! Objects references We think in terms of object graphs
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references We think in terms of object graphs
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references Accessible! Objects references Test Case n references We think in terms of object graphs
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references Accessible! Objects references Test Case n references We think in terms of object graphs No cross-talk No cross-talk
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class#A Static# Fields Class#B Static# Fields Static elds: owned by a class, NOT by an instance
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class#A Static# Fields Class#B Static# Fields Static elds: owned by a class, NOT by an instance These are leakage points references references
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads Writes
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads Reads Static% Fields Writes
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* Static% Fields
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception*
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception*
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* So, don’t touch them! These classes had no possible conflicts
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* So, don’t touch them! These classes had no possible conflicts Key Insight:! No need to re-initialize the entire application in order to isolate tests
@_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM
@_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit
@_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation
@_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation • No changes to JVM, no access to source code required
@_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time
@_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T
@_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T
@_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status
@_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status Release lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Release lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Release lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Release lock on T Acquire lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Acquire lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Notinitialized First new instance or static reference of T per test
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Re-initialize T’s super classes Notinitialized First new instance or static reference of T per test
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Re-initialize T’s super classes Notinitialized First new instance or static reference of T per test Re-initialize T
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process • Register a hook with test runner to tell us when a new test starts
@_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization?
@_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM?
@_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM? • RQ3: Does VMVM impact fault finding ability?
@_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches
@_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique
@_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique • Study performed on the popular Software Infrastructure Repository dataset
@_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! RQ1: VMVM vs Test Minimization Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% RQ1: VMVM vs Test Minimization Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% RQ1: VMVM vs Test Minimization Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% 49% RQ1: VMVM vs Test Minimization Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes • Range from 5,000 LoC - 5,692,450 LoC; 3 - 292 test classes; 3.5-15 years in age
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openre! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openre! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Average: 62% Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openre! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Max: 97% Average: 62% Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup
@_jon_bell_ICSE 2014 June 5, 2014 Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup • Result: Only average time per test is correlated with VMVM’s speedup (in fact, quite strongly; p < 0.0001)
@_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR)
@_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though?
@_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications
@_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications • Result: False positives occur when not isolated. VMVM shows no false positives or false negatives.
@_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases
@_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases • VMVM provides up to a 97% reduction in testing time through more efficient isolation (average 62%)
@_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases • VMVM provides up to a 97% reduction in testing time through more efficient isolation (average 62%) • VMVM does not risk a reduction in fault finding
@_jon_bell_ICSE 2014 June 5, 2014 Unit Test Virtualization with VMVM Jonathan Bell and Gail Kaiser Columbia University https://github.com/Programming-Systems-Lab/vmvm See a demo of VMVM at 2:30 today! Room MR G1-3 Forkm e on Github

Empfohlen

Efficient Dependency Detection for Safe Java Test Acceleration
Efficient Dependency Detection for Safe Java Test AccelerationEfficient Dependency Detection for Safe Java Test Acceleration
Efficient Dependency Detection for Safe Java Test Accelerationjon_bell
 
ICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards IICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards Isonal-mahajan
 
Introduction To The Research Method
Introduction To The Research MethodIntroduction To The Research Method
Introduction To The Research MethodProf (Dr.) Chamaru De Alwis
 
Australia presentation
Australia presentationAustralia presentation
Australia presentationBelĂŠn Parra
 
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at FlickrJohn Allspaw
 
Test Dependencies and the Future of Build Acceleration
Test Dependencies and the Future of Build AccelerationTest Dependencies and the Future of Build Acceleration
Test Dependencies and the Future of Build AccelerationNew York City College of Technology Computer Systems Technology Colloquium
 
How Did I Miss That Bug? Managing Cognitive Bias in Testing
How Did I Miss That Bug? Managing Cognitive Bias in TestingHow Did I Miss That Bug? Managing Cognitive Bias in Testing
How Did I Miss That Bug? Managing Cognitive Bias in TestingTechWell
 
7 essential truths of testing
7 essential truths of testing7 essential truths of testing
7 essential truths of testingbrucedgreen
 

Empfohlen

Efficient Dependency Detection for Safe Java Test Acceleration
Efficient Dependency Detection for Safe Java Test AccelerationEfficient Dependency Detection for Safe Java Test Acceleration
Efficient Dependency Detection for Safe Java Test Accelerationjon_bell
 
ICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards IICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards Isonal-mahajan
 
Introduction To The Research Method
Introduction To The Research MethodIntroduction To The Research Method
Introduction To The Research MethodProf (Dr.) Chamaru De Alwis
 
Australia presentation
Australia presentationAustralia presentation
Australia presentationBelĂŠn Parra
 
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at FlickrJohn Allspaw
 
Test Dependencies and the Future of Build Acceleration
Test Dependencies and the Future of Build AccelerationTest Dependencies and the Future of Build Acceleration
Test Dependencies and the Future of Build AccelerationNew York City College of Technology Computer Systems Technology Colloquium
 
How Did I Miss That Bug? Managing Cognitive Bias in Testing
How Did I Miss That Bug? Managing Cognitive Bias in TestingHow Did I Miss That Bug? Managing Cognitive Bias in Testing
How Did I Miss That Bug? Managing Cognitive Bias in TestingTechWell
 
7 essential truths of testing
7 essential truths of testing7 essential truths of testing
7 essential truths of testingbrucedgreen
 
DevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous deliveryDevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous deliveryDevDay Dresden
 
Whole team responsibility
Whole team responsibility Whole team responsibility
Whole team responsibility Emily Webber
 
Xtext Best Practices
Xtext Best PracticesXtext Best Practices
Xtext Best PracticesSebastian Zarnekow
 
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or CohenArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohencohen88or
 
Exploratory testing
Exploratory testingExploratory testing
Exploratory testingSandeep Kumar Sahu
 
ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD Daniel Tracy
 
2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi Reggiani2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi ReggianiEuropean Innovation Academy
 
Things Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingThings Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingTechWell
 
Move test planning before implementation
Move test planning before implementationMove test planning before implementation
Move test planning before implementationTed Cheng
 
Angular Unit Test
Angular Unit TestAngular Unit Test
Angular Unit TestMichael Haberman
 
Google, quality and you
Google, quality and youGoogle, quality and you
Google, quality and younelinger
 
End to end testing - strategies
End to end testing - strategiesEnd to end testing - strategies
End to end testing - strategiesanuvip
 
Testing smells
Testing smellsTesting smells
Testing smellsSidu Ponnappa
 
Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.especificacoes.com
 
How to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingHow to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingTechWell
 
Front end testing you won't hate
Front end testing you won't hateFront end testing you won't hate
Front end testing you won't hateWill Mitchell
 
Fundamentals of Testing 2
Fundamentals of Testing 2Fundamentals of Testing 2
Fundamentals of Testing 2Hoang Nguyen
 
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success Cloudtestr Inc.
 
Unit testing, principles
Unit testing, principlesUnit testing, principles
Unit testing, principlesRenato Primavera
 
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckDeliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckKevin Brockhoff
 
Replay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented ApplicationsReplay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented Applicationsjon_bell
 
A Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage EvolutionA Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage Evolutionjon_bell
 

Weitere ähnliche Inhalte

Ähnlich wie ICSE 2014: Unit Test Virtualization with VMVM

DevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous deliveryDevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous deliveryDevDay Dresden
 
Whole team responsibility
Whole team responsibility Whole team responsibility
Whole team responsibility Emily Webber
 
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or CohenArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohencohen88or
 
ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD Daniel Tracy
 
2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi Reggiani2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi ReggianiEuropean Innovation Academy
 
Things Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingThings Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingTechWell
 
Move test planning before implementation
Move test planning before implementationMove test planning before implementation
Move test planning before implementationTed Cheng
 
Google, quality and you
Google, quality and youGoogle, quality and you
Google, quality and younelinger
 
End to end testing - strategies
End to end testing - strategiesEnd to end testing - strategies
End to end testing - strategiesanuvip
 
Testing smells
Testing smellsTesting smells
Testing smellsSidu Ponnappa
 
Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.especificacoes.com
 
How to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingHow to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingTechWell
 
Front end testing you won't hate
Front end testing you won't hateFront end testing you won't hate
Front end testing you won't hateWill Mitchell
 
Fundamentals of Testing 2
Fundamentals of Testing 2Fundamentals of Testing 2
Fundamentals of Testing 2Hoang Nguyen
 
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success Cloudtestr Inc.
 
Unit testing, principles
Unit testing, principlesUnit testing, principles
Unit testing, principlesRenato Primavera
 
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckDeliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckKevin Brockhoff
 

Ähnlich wie ICSE 2014: Unit Test Virtualization with VMVM (20)

DevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous deliveryDevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous delivery
 
Whole team responsibility
Whole team responsibility Whole team responsibility
Whole team responsibility
 
Xtext Best Practices
Xtext Best PracticesXtext Best Practices
Xtext Best Practices
 
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or CohenArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
 
Exploratory testing
Exploratory testingExploratory testing
Exploratory testing
 
ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD
 
2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi Reggiani2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi Reggiani
 
Things Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingThings Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression Testing
 
Move test planning before implementation
Move test planning before implementationMove test planning before implementation
Move test planning before implementation
 
Angular Unit Test
Angular Unit TestAngular Unit Test
Angular Unit Test
 
Google, quality and you
Google, quality and youGoogle, quality and you
Google, quality and you
 
End to end testing - strategies
End to end testing - strategiesEnd to end testing - strategies
End to end testing - strategies
 
Testing smells
Testing smellsTesting smells
Testing smells
 
Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.
 
How to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingHow to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated Testing
 
Front end testing you won't hate
Front end testing you won't hateFront end testing you won't hate
Front end testing you won't hate
 
Fundamentals of Testing 2
Fundamentals of Testing 2Fundamentals of Testing 2
Fundamentals of Testing 2
 
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
 
Unit testing, principles
Unit testing, principlesUnit testing, principles
Unit testing, principles
 
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckDeliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
 

Mehr von jon_bell

Replay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented ApplicationsReplay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented Applicationsjon_bell
 
A Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage EvolutionA Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage Evolutionjon_bell
 
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)jon_bell
 
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMs
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMsPhosphor: Illuminating Dynamic Data Flow in Commodity JVMs
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMsjon_bell
 
Unit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing TimeUnit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing Timejon_bell
 
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...jon_bell
 
A Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
A Large-Scale, Longitudinal Study of User Profiles in World of WarcraftA Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
A Large-Scale, Longitudinal Study of User Profiles in World of Warcraftjon_bell
 

Mehr von jon_bell (7)

Replay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented ApplicationsReplay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented Applications
 
A Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage EvolutionA Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage Evolution
 
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
 
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMs
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMsPhosphor: Illuminating Dynamic Data Flow in Commodity JVMs
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMs
 
Unit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing TimeUnit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing Time
 
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
 
A Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
A Large-Scale, Longitudinal Study of User Profiles in World of WarcraftA Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
A Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
 

KĂźrzlich hochgeladen

Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...SĂŠrgio Sacani
 
Recombinant DNA technology (Immunological screening)
Recombinant DNA technology (Immunological screening)Recombinant DNA technology (Immunological screening)
Recombinant DNA technology (Immunological screening)PraveenaKalaiselvan1
 
Boyles law module in the grade 10 science
Boyles law module in the grade 10 scienceBoyles law module in the grade 10 science
Boyles law module in the grade 10 sciencefloriejanemacaya1
 
Presentation Vikram Lander by Vedansh Gupta.pptx
Presentation Vikram Lander by Vedansh Gupta.pptxPresentation Vikram Lander by Vedansh Gupta.pptx
Presentation Vikram Lander by Vedansh Gupta.pptxgindu3009
 
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptx
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptxUnlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptx
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptxanandsmhk
 
Call Girls in Mayapuri Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
Call Girls in Mayapuri Delhi 💯Call Us 🔝9953322196🔝 💯Escort.Call Girls in Mayapuri Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
Call Girls in Mayapuri Delhi 💯Call Us 🔝9953322196🔝 💯Escort.aasikanpl
 
Labelling Requirements and Label Claims for Dietary Supplements and Recommend...
Labelling Requirements and Label Claims for Dietary Supplements and Recommend...Labelling Requirements and Label Claims for Dietary Supplements and Recommend...
Labelling Requirements and Label Claims for Dietary Supplements and Recommend...Lokesh Kothari
 
A relative description on Sonoporation.pdf
A relative description on Sonoporation.pdfA relative description on Sonoporation.pdf
A relative description on Sonoporation.pdfnehabiju2046
 
Biological Classification BioHack (3).pdf
Biological Classification BioHack (3).pdfBiological Classification BioHack (3).pdf
Biological Classification BioHack (3).pdfmuntazimhurra
 
Grafana in space: Monitoring Japan's SLIM moon lander in real time
Grafana in space: Monitoring Japan's SLIM moon lander in real timeGrafana in space: Monitoring Japan's SLIM moon lander in real time
Grafana in space: Monitoring Japan's SLIM moon lander in real timeSatoshi NAKAHIRA
 
All-domain Anomaly Resolution Office U.S. Department of Defense (U) Case: “Eg...
All-domain Anomaly Resolution Office U.S. Department of Defense (U) Case: “Eg...All-domain Anomaly Resolution Office U.S. Department of Defense (U) Case: “Eg...
All-domain Anomaly Resolution Office U.S. Department of Defense (U) Case: “Eg...Sérgio Sacani
 
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCE
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCESTERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCE
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCEPRINCE C P
 
Nightside clouds and disequilibrium chemistry on the hot Jupiter WASP-43b
Nightside clouds and disequilibrium chemistry on the hot Jupiter WASP-43bNightside clouds and disequilibrium chemistry on the hot Jupiter WASP-43b
Nightside clouds and disequilibrium chemistry on the hot Jupiter WASP-43bSĂŠrgio Sacani
 
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral Analysis
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral AnalysisRaman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral Analysis
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral AnalysisDiwakar Mishra
 
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptx
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptxSOLUBLE PATTERN RECOGNITION RECEPTORS.pptx
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptxkessiyaTpeter
 
PossibleEoarcheanRecordsoftheGeomagneticFieldPreservedintheIsuaSupracrustalBe...
PossibleEoarcheanRecordsoftheGeomagneticFieldPreservedintheIsuaSupracrustalBe...PossibleEoarcheanRecordsoftheGeomagneticFieldPreservedintheIsuaSupracrustalBe...
PossibleEoarcheanRecordsoftheGeomagneticFieldPreservedintheIsuaSupracrustalBe...SĂŠrgio Sacani
 
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |aasikanpl
 
Formation of low mass protostars and their circumstellar disks
Formation of low mass protostars and their circumstellar disksFormation of low mass protostars and their circumstellar disks
Formation of low mass protostars and their circumstellar disksSĂŠrgio Sacani
 

KĂźrzlich hochgeladen (20)

Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
 
Recombinant DNA technology (Immunological screening)
Recombinant DNA technology (Immunological screening)Recombinant DNA technology (Immunological screening)
Recombinant DNA technology (Immunological screening)
 
Boyles law module in the grade 10 science
Boyles law module in the grade 10 scienceBoyles law module in the grade 10 science
Boyles law module in the grade 10 science
 
Presentation Vikram Lander by Vedansh Gupta.pptx
Presentation Vikram Lander by Vedansh Gupta.pptxPresentation Vikram Lander by Vedansh Gupta.pptx
Presentation Vikram Lander by Vedansh Gupta.pptx
 
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptx
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptxUnlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptx
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptx
 
Call Girls in Mayapuri Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
Call Girls in Mayapuri Delhi 💯Call Us 🔝9953322196🔝 💯Escort.Call Girls in Mayapuri Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
Call Girls in Mayapuri Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
 
Labelling Requirements and Label Claims for Dietary Supplements and Recommend...
Labelling Requirements and Label Claims for Dietary Supplements and Recommend...Labelling Requirements and Label Claims for Dietary Supplements and Recommend...
Labelling Requirements and Label Claims for Dietary Supplements and Recommend...
 
A relative description on Sonoporation.pdf
A relative description on Sonoporation.pdfA relative description on Sonoporation.pdf
A relative description on Sonoporation.pdf
 
9953056974 Young Call Girls In Mahavir enclave Indian Quality Escort service
9953056974 Young Call Girls In Mahavir enclave Indian Quality Escort service9953056974 Young Call Girls In Mahavir enclave Indian Quality Escort service
9953056974 Young Call Girls In Mahavir enclave Indian Quality Escort service
 
Biological Classification BioHack (3).pdf
Biological Classification BioHack (3).pdfBiological Classification BioHack (3).pdf
Biological Classification BioHack (3).pdf
 
Grafana in space: Monitoring Japan's SLIM moon lander in real time
Grafana in space: Monitoring Japan's SLIM moon lander in real timeGrafana in space: Monitoring Japan's SLIM moon lander in real time
Grafana in space: Monitoring Japan's SLIM moon lander in real time
 
All-domain Anomaly Resolution Office U.S. Department of Defense (U) Case: “Eg...
All-domain Anomaly Resolution Office U.S. Department of Defense (U) Case: “Eg...All-domain Anomaly Resolution Office U.S. Department of Defense (U) Case: “Eg...
All-domain Anomaly Resolution Office U.S. Department of Defense (U) Case: “Eg...
 
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCE
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCESTERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCE
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCE
 
CELL -Structural and Functional unit of life.pdf
CELL -Structural and Functional unit of life.pdfCELL -Structural and Functional unit of life.pdf
CELL -Structural and Functional unit of life.pdf
 
Nightside clouds and disequilibrium chemistry on the hot Jupiter WASP-43b
Nightside clouds and disequilibrium chemistry on the hot Jupiter WASP-43bNightside clouds and disequilibrium chemistry on the hot Jupiter WASP-43b
Nightside clouds and disequilibrium chemistry on the hot Jupiter WASP-43b
 
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral Analysis
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral AnalysisRaman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral Analysis
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral Analysis
 
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptx
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptxSOLUBLE PATTERN RECOGNITION RECEPTORS.pptx
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptx
 
PossibleEoarcheanRecordsoftheGeomagneticFieldPreservedintheIsuaSupracrustalBe...
PossibleEoarcheanRecordsoftheGeomagneticFieldPreservedintheIsuaSupracrustalBe...PossibleEoarcheanRecordsoftheGeomagneticFieldPreservedintheIsuaSupracrustalBe...
PossibleEoarcheanRecordsoftheGeomagneticFieldPreservedintheIsuaSupracrustalBe...
 
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |
 
Formation of low mass protostars and their circumstellar disks
Formation of low mass protostars and their circumstellar disksFormation of low mass protostars and their circumstellar disks
Formation of low mass protostars and their circumstellar disks
 

ICSE 2014: Unit Test Virtualization with VMVM

  • 1. @_jon_bell_ICSE 2014 June 5, 2014 Unit Test Virtualization with VMVM Jonathan Bell and Gail Kaiser Columbia University Forkm e on Github
  • 2. @_jon_bell_ICSE 2014 June 5, 2014 Good news: We have tests! No judgement on whether they are complete or not, but we sure have a lot of them Number of tests Apache Tomcat 1,734 Closure Compiler 7,949 Commons I/O 1,022
  • 3. @_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests!
  • 4. @_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests! • Much work has focused on improving the situation: • Test Suite Prioritization • E.g. Wong [ISSRE ’97], Rothermel [ICSM ’99]; Elbaum [ICSE ’01]; Srivastava [ISSTA ’02] and more
  • 5. @_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests! • Much work has focused on improving the situation: • Test Suite Prioritization • E.g. Wong [ISSRE ’97], Rothermel [ICSM ’99]; Elbaum [ICSE ’01]; Srivastava [ISSTA ’02] and more • Test Suite Minimization • E.g. Harrold [TOSEM ’93]; Wong [ICSE ’95]; Chen [IST ’98]; Jones [TOSEM ’03]; Tallam [PASTE ’05]; Jeffrey [TSE ’07]; Orso [ICSE ’09] Hao [ICSE ’12] and more
  • 6. @_jon_bell_ICSE 2014 June 5, 2014 Testing still takes too long.
  • 7. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization
  • 8. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62%
  • 9. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Apache Tomcat: From 26 minutes to 18 minutes
  • 10. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Integrates with JUnit, ant, and mvn on unmodied JVMs. Apache Tomcat: From 26 minutes to 18 minutes
  • 11. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Integrates with JUnit, ant, and mvn on unmodied JVMs. Apache Tomcat: From 26 minutes to 18 minutes Available on GitHub
  • 12. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite
  • 13. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite
  • 14. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test!
  • 15. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM?
  • 16. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM? Unit tests as fast as 3-5 ms
  • 17. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec)
  • 18. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test!Up to 4,153%, avg 618% Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec) *From our study of 20 popular FOSS apps
  • 19. @_jon_bell_ICSE 2014 June 5, 2014 Do applications really use a new JVM for each test? • Checked out the 1,000 largest Java projects from Ohloh • 81% of projects with more than 1,000 tests do it • 71% of projects with more than 1 million LOC do it • Overall: 41% of all of the projects do
  • 20. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent
  • 21. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests?
  • 22. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests? • Dangerous: If wrong, can have false positives or false negatives (Muşlu [FSE ’11], Zhang [ISSTA ’14])
  • 23. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue();
  • 24. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue(); This eld is set once, when the class that owns it is initialized
  • 25. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue(); This eld is set once, when the class that owns it is initialized This eld’s value is dependent on an external property
  • 26. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue();
  • 27. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue();
  • 28. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test
  • 29. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test But our static eld is stuck! TestDontAllowEqualsInValue
  • 30. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test TestAllowEqualsInValue
  • 31. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach Unit Test Virtualization: Allow tests to leave side-effects. But efciently contain them.
  • 32. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs
  • 33. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance We think in terms of object graphs
  • 34. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Accessible! Objects references We think in terms of object graphs
  • 35. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references We think in terms of object graphs
  • 36. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references Accessible! Objects references Test Case n references We think in terms of object graphs
  • 37. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references Accessible! Objects references Test Case n references We think in terms of object graphs No cross-talk No cross-talk
  • 38. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class#A Static# Fields Class#B Static# Fields Static elds: owned by a class, NOT by an instance
  • 39. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class#A Static# Fields Class#B Static# Fields Static elds: owned by a class, NOT by an instance These are leakage points references references
  • 40. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2
  • 41. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes
  • 42. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads
  • 43. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads Writes
  • 44. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads Reads Static% Fields Writes
  • 45. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* Static% Fields
  • 46. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception*
  • 47. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception*
  • 48. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* So, don’t touch them! These classes had no possible conflicts
  • 49. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* So, don’t touch them! These classes had no possible conflicts Key Insight:! No need to re-initialize the entire application in order to isolate tests
  • 50. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM
  • 51. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit
  • 52. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation
  • 53. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation • No changes to JVM, no access to source code required
  • 54. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time
  • 55. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T
  • 56. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T
  • 57. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status
  • 58. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status Release lock on T Notinitialized
  • 59. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Release lock on T Notinitialized
  • 60. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Release lock on T Notinitialized
  • 61. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Release lock on T Acquire lock on T Notinitialized
  • 62. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Acquire lock on T Notinitialized
  • 63. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Notinitialized
  • 64. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Notinitialized First new instance or static reference of T per test
  • 65. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Re-initialize T’s super classes Notinitialized First new instance or static reference of T per test
  • 66. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization Emulate exactly what happens when a class is initialized the rst time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Re-initialize T’s super classes Notinitialized First new instance or static reference of T per test Re-initialize T
  • 67. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization • Does not require any modications to the JVM and runs on commodity JVMs
  • 68. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization • Does not require any modications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class
  • 69. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization • Does not require any modications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java
  • 70. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization • Does not require any modications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process
  • 71. @_jon_bell_ICSE 2014 June 5, 2014 Efcient Reinitialization • Does not require any modications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process • Register a hook with test runner to tell us when a new test starts
  • 72. @_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization?
  • 73. @_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM?
  • 74. @_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM? • RQ3: Does VMVM impact fault nding ability?
  • 75. @_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches
  • 76. @_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault nding ability, Harrold [TOSEM ‘93]'s technique
  • 77. @_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault nding ability, Harrold [TOSEM ‘93]'s technique • Study performed on the popular Software Infrastructure Repository dataset
  • 78. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! RQ1: VMVM vs Test Minimization Larger is better
  • 79. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% RQ1: VMVM vs Test Minimization Larger is better
  • 80. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% RQ1: VMVM vs Test Minimization Larger is better
  • 81. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% 49% RQ1: VMVM vs Test Minimization Larger is better
  • 82. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes
  • 83. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes
  • 84. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes • Range from 5,000 LoC - 5,692,450 LoC; 3 - 292 test classes; 3.5-15 years in age
  • 85. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openre! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Larger is better
  • 86. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openre! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Average: 62% Larger is better
  • 87. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openre! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Max: 97% Average: 62% Larger is better
  • 88. @_jon_bell_ICSE 2014 June 5, 2014 Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup
  • 89. @_jon_bell_ICSE 2014 June 5, 2014 Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup • Result: Only average time per test is correlated with VMVM’s speedup (in fact, quite strongly; p < 0.0001)
  • 90. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault nding from seeded faults (SIR)
  • 91. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault nding from seeded faults (SIR) • Does VMVM correctly isolate tests though?
  • 92. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault nding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications
  • 93. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault nding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications • Result: False positives occur when not isolated. VMVM shows no false positives or false negatives.
  • 94. @_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases
  • 95. @_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases • VMVM provides up to a 97% reduction in testing time through more efcient isolation (average 62%)
  • 96. @_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases • VMVM provides up to a 97% reduction in testing time through more efcient isolation (average 62%) • VMVM does not risk a reduction in fault nding
  • 97. @_jon_bell_ICSE 2014 June 5, 2014 Unit Test Virtualization with VMVM Jonathan Bell and Gail Kaiser Columbia University https://github.com/Programming-Systems-Lab/vmvm See a demo of VMVM at 2:30 today! Room MR G1-3 Forkm e on Github