SlideShare ist ein Scribd-Unternehmen logo

ICSE 2014: Unit Test Virtualization with VMVM

•
•
J
jon_bell

Presentation of the ACM SigSoft Distinguished Paper, "Unit Test Virtualization with VMVM" given at ICSE 2014. Unit Test Virtualization significantly decreases the amount of time necessary to run unit tests.

WissenschaftTechnologieBusiness
1 von 97
Downloaden Sie, um offline zu lesen
@_jon_bell_ICSE 2014 June 5, 2014 Unit Test Virtualization with VMVM Jonathan Bell and Gail Kaiser Columbia University Forkm e on Github
@_jon_bell_ICSE 2014 June 5, 2014 Good news: We have tests! No judgement on whether they are complete or not, but we sure have a lot of them Number of tests Apache Tomcat 1,734 Closure Compiler 7,949 Commons I/O 1,022
@_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests!
@_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests! • Much work has focused on improving the situation: • Test Suite Prioritization • E.g. Wong [ISSRE ’97], Rothermel [ICSM ’99]; Elbaum [ICSE ’01]; Srivastava [ISSTA ’02] and more
@_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests! • Much work has focused on improving the situation: • Test Suite Prioritization • E.g. Wong [ISSRE ’97], Rothermel [ICSM ’99]; Elbaum [ICSE ’01]; Srivastava [ISSTA ’02] and more • Test Suite Minimization • E.g. Harrold [TOSEM ’93]; Wong [ICSE ’95]; Chen [IST ’98]; Jones [TOSEM ’03]; Tallam [PASTE ’05]; Jeffrey [TSE ’07]; Orso [ICSE ’09] Hao [ICSE ’12] and more
@_jon_bell_ICSE 2014 June 5, 2014 Testing still takes too long.
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62%
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Apache Tomcat: From 26 minutes to 18 minutes
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Integrates with JUnit, ant, and mvn on unmodified JVMs. Apache Tomcat: From 26 minutes to 18 minutes
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Integrates with JUnit, ant, and mvn on unmodified JVMs. Apache Tomcat: From 26 minutes to 18 minutes Available on GitHub
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test!
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM?
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM? Unit tests as fast as 3-5 ms
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec)
@_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test!Up to 4,153%, avg 618% Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec) *From our study of 20 popular FOSS apps
@_jon_bell_ICSE 2014 June 5, 2014 Do applications really use a new JVM for each test? • Checked out the 1,000 largest Java projects from Ohloh • 81% of projects with more than 1,000 tests do it • 71% of projects with more than 1 million LOC do it • Overall: 41% of all of the projects do
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests?
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests? • Dangerous: If wrong, can have false positives or false negatives (Muşlu [FSE ’11], Zhang [ISSTA ’14])
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue();
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue(); This field is set once, when the class that owns it is initialized
@_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue(); This field is set once, when the class that owns it is initialized This field’s value is dependent on an external property
@_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue();
@_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue();
@_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test
@_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test But our static field is stuck! TestDontAllowEqualsInValue
@_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test TestAllowEqualsInValue
@_jon_bell_ICSE 2014 June 5, 2014 Our Approach Unit Test Virtualization: Allow tests to leave side-effects. But efficiently contain them.
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance We think in terms of object graphs
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Accessible! Objects references We think in terms of object graphs
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references We think in terms of object graphs
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references Accessible! Objects references Test Case n references We think in terms of object graphs
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references Accessible! Objects references Test Case n references We think in terms of object graphs No cross-talk No cross-talk
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class#A Static# Fields Class#B Static# Fields Static fields: owned by a class, NOT by an instance
@_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class#A Static# Fields Class#B Static# Fields Static fields: owned by a class, NOT by an instance These are leakage points references references
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads Writes
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads Reads Static% Fields Writes
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* Static% Fields
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception*
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception*
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* So, don’t touch them! These classes had no possible conflicts
@_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* So, don’t touch them! These classes had no possible conflicts Key Insight:! No need to re-initialize the entire application in order to isolate tests
@_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM
@_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit
@_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation
@_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation • No changes to JVM, no access to source code required
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Release lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Release lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Release lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Release lock on T Acquire lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Acquire lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Notinitialized
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Notinitialized First new instance or static reference of T per test
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Re-initialize T’s super classes Notinitialized First new instance or static reference of T per test
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Re-initialize T’s super classes Notinitialized First new instance or static reference of T per test Re-initialize T
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process
@_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process • Register a hook with test runner to tell us when a new test starts
@_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization?
@_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM?
@_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM? • RQ3: Does VMVM impact fault finding ability?
@_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches
@_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique
@_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique • Study performed on the popular Software Infrastructure Repository dataset
@_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! RQ1: VMVM vs Test Minimization Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% RQ1: VMVM vs Test Minimization Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% RQ1: VMVM vs Test Minimization Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% 49% RQ1: VMVM vs Test Minimization Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes • Range from 5,000 LoC - 5,692,450 LoC; 3 - 292 test classes; 3.5-15 years in age
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Average: 62% Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Max: 97% Average: 62% Larger is better
@_jon_bell_ICSE 2014 June 5, 2014 Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup
@_jon_bell_ICSE 2014 June 5, 2014 Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup • Result: Only average time per test is correlated with VMVM’s speedup (in fact, quite strongly; p < 0.0001)
@_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR)
@_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though?
@_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications
@_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications • Result: False positives occur when not isolated. VMVM shows no false positives or false negatives.
@_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases
@_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases • VMVM provides up to a 97% reduction in testing time through more efficient isolation (average 62%)
@_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases • VMVM provides up to a 97% reduction in testing time through more efficient isolation (average 62%) • VMVM does not risk a reduction in fault finding
@_jon_bell_ICSE 2014 June 5, 2014 Unit Test Virtualization with VMVM Jonathan Bell and Gail Kaiser Columbia University https://github.com/Programming-Systems-Lab/vmvm See a demo of VMVM at 2:30 today! Room MR G1-3 Forkm e on Github

Empfohlen

Efficient Dependency Detection for Safe Java Test Acceleration
Efficient Dependency Detection for Safe Java Test AccelerationEfficient Dependency Detection for Safe Java Test Acceleration
Efficient Dependency Detection for Safe Java Test Accelerationjon_bell
 
ICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards IICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards Isonal-mahajan
 
Introduction To The Research Method
Introduction To The Research MethodIntroduction To The Research Method
Introduction To The Research MethodProf (Dr.) Chamaru De Alwis
 
Australia presentation
Australia presentationAustralia presentation
Australia presentationBelén Parra
 
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at FlickrJohn Allspaw
 
Test Dependencies and the Future of Build Acceleration
Test Dependencies and the Future of Build AccelerationTest Dependencies and the Future of Build Acceleration
Test Dependencies and the Future of Build AccelerationNew York City College of Technology Computer Systems Technology Colloquium
 
How Did I Miss That Bug? Managing Cognitive Bias in Testing
How Did I Miss That Bug? Managing Cognitive Bias in TestingHow Did I Miss That Bug? Managing Cognitive Bias in Testing
How Did I Miss That Bug? Managing Cognitive Bias in TestingTechWell
 
7 essential truths of testing
7 essential truths of testing7 essential truths of testing
7 essential truths of testingbrucedgreen
 

Empfohlen

Efficient Dependency Detection for Safe Java Test Acceleration
Efficient Dependency Detection for Safe Java Test AccelerationEfficient Dependency Detection for Safe Java Test Acceleration
Efficient Dependency Detection for Safe Java Test Accelerationjon_bell
 
ICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards IICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards Isonal-mahajan
 
Introduction To The Research Method
Introduction To The Research MethodIntroduction To The Research Method
Introduction To The Research MethodProf (Dr.) Chamaru De Alwis
 
Australia presentation
Australia presentationAustralia presentation
Australia presentationBelén Parra
 
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at FlickrJohn Allspaw
 
Test Dependencies and the Future of Build Acceleration
Test Dependencies and the Future of Build AccelerationTest Dependencies and the Future of Build Acceleration
Test Dependencies and the Future of Build AccelerationNew York City College of Technology Computer Systems Technology Colloquium
 
How Did I Miss That Bug? Managing Cognitive Bias in Testing
How Did I Miss That Bug? Managing Cognitive Bias in TestingHow Did I Miss That Bug? Managing Cognitive Bias in Testing
How Did I Miss That Bug? Managing Cognitive Bias in TestingTechWell
 
7 essential truths of testing
7 essential truths of testing7 essential truths of testing
7 essential truths of testingbrucedgreen
 
DevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous deliveryDevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous deliveryDevDay Dresden
 
Whole team responsibility
Whole team responsibility Whole team responsibility
Whole team responsibility Emily Webber
 
Xtext Best Practices
Xtext Best PracticesXtext Best Practices
Xtext Best PracticesSebastian Zarnekow
 
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or CohenArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohencohen88or
 
Exploratory testing
Exploratory testingExploratory testing
Exploratory testingSandeep Kumar Sahu
 
ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD Daniel Tracy
 
2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi Reggiani2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi ReggianiEuropean Innovation Academy
 
Things Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingThings Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingTechWell
 
Move test planning before implementation
Move test planning before implementationMove test planning before implementation
Move test planning before implementationTed Cheng
 
Angular Unit Test
Angular Unit TestAngular Unit Test
Angular Unit TestMichael Haberman
 
Google, quality and you
Google, quality and youGoogle, quality and you
Google, quality and younelinger
 
End to end testing - strategies
End to end testing - strategiesEnd to end testing - strategies
End to end testing - strategiesanuvip
 
Testing smells
Testing smellsTesting smells
Testing smellsSidu Ponnappa
 
Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.especificacoes.com
 
How to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingHow to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingTechWell
 
Front end testing you won't hate
Front end testing you won't hateFront end testing you won't hate
Front end testing you won't hateWill Mitchell
 
Fundamentals of Testing 2
Fundamentals of Testing 2Fundamentals of Testing 2
Fundamentals of Testing 2Hoang Nguyen
 
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success Cloudtestr Inc.
 
Unit testing, principles
Unit testing, principlesUnit testing, principles
Unit testing, principlesRenato Primavera
 
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckDeliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckKevin Brockhoff
 
Replay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented ApplicationsReplay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented Applicationsjon_bell
 
A Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage EvolutionA Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage Evolutionjon_bell
 

Weitere ähnliche Inhalte

Ähnlich wie ICSE 2014: Unit Test Virtualization with VMVM

DevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous deliveryDevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous deliveryDevDay Dresden
 
Whole team responsibility
Whole team responsibility Whole team responsibility
Whole team responsibility Emily Webber
 
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or CohenArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohencohen88or
 
ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD Daniel Tracy
 
2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi Reggiani2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi ReggianiEuropean Innovation Academy
 
Things Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingThings Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingTechWell
 
Move test planning before implementation
Move test planning before implementationMove test planning before implementation
Move test planning before implementationTed Cheng
 
Google, quality and you
Google, quality and youGoogle, quality and you
Google, quality and younelinger
 
End to end testing - strategies
End to end testing - strategiesEnd to end testing - strategies
End to end testing - strategiesanuvip
 
Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.especificacoes.com
 
How to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingHow to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingTechWell
 
Front end testing you won't hate
Front end testing you won't hateFront end testing you won't hate
Front end testing you won't hateWill Mitchell
 
Fundamentals of Testing 2
Fundamentals of Testing 2Fundamentals of Testing 2
Fundamentals of Testing 2Hoang Nguyen
 
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success Cloudtestr Inc.
 
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckDeliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckKevin Brockhoff
 

Ähnlich wie ICSE 2014: Unit Test Virtualization with VMVM (20)

DevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous deliveryDevDay 2016: Dave Farley - Acceptance testing for continuous delivery
DevDay 2016: Dave Farley - Acceptance testing for continuous delivery
 
Whole team responsibility
Whole team responsibility Whole team responsibility
Whole team responsibility
 
Xtext Best Practices
Xtext Best PracticesXtext Best Practices
Xtext Best Practices
 
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or CohenArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
ArcSight & RSA ECAT Integration - By We-Ankor & Or Cohen
 
Exploratory testing
Exploratory testingExploratory testing
Exploratory testing
 
ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD ChefConf2014 - Chef TDD
ChefConf2014 - Chef TDD
 
2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi Reggiani2014 Micro Experiments with Google Tools by Luigi Reggiani
2014 Micro Experiments with Google Tools by Luigi Reggiani
 
Things Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingThings Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression Testing
 
Move test planning before implementation
Move test planning before implementationMove test planning before implementation
Move test planning before implementation
 
Angular Unit Test
Angular Unit TestAngular Unit Test
Angular Unit Test
 
Google, quality and you
Google, quality and youGoogle, quality and you
Google, quality and you
 
End to end testing - strategies
End to end testing - strategiesEnd to end testing - strategies
End to end testing - strategies
 
Testing smells
Testing smellsTesting smells
Testing smells
 
Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.Keynote - DevOps and the bottom line.
Keynote - DevOps and the bottom line.
 
How to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingHow to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated Testing
 
Front end testing you won't hate
Front end testing you won't hateFront end testing you won't hate
Front end testing you won't hate
 
Fundamentals of Testing 2
Fundamentals of Testing 2Fundamentals of Testing 2
Fundamentals of Testing 2
 
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
Cloudtestr Webinar - 5 Ways to Maximize Test Automation Success
 
Unit testing, principles
Unit testing, principlesUnit testing, principles
Unit testing, principles
 
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't SuckDeliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
 

Mehr von jon_bell

Replay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented ApplicationsReplay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented Applicationsjon_bell
 
A Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage EvolutionA Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage Evolutionjon_bell
 
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)jon_bell
 
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMs
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMsPhosphor: Illuminating Dynamic Data Flow in Commodity JVMs
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMsjon_bell
 
Unit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing TimeUnit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing Timejon_bell
 
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...jon_bell
 
A Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
A Large-Scale, Longitudinal Study of User Profiles in World of WarcraftA Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
A Large-Scale, Longitudinal Study of User Profiles in World of Warcraftjon_bell
 

Mehr von jon_bell (7)

Replay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented ApplicationsReplay without Recording of Production Bugs for Service Oriented Applications
Replay without Recording of Production Bugs for Service Oriented Applications
 
A Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage EvolutionA Large-Scale Study of Test Coverage Evolution
A Large-Scale Study of Test Coverage Evolution
 
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
 
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMs
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMsPhosphor: Illuminating Dynamic Data Flow in Commodity JVMs
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMs
 
Unit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing TimeUnit Test Virtualization: Optimizing Testing Time
Unit Test Virtualization: Optimizing Testing Time
 
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
 
A Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
A Large-Scale, Longitudinal Study of User Profiles in World of WarcraftA Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
A Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
 

Kürzlich hochgeladen

(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)
(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)
(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)riyaescorts54
 
Q4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptxQ4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptxtuking87
 
User Guide: Capricorn FLX™ Weather Station
User Guide: Capricorn FLX™ Weather StationUser Guide: Capricorn FLX™ Weather Station
User Guide: Capricorn FLX™ Weather StationColumbia Weather Systems
 
final waves properties grade 7 - third quarter
final waves properties grade 7 - third quarterfinal waves properties grade 7 - third quarter
final waves properties grade 7 - third quarterHanHyoKim
 
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...D. B. S. College Kanpur
 
Servosystem Theory / Cybernetic Theory by Petrovic
Servosystem Theory / Cybernetic Theory by PetrovicServosystem Theory / Cybernetic Theory by Petrovic
Servosystem Theory / Cybernetic Theory by PetrovicAditi Jain
 
Organic farming with special reference to vermiculture
Organic farming with special reference to vermicultureOrganic farming with special reference to vermiculture
Organic farming with special reference to vermicultureTakeleZike1
 
Pests of jatropha_Bionomics_identification_Dr.UPR.pdf
Pests of jatropha_Bionomics_identification_Dr.UPR.pdfPests of jatropha_Bionomics_identification_Dr.UPR.pdf
Pests of jatropha_Bionomics_identification_Dr.UPR.pdfPirithiRaju
 
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptx
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptxECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptx
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptxmaryFF1
 
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2AuEnriquezLontok
 
User Guide: Orion™ Weather Station (Columbia Weather Systems)
User Guide: Orion™ Weather Station (Columbia Weather Systems)User Guide: Orion™ Weather Station (Columbia Weather Systems)
User Guide: Orion™ Weather Station (Columbia Weather Systems)Columbia Weather Systems
 
Four Spheres of the Earth Presentation.ppt
Four Spheres of the Earth Presentation.pptFour Spheres of the Earth Presentation.ppt
Four Spheres of the Earth Presentation.pptJoemSTuliba
 
GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024Jene van der Heide
 
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...Universidade Federal de Sergipe - UFS
 
办理麦克马斯特大学毕业证成绩单|购买加拿大文凭证书
办理麦克马斯特大学毕业证成绩单|购买加拿大文凭证书办理麦克马斯特大学毕业证成绩单|购买加拿大文凭证书
办理麦克马斯特大学毕业证成绩单|购买加拿大文凭证书zdzoqco
 
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...Universidade Federal de Sergipe - UFS
 
Pests of Bengal gram_Identification_Dr.UPR.pdf
Pests of Bengal gram_Identification_Dr.UPR.pdfPests of Bengal gram_Identification_Dr.UPR.pdf
Pests of Bengal gram_Identification_Dr.UPR.pdfPirithiRaju
 

Kürzlich hochgeladen (20)

(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)
(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)
(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)
 
Q4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptxQ4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptx
 
User Guide: Capricorn FLX™ Weather Station
User Guide: Capricorn FLX™ Weather StationUser Guide: Capricorn FLX™ Weather Station
User Guide: Capricorn FLX™ Weather Station
 
final waves properties grade 7 - third quarter
final waves properties grade 7 - third quarterfinal waves properties grade 7 - third quarter
final waves properties grade 7 - third quarter
 
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...
 
AZOTOBACTER AS BIOFERILIZER.PPTX
AZOTOBACTER AS BIOFERILIZER.PPTXAZOTOBACTER AS BIOFERILIZER.PPTX
AZOTOBACTER AS BIOFERILIZER.PPTX
 
Servosystem Theory / Cybernetic Theory by Petrovic
Servosystem Theory / Cybernetic Theory by PetrovicServosystem Theory / Cybernetic Theory by Petrovic
Servosystem Theory / Cybernetic Theory by Petrovic
 
Organic farming with special reference to vermiculture
Organic farming with special reference to vermicultureOrganic farming with special reference to vermiculture
Organic farming with special reference to vermiculture
 
Pests of jatropha_Bionomics_identification_Dr.UPR.pdf
Pests of jatropha_Bionomics_identification_Dr.UPR.pdfPests of jatropha_Bionomics_identification_Dr.UPR.pdf
Pests of jatropha_Bionomics_identification_Dr.UPR.pdf
 
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptx
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptxECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptx
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptx
 
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2
 
User Guide: Orion™ Weather Station (Columbia Weather Systems)
User Guide: Orion™ Weather Station (Columbia Weather Systems)User Guide: Orion™ Weather Station (Columbia Weather Systems)
User Guide: Orion™ Weather Station (Columbia Weather Systems)
 
Four Spheres of the Earth Presentation.ppt
Four Spheres of the Earth Presentation.pptFour Spheres of the Earth Presentation.ppt
Four Spheres of the Earth Presentation.ppt
 
GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024
 
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
 
办理麦克马斯特大学毕业证成绩单|购买加拿大文凭证书
办理麦克马斯特大学毕业证成绩单|购买加拿大文凭证书办理麦克马斯特大学毕业证成绩单|购买加拿大文凭证书
办理麦克马斯特大学毕业证成绩单|购买加拿大文凭证书
 
PLASMODIUM. PPTX
PLASMODIUM. PPTXPLASMODIUM. PPTX
PLASMODIUM. PPTX
 
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
 
Interferons.pptx.
Interferons.pptx.Interferons.pptx.
Interferons.pptx.
 
Pests of Bengal gram_Identification_Dr.UPR.pdf
Pests of Bengal gram_Identification_Dr.UPR.pdfPests of Bengal gram_Identification_Dr.UPR.pdf
Pests of Bengal gram_Identification_Dr.UPR.pdf
 

ICSE 2014: Unit Test Virtualization with VMVM

  • 1. @_jon_bell_ICSE 2014 June 5, 2014 Unit Test Virtualization with VMVM Jonathan Bell and Gail Kaiser Columbia University Forkm e on Github
  • 2. @_jon_bell_ICSE 2014 June 5, 2014 Good news: We have tests! No judgement on whether they are complete or not, but we sure have a lot of them Number of tests Apache Tomcat 1,734 Closure Compiler 7,949 Commons I/O 1,022
  • 3. @_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests!
  • 4. @_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests! • Much work has focused on improving the situation: • Test Suite Prioritization • E.g. Wong [ISSRE ’97], Rothermel [ICSM ’99]; Elbaum [ICSE ’01]; Srivastava [ISSTA ’02] and more
  • 5. @_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests! • Much work has focused on improving the situation: • Test Suite Prioritization • E.g. Wong [ISSRE ’97], Rothermel [ICSM ’99]; Elbaum [ICSE ’01]; Srivastava [ISSTA ’02] and more • Test Suite Minimization • E.g. Harrold [TOSEM ’93]; Wong [ICSE ’95]; Chen [IST ’98]; Jones [TOSEM ’03]; Tallam [PASTE ’05]; Jeffrey [TSE ’07]; Orso [ICSE ’09] Hao [ICSE ’12] and more
  • 6. @_jon_bell_ICSE 2014 June 5, 2014 Testing still takes too long.
  • 7. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization
  • 8. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62%
  • 9. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Apache Tomcat: From 26 minutes to 18 minutes
  • 10. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Integrates with JUnit, ant, and mvn on unmodified JVMs. Apache Tomcat: From 26 minutes to 18 minutes
  • 11. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Integrates with JUnit, ant, and mvn on unmodified JVMs. Apache Tomcat: From 26 minutes to 18 minutes Available on GitHub
  • 12. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite
  • 13. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite
  • 14. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test!
  • 15. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM?
  • 16. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM? Unit tests as fast as 3-5 ms
  • 17. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec)
  • 18. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test!Up to 4,153%, avg 618% Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec) *From our study of 20 popular FOSS apps
  • 19. @_jon_bell_ICSE 2014 June 5, 2014 Do applications really use a new JVM for each test? • Checked out the 1,000 largest Java projects from Ohloh • 81% of projects with more than 1,000 tests do it • 71% of projects with more than 1 million LOC do it • Overall: 41% of all of the projects do
  • 20. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent
  • 21. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests?
  • 22. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests? • Dangerous: If wrong, can have false positives or false negatives (Muşlu [FSE ’11], Zhang [ISSTA ’14])
  • 23. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue();
  • 24. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue(); This field is set once, when the class that owns it is initialized
  • 25. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue(); This field is set once, when the class that owns it is initialized This field’s value is dependent on an external property
  • 26. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue();
  • 27. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue();
  • 28. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test
  • 29. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test But our static field is stuck! TestDontAllowEqualsInValue
  • 30. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test TestAllowEqualsInValue
  • 31. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach Unit Test Virtualization: Allow tests to leave side-effects. But efficiently contain them.
  • 32. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs
  • 33. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance We think in terms of object graphs
  • 34. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Accessible! Objects references We think in terms of object graphs
  • 35. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references We think in terms of object graphs
  • 36. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references Accessible! Objects references Test Case n references We think in terms of object graphs
  • 37. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references Accessible! Objects references Test Case n references We think in terms of object graphs No cross-talk No cross-talk
  • 38. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class#A Static# Fields Class#B Static# Fields Static fields: owned by a class, NOT by an instance
  • 39. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class#A Static# Fields Class#B Static# Fields Static fields: owned by a class, NOT by an instance These are leakage points references references
  • 40. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2
  • 41. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes
  • 42. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads
  • 43. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads Writes
  • 44. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads Reads Static% Fields Writes
  • 45. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* Static% Fields
  • 46. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception*
  • 47. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception*
  • 48. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* So, don’t touch them! These classes had no possible conflicts
  • 49. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* So, don’t touch them! These classes had no possible conflicts Key Insight:! No need to re-initialize the entire application in order to isolate tests
  • 50. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM
  • 51. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit
  • 52. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation
  • 53. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation • No changes to JVM, no access to source code required
  • 54. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time
  • 55. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T
  • 56. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T
  • 57. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status
  • 58. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Release lock on T Notinitialized
  • 59. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Release lock on T Notinitialized
  • 60. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Release lock on T Notinitialized
  • 61. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Release lock on T Acquire lock on T Notinitialized
  • 62. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Acquire lock on T Notinitialized
  • 63. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Notinitialized
  • 64. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Notinitialized First new instance or static reference of T per test
  • 65. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Re-initialize T’s super classes Notinitialized First new instance or static reference of T per test
  • 66. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Re-initialize T’s super classes Notinitialized First new instance or static reference of T per test Re-initialize T
  • 67. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs
  • 68. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class
  • 69. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java
  • 70. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process
  • 71. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process • Register a hook with test runner to tell us when a new test starts
  • 72. @_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization?
  • 73. @_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM?
  • 74. @_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM? • RQ3: Does VMVM impact fault finding ability?
  • 75. @_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches
  • 76. @_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique
  • 77. @_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique • Study performed on the popular Software Infrastructure Repository dataset
  • 78. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! RQ1: VMVM vs Test Minimization Larger is better
  • 79. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% RQ1: VMVM vs Test Minimization Larger is better
  • 80. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% RQ1: VMVM vs Test Minimization Larger is better
  • 81. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% 49% RQ1: VMVM vs Test Minimization Larger is better
  • 82. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes
  • 83. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes
  • 84. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes • Range from 5,000 LoC - 5,692,450 LoC; 3 - 292 test classes; 3.5-15 years in age
  • 85. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Larger is better
  • 86. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Average: 62% Larger is better
  • 87. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Max: 97% Average: 62% Larger is better
  • 88. @_jon_bell_ICSE 2014 June 5, 2014 Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup
  • 89. @_jon_bell_ICSE 2014 June 5, 2014 Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup • Result: Only average time per test is correlated with VMVM’s speedup (in fact, quite strongly; p < 0.0001)
  • 90. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR)
  • 91. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though?
  • 92. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications
  • 93. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications • Result: False positives occur when not isolated. VMVM shows no false positives or false negatives.
  • 94. @_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases
  • 95. @_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases • VMVM provides up to a 97% reduction in testing time through more efficient isolation (average 62%)
  • 96. @_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases • VMVM provides up to a 97% reduction in testing time through more efficient isolation (average 62%) • VMVM does not risk a reduction in fault finding
  • 97. @_jon_bell_ICSE 2014 June 5, 2014 Unit Test Virtualization with VMVM Jonathan Bell and Gail Kaiser Columbia University https://github.com/Programming-Systems-Lab/vmvm See a demo of VMVM at 2:30 today! Room MR G1-3 Forkm e on Github