Top 10 Amil baba list Famous Amil baba In Pakistan Amil baba Kala jadu in Raw...
Next Generation Developer Testing: Parameterized Testing
1. Next Generation Developer Testing:
Parameterized Testing
Tao Xie
Department of Computer Science
University of Illinois at Urbana-Champaign
Email: taoxie@illinois.edu
http://taoxie.cs.illinois.edu/
In Collaboration with Microsoft Research
2. 2
Introduction
Developer (aka unit) testing – a widely adopted practice for
ensuring high quality software
A conventional unit test (CUT): small program with test inputs
and test assertions
void AddTest() {
HashSet set = new HashSet();
set.Add(7);
set.Add(3);
Assert.IsTrue(set.Count == 2);
}
Test Scenario
Test Assertions
Test Data
3. 3
Parameterized Unit Tests (PUTs)
Recent advances in unit testing introduced PUTs
void AddSpec(int x, int y)
{
HashSet set = new HashSet();
set.Add(x);
set.Add(y);
Assert.AreEqual(x == y, set.Count == 1);
Assert.AreEqual(x != y, set.Count == 2);
}
PUTs separate two concerns:
• Specification of externally visible behavior (assertions)
• Selection of internally relevant test inputs (coverage)
4. 4
Parameterized Unit Tests (PUTs)
More beneficial than CUTs
• Help describe behaviors for all test arguments
Address two main issues with CUTs
• Missing test data required for exercising important
behaviors
Low fault-detection capability
• Including test data that exercises the same behaviour
Redundant unit tests
5. 5
An Example using IntStack
public void CUT1() {
int elem = 1;
IntStack stk = new IntStack();
stk.Push(elem);
Assert.AreEqual(1, stk.Count());
}
Three CUTs
public void CUT2() {
int elem = 30;
IntStack stk = new IntStack();
stk.Push(elem);
Assert.AreEqual(1, stk.Count());
}
public void CUT3() {
int elem1 = 1, elem2 = 30;
IntStack stk = new IntStack();
stk.Push(elem1);
stk.Push(elem2);
Assert.AreEqual(2, stk.Count());
}
CUT1 and CUT2 exercise push with
different test data
CUT3 exercises push when stack is not
empty
Two main issues with CUTs:
Fault-detection capability issue:
undetected defect where things go wrong
when passing a negative value to push
Redundant test issue: CUT2 is
redundant with respect to CUT1
6. 6
An Example using IntStack
public void CUT1() {
int elem = 1;
IntStack stk = new IntStack();
stk.Push(elem);
Assert.AreEqual(1, stk.Count());
}
Three CUTs
public void CUT2() {
int elem = 30;
IntStack stk = new IntStack();
stk.Push(elem);
Assert.AreEqual(1, stk.Count());
}
public void CUT3() {
int elem1 = 1, elem2 = 30;
IntStack stk = new IntStack();
stk.Push(elem1);
stk.Push(elem2);
Assert.AreEqual(2, stk.Count());
}
No need to describe test data
• Generated automatically
Single PUT replaces multiple CUTs
• With reduced size of test code
public void PUT(int[] elem) {
Assume.IsTrue(elem != null);
IntStack stk = new IntStack();
for(int i in elem)
stk.push(elem);
Assert.AreEqual(elem.Length, stk.count());
}
An equivalent PUT
7. Parameterized Unit Tests are
Algebraic Specifications
• A Parameterized Unit Test can be read as a
universally quantified, conditional axiom.
void ReadWrite(Storage s, string name, string data) {
Assume.IsTrue(s!=null && name!=null && data!=null);
s.WriteResource(name, data);
var readData = s.ReadResource(name);
Assert.AreEqual(data, readData);
}
Storage s, string name, string data:
s ≠ null ⋀ name ≠ null ⋀ data ≠ null ⇒
equals(
ReadResource(WriteResource(s,name,data).state, name).retval,
data)
8. Parameterized Unit Testing
is going mainstream
Parameterized Unit Tests (PUTs) commonly supported by various
test frameworks
• .NET: Supported by .NET test frameworks
– http://www.mbunit.com/
– http://www.nunit.org/ …
• Java: Supported by JUnit 4.X
– http://www.junit.org/
Generating test inputs for PUTs supported by tools
• .NET: Supported by Microsoft Visual Studio 2015 IntelliTest
– Formerly Microsoft Research Pex: http://research.microsoft.com/pex/
• Java: Supported by Agitar AgitarOne
– http://www.agitar.com/
9. Parameterized Tests in JUnit
9
https://github.com/junit-team/junit/wiki/Parameterized-tests
11. Assumptions and Assertions
void PexAssume.IsTrue(bool c) {
if (!c)
throw new AssumptionViolationException();
}
void PexAssert.IsTrue(bool c) {
if (!c)
throw new AssertionViolationException();
}
• Assumptions and assertions induce branches
• Executions which cause assumption violations are
ignored, not reported as errors or test cases
12. Test Data Generation
• Human
– Expensive, incomplete, …
• Brute Force
– Pairwise, predefined data, etc…
• Semi - Random
– Cheap, Fast
– “It passed a thousand tests” feeling
• Dynamic Symbolic Execution:
IntelliTest/Pex, SAGE, CUTE, …
– Automated white-box
– Not random – Constraint Solving
13. 13
void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
T
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
a==null &&
a.Length>0 &&
a[0]==1234567890
Generates test data systematically
Done: There is no path left.
Background: DSE
14. Pex4Fun
http://pex4fun.com/
Nikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith Bishop. Teaching and Learning
Programming and Software Engineering via Interactive Gaming. In ICSE 2013, SEE.
http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf
1,703,247 clicked 'Ask Pex!'
15. Pex4Fun
• Click http://pex4fun.com/default.aspx?language=CSharp&sa
mple=_Template
• Copy and modify the following code snippet in the code
editing box (or simply click here)
using System;
using Microsoft.Pex.Framework;
using Microsoft.Pex.Framework.Settings;
[PexClass]
public class Program
{
[PexMethod]//[PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)]
public static string testMethod(int x, int y)
{
PexAssume.IsTrue(y >= 0);//replace here with your assumption
//... enter your code under test here
//if (x == 10000) throw new Exception();
PexAssert.IsTrue(y >= 0);//replace here with your assertion
return PexSymbolicValue.GetPathConditionString();
}
}
https://sites.google.com/site/teachpex/Home/pex-usage-tips
16. Microsoft Visual Studio 2015
IntelliTest
16
https://msdn.microsoft.com/en-us/library/dn823749.aspx
17. Microsoft Visual Studio 2015
IntelliTest
17https://msdn.microsoft.com/en-us/library/dn823749.aspx
18. 18
Recall: An Example using IntStack
public void CUT1() {
int elem = 1;
IntStack stk = new IntStack();
stk.Push(elem);
Assert.AreEqual(1, stk.Count());
}
Three CUTs
public void CUT2() {
int elem = 30;
IntStack stk = new IntStack();
stk.Push(elem);
Assert.AreEqual(1, stk.Count());
}
public void CUT3() {
int elem1 = 1, elem2 = 30;
IntStack stk = new IntStack();
stk.Push(elem1);
stk.Push(elem2);
Assert.AreEqual(2, stk.Count());
}
No need to describe test data
• Generated automatically
Single PUT replaces multiple CUTs
• With reduced size of test code
public void PUT(int[] elem) {
Assume.IsTrue(elem != null);
IntStack stk = new IntStack();
for(int i in elem)
stk.push(elem);
Assert.AreEqual(elem.Length, stk.count());
}
An equivalent PUT
19. 19
Test Generalization: CUTs PUT
Major Steps
• S1: Parameterize
• S2: Generalize Test Oracle
• S3: Add Assumptions
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011
http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
20. Example
00: public class SettingsGroup{
01: MSS storage; ...
02: public SettingsGroup(MSS storage) {
03: this.storage = storage;
04: }
05: public void SaveSetting(string sn, object sv) {
06: object ov = storage.GetSetting( sn );
07: //Avoid change if there is no real change
08: if (ov != null ) {
09: if(ov is string && sv is string && (string)ov==(string)sv ||
10: ov is int && sv is int && (int)ov==(int)sv ||
11: ov is bool&& sv is bool&& (bool)ov==(bool)sv ||
12: ov is Enum&& sv is Enum&& ov.Equals(sv))
13: return;
14: }
15: storage.SaveSetting(sn, sv);
16: if (Changed != null)
17: Changed(this, new SettingsEventArgs(sn));
18: }}
20
SettingsGroup class of NUnit with
the SaveSetting method under test
21. An Existing CUT
00: public class SettingsGroup{
01: MSS storage; ...
02: public SettingsGroup(MSS storage) {
03: this.storage = storage;
04: }
05: public void SaveSetting(string sn, object sv) {
06: object ov = storage.GetSetting( sn );
07: //Avoid change if there is no real change
08: if (ov != null ) {
09: if(ov is string && sv is string && (string)ov==(string)sv ||
10: ov is int && sv is int && (int)ov==(int)sv ||
11: ov is bool&& sv is bool&& (bool)ov==(bool)sv ||
12: ov is Enum&& sv is Enum&& ov.Equals(sv))
13: return;
14: }
15: storage.SaveSetting(sn, sv);
16: if (Changed != null)
17: Changed(this, new SettingsEventArgs(sn));
18: }}
21
00: //tg is of type SettingsGroup
01: [Test]
02: public void TestSettingsGroup() {
03: tg.SaveSetting("X",5);
04: tg.SaveSetting("NAME","Tom");
05: Assert.AreEqual(5,tg.GetSetting("X"));
06: Assert.AreEqual("Tom",tg.GetSetting("NAME"));
07: }
Existing CUT
SettingsGroup class of NUnit with
the SaveSetting method under test
22. ?
Issues with Existing CUT
22
Only CUT for verifying SaveSetting method
• Does not verify the behavior for the types bool and enum
• Does not cover the true branch in Line 8
Does test generalization addresses these two
issues
…
06: object ov = storage.GetSetting( sn );
07: //Avoid change if there is no real change
08: if (ov != null ) { …
23. 23
S1 - Parameterize
Promote all primitive values as arguments
• name of setting as a parameter of type string
• string “TOM” and int 5 as a parameter of type object
helping IntelliTest/Pex generate concrete values based on the
constraints encountered in different paths
Promote non-primitive objects such as receiver objects
as arguments
helping IntelliTest/Pex generate object states for the receiver
objects that can cover additional paths
//Original CUT
02: public void TestSettingsGroup() {
03: tg.SaveSetting("X",5);
04: tg.SaveSetting("NAME","Tom");….
//New PUT
02: public void TestSave(
SettingsGroup st,
string sn, object sv){
….
24. 24
S2 – Generalize Test Oracle
Replace the constant value, “TOM” and 5, with
the relevant parameter of the PUT
//Original CUT
02: public void TestSettingsGroup() {
03: tg.SaveSetting("X",5);
04: tg.SaveSetting("NAME","Tom");
05: Assert.AreEqual(5,tg.GetSetting("X"));
06: Assert.AreEqual("Tom",tg.GetSetting("NAME"));
….
//New PUT
02: public void TestSave(SettingsGroup st,
string sn, object sv){
03: st.SaveSetting(sn, sv);
04: PexAssert.AreEqual(sv,st.GetSetting(sn));
25. //New PUT
02: public void TestSave(SettingsGroup st,
string sn, object sv){
03: st.SaveSetting(sn, sv);
04: PexAssert.AreEqual(sv,st.GetSetting(sn));
25
S3 – Add Assumptions
IntelliTest/Pex requires guidance in generating
legal values for the parameters
E.g., Add the tag PexAssumeUnderTest (PAUT) with the parameter, i.e.,
generated value should not be null
//New PUT
02: public void TestSave([PAUT]SettingsGroup
st, [PAUT]string sn, [PAUT] object sv){
03: st.SaveSetting(sn, sv);
04: PexAssert.AreEqual(sv,st.GetSetting(sn));
26. 26
Example Summary
Resulting PUT
Result of S1 Parameters “sn” and “sv”
Result of S2 Line 4: “st.GetSetting(sn)”
Result of S3 Parameter “st”
00: //PAUT: PexAssumeUnderTest
01: [PexMethod]
02: public void TestSave([PAUT]SettingsGroup st,
[PAUT] string sn, [PAUT] object sv) {
03: st.SaveSetting(sn, sv);
04: PexAssert.AreEqual(sv,st.GetSetting(sn));
05: }
27. 27
Example - Test Generalization Results
Achieved 10% branch cov
Required 2 method calls
with values of type string and
int
2: public void
TestSave([PAUT]SettingsGroup st,
[PAUT]string sn, [PAUT] object sv){
3: st.SaveSetting(sn, sv);
4: PexAssert.AreEqual
(sv,st.GetSetting(sn));
2: public void TestSettingsGroup() {
3: tg.SaveSetting("X",5);
4: tg.SaveSetting("NAME","Tom");….
Achieved 90% branch cov
Required only 1 method call,
sufficient to test for all types
(parameter of type object)
Original CUT New PUT
30. Pattern
Roundtrip
• For an API f(x), f-1(f(x)) = x for all x
void ToStringParseRoundtrip(int value) {
string s = value.ToString();
int parsed = int.Parse(s);
Assert.AreEqual(value, parsed);
}
31. Pattern
State Relation
• Observe a state change
void ContainedAfterAdd(string value) {
var list = new List<string>();
list.Add(value);
Assert.IsTrue(list.Contains(value));
}
32. Pattern
Commutative Diagram
• Given two implementations f and g of the same
function, each possible requiring a different number
of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(…
(gm(x)…)), then it should hold that
f1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.
string Multiply(string x, string y);
int Multiply(int x, int y);
void CommutativeDiagram1(int x, int y) {
string z1 = Multiply(x, y).ToString();
string z2 = Multiply(x.ToString(), y.ToString());
PexAssert.AreEqual(z1, z2);
}
34. Behind the Scene of Code Hunt
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
34
35. It’s a game!
• iterative gameplay
• adaptive
• personalized
• no cheating
• clear winning criterion
code
test cases
Bishop et al. Code Hunt: Experience with Coding Contests at Scale. ICSE 2015, JSEET
http://taoxie.cs.illinois.edu/publications/icse15jseet-codehunt.pdf
36. Next Generation Developer
Testing: Parameterized Testing
Microsoft
Visual Studio 2015
- IntelliTest
JUnit TheoriesPexMethod
Test Generalization
CUTs PUTs
PUT Patterns
PUTs
…
38. Next Generation Developer
Testing: Parameterized Testing
Microsoft
Visual Studio 2015
- IntelliTest
JUnit TheoriesPexMethod
Test Generalization
CUTs PUTs
PUT Patterns
PUTs
…
http://taoxie.cs.illinois.edu/taoxie@illinois.edu
https://sites.google.com/site/teachpex/
39. Pattern
Roundtrip
• For an API f(x), f-1(f(x)) = x for all x
void PropertyRoundtrip(string value) {
var target = new Foo();
target.Name = value;
var roundtripped = target.Name;
Assert.AreEqual(value, roundtripped);
}
http://research.microsoft.com/pex/patterns.pdf
40. Pattern
Sanitized Roundtrip
• For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x
void ParseToString(string x) {
var normalized = int.Parse(x);
var intermediate = normalized.ToString();
var roundtripped = int.Parse(intermediate);
Assert(normalized == roundtripped);
}
http://research.microsoft.com/pex/patterns.pdf
41. Pattern
Reachability
• Indicate which portions of a PUT should be
reachable.
[PexExpectedGoals]
public void InvariantAfterParsing(string input)
{
ComplexDataStructure x;
bool success = ComplexDataStructure.TryParse(
input, out x);
PexAssume.IsTrue(success);
PexGoal.Reached();
x.AssertInvariant();
}
http://research.microsoft.com/pex/patterns.pdf
42. Pattern
Regression Tests
• Generated test asserts any observed value
– Return value, out parameters, PexGoal
• When code evolves, breaking changes in
observable will be discovered
int AddTest(int a, int b) {
return a + b; }
void AddTest01() {
var result = AddTest(0, 0);
Assert.AreEqual(0, result);
} http://research.microsoft.com/pex/patterns.pdf
43. Pattern
Same Observable Behavior
• Given two methods f(x) and g(x), and a method b(y)
that observes the result or the exception behavior of
a method, assert that f(x) and g(x) have same
observable behavior under b, i.e. b(f(x)) = b(g(x))
for all x.
public void ConcatsBehaveTheSame(
string left, string right)
{
PexAssert.AreBehaviorsEqual(
() => StringFormatter.ConcatV1(left, right),
() => StringFormatter.ConcatV2(left, right));
}
http://research.microsoft.com/pex/patterns.pdf
44. Pattern
Allowed Exception
• Allowed exception -> negative test case
[PexAllowedException(typeof(ArgumentException))]
void Test(object item) {
var foo = new Foo(item) // validates item
// generated test (C#)
[ExpectedException(typeof(ArgumentException))]
void Test01() {
Test(null); // argument check
}
http://research.microsoft.com/pex/patterns.pdf
45. 45
S4 – Add Factory Method
IntelliTest/Pex requires guidance handling non-
primitive objects
e.g., Add factory methods that generate instances of non-primitive objects
00: //PAUT: PexAssumeUnderTest
01: //MSS: MemorySettingsStorage (class)
01: [PexFactoryMethod(typeof(MSS))]
02: public static MSS Create([PAUT] string[] sn, [PAUT]object[] sv) {
03: PexAssume.IsTrue(sn.Length == sv.Length);
04: PexAssume.IsTrue(sn.Length > 0);
05: MSS mss = new MSS();
06: for(int count = 0; count < sn.Length; count++) {
07: mss.SaveSetting(sn[count], sv[count]);
08: }
09: return mss;
10: }
• Help create different object states for MSS
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011
http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
46. 46
S5 – Add Mock Object
Pex faces challenges in handling code that
interacts with external environment, e.g., file system
• Write mock objects to assist Pex and to test features in isolation
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011
http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
47. 47
Empirical Study
RQ1: Branch coverage
• How much higher percentage of branch coverage is achieved
by retrofitted PUTs compared to existing CUTs?
RQ2: Defect detection
• How many new defects (that are not detected by existing
CUTs) are detected by PUTs and vice-versa?
RQ3: Generalization effort
• How much effort is required for generalizing CUTs to PUTs?
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011
http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
48. Study Setup
Three Subject Applications
48
Subjects Downloads Code Under Test
# Classes # Methods KLOC
Test Code
# Classes # CUTs KLOC
NUnit 193,563 9 87 1.4 9 49 0.9
DSA 3,239 27 259 2.4 20 337 2.5
QuickGraph 7,969 56 463 6.2 9 21 1.2
TOTAL 92 809 10 38 407 4.6
High downloads count
Lines of code under test: 10 KLOC
Number of CUTs: 407
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011
http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
49. Study Setup – cont.
49
Conducted by the first and second authors
• No prior knowledge of subjects
• Two years of experience with PUTs and Pex
• Retrofitted 407 CUTs (4.6 KLOC) as 224 PUTs (4.0 KLOC)
Generated three categories of CUTs
• C1: Existing CUTs
• C2: CUTs generated from PUTs
• C3: Existing CUTs + RTs (tests generated using Randoop)
Help show that benefits of generalization cannot be achieved by
generating tests randomly
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011
http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
50. RQ1: Branch Coverage
50
Subjects # RTs Branch Coverage
CUTs (%) CUTs + RTs (%) PUTs (%)
Overall
Increase
(%)
Maximum
Increase
(%)
NUnit 144 78 78 88 10 52
DSA 615 91 91 92 1 1
QuickGrap
h
3628 87 88 89 2 11
CUTs + RTs:
Added 144, 615, and 3628 tests using Randoop
Branch coverage increased by 0%, 0%, and 1%
CUTs generated from PUTs:
Branch coverage increased by 10%, 1%, and 2%
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011
http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
51. RQ2: Defect Detection
51
RTs: CUTs generated using Randoop
Pex without generalized PUTs: CUTs generated by applying Pex on public
methods
CUTs generated from PUTs: CUTs generated by applying Pex on generalized PUTs
• Detected all defects detected by the first two categories
CUTs category #Failing Tests
DSA NUnit QuickGraph
# Real Defects
Basic CUTs 0 0 0 0
RTs 90 25 738 4
Pex without
generalized PUTs
23 170 17 2
CUTs generated
from PUTs
15 4 0 19
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011
http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
52. RQ3: Generalization Effort
52
Both authors conducted comparable amount of
generalization
Effort spent in hours
• NUnit: 2.8 hrs
• DSA: 13.8 hrs
• QuickGraph: 1.5 hrs
Effort is worthwhile compared to benefits of test
generalization
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011
http://taoxie.cs.illinois.edu/publications/fase11-put.pdf