The slides about Microsoft IntelliTest that I presented on a meetup organized by IncQuery Labs Ltd.

1. Budapest University of Technology and Economics
Department of Measurement and Information Systems
Budapest University of Technology and Economics
Fault Tolerant Systems Research Group
White-box Unit Test Generation
with Microsoft IntelliTest
Dávid Honfi
1

2. Focus
2
module B
module C
service
Unit testing
module A
Code-based
(White-box)

3. White-box test design
public int M1(int a, int b)
{
if(a == 0)
{
Console.WriteLine(ERROR_MSG);
return -1;
}
if(b > a) return b*a+5;
else return (a+b)/2;
}
Let’s cover all
the statements!
a==0
a!=0 && b>a
a!=0 && b<=a
a==0 a!=0 && b<=a a!=0 & b>a
a 0 1 2
b 0 0 3

4. What’s missing?
 Usually encoding observed outputs
 Where can it be employed?
oBasic, common bugs (e.g., uncaught exceptions)
oViolation of assertions or contracts
oDeviations from existing outputs (regression)
 Automation with code exploration
test case = input + expected output

5. Symbolic execution: the idea
 Static program analysis technique from the ’70s
 Application for test generation
o Symbolic variables instead of normal ones
o Constraints forming for each path with symb. variables
o Constraint solving (e.g., SMT solver)
o A solution yields an input to execute a given path
 New century, new progress:
o Enough computing power (e.g., for SMT solvers)
o New ideas, extensions, algorithms and tools
5

6. Existing tools using symbolic execution
 .NET: Microsoft IntelliTest (a.k.a. Pex)
 x86 binary: Microsoft SAGE
 Java
oNASA Symbolic PathFinder
oCATG
o…
 JavaScript: Jalangi
 C: KLEE

7. IntelliTest
Microsoft

8. DEMO 1.
The basic use of IntelliTest

9. public int M1(int a, int b)
{
if(a == 0)
{
Console.WriteLine(ERROR_MSG);
return -1;
}
if(b > a) return b*a+5;
else return (a+b)/2;
}
Details of IntelliTest (Dynamic SE)
Source code
Concrete
input values
Symbolic
variables
Constraint
solver
Constraints
Concrete
execution
Symbolic
execution
a==0
a:=0
b:=0 a,b
Transformationa!=0
a:=1
b:=0

10. public int M1(int a, int b)
{
if(a == 0)
{
Console.WriteLine(ERROR_MSG);
return -1;
}
if(b > a) return b*a+5;
else return (a+b)/2;
}
Details of IntelliTest (Dynamic SE)
Source code
Concrete
input values
Symbolic
variables
Constraint
solver
Constraints
Concrete
execution
Symbolic
execution
a!=0 &&
b<=a
a!=0 && b>a
a:=1
b:=0
a:=2
b:=3

11. Parameterized Unit Testing
 Idea: Using test methods as specifications
o Easy to understand, easy to check, etc.
o But: too specific (used for a code unit), verbose, etc.
 Parameterized Unit Test (PUT)
o Wrapper method for method/unit under test
o Main elements
• Inputs of the unit
• Assumptions for input space restriction
• Call to the unit
• Assertions for expected results
o Serves as a specification  Test generators can use it
11

12. Example: Parameterized Unit Testing
12
void ReduceQuantityPUT(Product prod, int soldCount) {
// Assumptions
Assume.IsTrue(prod != null && prod.Quantity > 0);
Assume.IsTrue(soldCount > 0);
int oldQuantity = prod.Quantity;
// Calling the UUT
int newQuantity = StorageManager.ReduceQuantity(prod,soldCount);
// Assertions
Assert.IsTrue(newQuantity >= 0);
Assert.IsTrue(newQuantity < oldQuantity);
}
/// The method reduces the quantity of the specified
/// product. The product is known to be NOT null with
/// quantity larger than 0, also the sold amount is
/// always more than zero. The method has effects on
/// the database, and returns the new quantity of the
/// product. If the quantity would be negative, the
/// method reduces the quantity to zero.
int ReduceQuantity(Product prod, int soldCount) { … }

13. Example: Parameterized Unit Testing
13
void ReduceQuantityPUT(Product prod, int soldCount) {
// Assumptions
Assume.IsTrue(prod != null && prod.Quantity > 0);
Assume.IsTrue(soldCount > 0);
int oldQuantity = prod.Quantity;
// Calling the UUT
int newQuantity = StorageManager.ReduceQuantity(prod,soldCount);
// Assertions
Assert.IsTrue(newQuantity >= 0);
Assert.IsTrue(newQuantity < oldQuantity);
}
/// The method reduces the quantity of the specified
/// product. The product is known to be NOT null with
/// quantity larger than 0, also the sold amount is
/// always more than zero. The method has effects on
/// the database, and returns the new quantity of the
/// product. If the quantity would be negative, the
/// method reduces the quantity to zero.
int ReduceQuantity(Product prod, int soldCount) { … }

14. DEMO 2.
Parameterized Unit Testing with IntelliTest

15. Challenges of SE-based techniques
1. Exponential growth of execution paths
2. Complex arithmetic expressions
3. Floating point operations
4. Compound structures and objects
5. Pointer operations
6. Interaction with the environment
7. Multithreading
8. …
15
T. Chen et al. „State of the art: Dynamic symbolic execution for automated test generation”.
Future Generation Computer Systems, 29(7), 2013

16. Challenges of SE-based techniques
1. Exponential growth of execution paths
2. Complex arithmetic expressions
3. Floating point operations
4. Compound structures and objects
5. Pointer operations
6. Interaction with the environment
7. Multithreading
8. …
16
T. Chen et al. „State of the art: Dynamic symbolic execution for automated test generation”.
Future Generation Computer Systems, 29(7), 2013
SEViz

17. Visualizing symbolic execution
 SEViz: Symbolic Execution VIsualiZer
 Goal: Help identifying test generation problems
o Long execution paths
o Unsolvable constraints
o …
 Solution: Symbolic execution tree with metadata
17
Mapping to
source code
Calls to constraint solver
1
No Yes
End of execution paths
No test Failed test Passed test
1 11 1 1 1
SHAPE BORDER COLOR

18. Visualizing symbolic execution
 Additional metadata
o Sequence number
o Path condition
o Incremental path condition (based on parent)
o Status: can have siblings or not
o Source code mapping (if available)
18

19. DEMO 3.
Visualizing test generation with SEViz

20. Challenges of SE-based techniques
1. Exponential growth of execution paths
2. Complex arithmetic expressions
3. Floating point operations
4. Compound structures and objects
5. Pointer operations
6. Interaction with the environment
7. Multithreading
8. …
20
T. Chen et al. „State of the art: Dynamic symbolic execution for automated test generation”.
Future Generation Computer Systems, 29(7), 2013
SEViz
Automated
isolation

21. Existing solutions
 Stubbing and mocking (faking)
o Fixed values and checks for all DSE executions
o Not suitable for test generation
 Parameterized mocking
o Interaction with DSE is possible
• More relevant test cases
• Custom behavior in mocks: e.g., state change of objects
o Introduces complexity for users of DSE
• Requires large amount of time and effort
• Not trivial task in case of complex structures
o Fakes cannot be generated under certain conditions
21

22. Approach for automated isolation
 Automated isolation on source code level
1. Abstract syntax tree transformations in the SUT
2. Parameterized sandbox synthesization
22
SUT
C1
C2
Environment
i(C1,C2)
i(C2,E)
i(C2,C1)
i(C1,E)
SUT
C1’ C2’
Environment
SB i'(C2,E)i'(C1,E)
i(C2,C1)
i(C1,C2)
Replaced
external
objects
Replaced
external
calls
Sandbox

23. Example of AST transformation
23
public class WeekendNotifier {
public bool IsWeekendNear() {
DateTime date = DateTime.GetNow();
date.AddDays(2);
if(date.GetDay() == "Saturday") return true;
return false;
}
}
public class WeekendNotifier {
public bool IsWeekendNear() {
DynamicFake date = Fake.DateTimeGetNow();
Fake.DateTimeAddDays(2,date);
if(Fake.DateTimeGetDay(date) == "Saturday") return true;
return false;
}
}

24. Example of parameterized sandbox
24
public static class Fake {
public DynamicFake DateTimeGetNow() {
// Return a state container object instead of the original
return new DynamicFake();
}
public void DateTimeAddDays(int days, DynamicFake date) {
// TODO: State change of date using DSE
}
public int DateTimeGetDay(DynamicFake date) {
// Obtaining return value from DSE
return DSEEngine.ChooseValue<int>();
}
}

25. Summary
 White-box test generation: symbolic execution
 IntelliTest, a state-of-the-art test generator
o Dynamic Symbolic Execution (DSE)
o Parameterized Unit Tests
 DSE has several challenges, e.g.:
o Test generation problem indentification
o Environment dependencies
 My research
o SEViz: A tool for visualizing symbolic execution
o Automated isolation with source code transformation
25

26. Resources
 IntelliTest
https://www.visualstudio.com/en-
us/docs/test/developer-testing/intellitest-manual
https://www.codehunt.com/
 SEViz [ICST’15 Tool Paper]
https://ftsrg.github.io/seviz
 Automated isolation
https://pp.bme.hu/eecs/article/view/9768
26