Property Based Testing is an process to build robust systems.
It facilitates a deeper understanding of the system under test. It can be used on any testing level: unit, integration or functional.
The presentation introduces how Property Based Testing works, how to use it with PHPUnit, and in what way it differentiates from example based tests.
It talks about strategies to find good properties to check for.
This presentation was built for the Meet-Magento conference 2020 in Mumbai.
7. “Don’t write tests. Generate them.”
Prof. John Hughes, Inventor of QuickCheck
8. Example Based Testing (EBT)
• Think of an example
• Call method(s)
• Verify result
Property Based Testing (PBT)
• Think of range of inputs
• Let computer generate inputs
• Call method(s)
• Verify properties of result
• If a failure is found, shrink inputs
Property Based Testing?
9. Advantages of Property Based Testing
• Replace many example-based tests
• Find more bugs than example-based tests
• Test complex systems
• Test “black box” systems
• Facilitates thinking about system under development
11. What does it look like?
class ExampleStringToUpperTest extends TestCase
{
public function testLengthStaysTheSame()
{
$property = Property::forAll(
[ Generator::strings() ],
function (string $input): bool {
return mb_strlen($input) === mb_strlen(mb_strtoupper($input));
}
);
$this->assertThat($property, PropertyConstraint::check(200));
}
12. What does it look like?
Failed asserting that property is true.
Test runs: 7, seed: 1580127726247, smallest shrunk value(s):
array (
0 => ' ' . "0" . '',
)
vendor/steos/quickcheck/src/QuickCheck/PHPUnit/PropertyConstraint.php:35
test/ExampleStringTest.php:22
Failed asserting that property is true.
Test runs: 9, seed: 1580132863563, smallest shrunk value(s):
array (
0 => '' . "0" . ' ' . "0" . ‘‘,
)
vendor/steos/quickcheck/src/QuickCheck/PHPUnit/PropertyConstraint.php:35
test/ExampleStringTest.php:22
13. When to use Property Based Testing
• During Design
• During Implementation (like TDD)
• After Implementation
• Reproducing a bug
14. Figuring out the properties to test
• This is … not an easy thing (at first)
• But there are strategies to follow
15. Figuring out the properties to test
• reverse?
• depend on the order of an input sequence?
• depend on grouping of arguments? (Associative)
• depend on the order of arguments? (Commutative)
• have an identity value?
• change it’s output if it is called multiple times? (Idempotent)
Algebraic Properties
Does the Algorithm…
16. Figuring out the properties to test
Functionality
Like TDD, but better:
Generate the input values instead of hardcoding them.
Do not re-implement functionality!
17. Figuring out the properties to test
1. Generate names for directories and files
2. Create directories
3. Create files
4. Execute command (list files)
5. Assert number of files matches created files
Functionality
Example test for the ls CLI utility:
18. Figuring out the properties to test
Commerce is very stateful.
Shopping Cart customizations tend to be stateful, too.
Mostly stateful
19. Figuring out the properties to test
Model ó System
Modelling Stateful Systems
The model behaves like the System Under Test (SUT),
but it doesn’t use persistence or have a REST API.
20. Figuring out the properties to test
1. Create Model of System
2. Generate actions
3. Apply actions to Model and System
4. Check Model and System state match
Modelling Stateful Systems
The model behaves like the System Under Test (SUT),
but it doesn’t use persistence or have a REST API.
21. Property Based Testing in the Design Phase
• New Systems:
Create model before starting with the real implementation
• Existing systems:
The model can be partial (only the functionality to test)
• The model development is guided by tests (TDD like)
• Building the model gives me a better understanding of the
real system
22. Property Based Testing in the Design Phase
Plan
Create Model
Generate
Action
Write
Test
Write
Model
23. Property Based Testing in the Design Phase
My current task:
Downloadable Products for a different system.
I’ve used the same approach with Magento 2, too.
For example...
24. Property Based Testing in the Design Phase
1. Sketch out operations
Admin
• Create new
• Add file
• Replace file
• Remove file
• Delete product, keep downloads
• Delete product, remove downloads
Customer
• Purchase downloadable
• List available files
• Download file
Reporting
• Downloads per file
• …
25. Property Based Testing in the Design Phase
2. Generate Action
• Generate data for the initial operation on the system
• Return it as an array
• Write test for that operation
26. Property Based Testing in the Design Phase
2. Generate Action
private function genCreateDownloadableProductAction(): Gen
{
$contents = Gen::strings();
return Gen::tuples($this->genFileName(), $contents, $this->genLabel())
->map(function (array $tuple): array {
[$file, $contents, $label] = $tuple;
return ['create', $this->makeFilePath($file, $contents), $label];
});
}
27. Property Based Testing in the Design Phase
3. Write Test
public function test_Create()
{
$property = Property::forAll(
[$this->genCreateDownloadableProductAction()],
function (array $createAction): bool {
$model = new DownloadableProductModel();
$this->applyToModel($model, $createAction);
return $model->listFileLabels() === [$createAction[2]];
}
);
$this->assertThat($property, PropertyConstraint::check());
}
28. Property Based Testing in the Design Phase
4. Build Model
class DownloadableProductModel
{
// ...
public function create(string $file, string $label): void
{
$id = 'file_' . (count($this->files) + 1);
$this->files[$id] = [$file, $label];
}
29. Property Based Testing in the Design Phase
1. Create generator for next action
2. Think of way to verify action on model
3. Write test applying action to model
4. Implement action on model
5. GOTO 1
Build next Action...
All the while learning what I overlooked in the planning phase!
30. Property Based Testing in the Design Phase
When the model is complete
... the real work begins:
Implement the System with Property Based Tests
(and Example Based Tests where it makes sense).
Finally:
Use the model to check the system!
31. Summary
• Be curious!
Lets look outside of our comfort zone what others are doing.
• New skills require practice and patience. We need to be kind
to ourselves if something doesn’t work at first, but persist!
• PBT makes systems robust.
It facilitates thinking as much as the coding process.
