Mutation testing for a safer Future by Mathieu Godart - Sêmeia

1. Mutation Testing,
for a Safer Future
Mathieu Godart – Sêmeia

2. Who test your code?

3. You, i.e. your tests.

4. « Quis custodiet ipsos
custodes? »
« Who will guard the
guards? »
— Juvenal, first century AD

5. « Quis custodiet ipsos
custodes? »
« Who will guard the
guards? »
— Juvenal, first century AD

6. Who test your tests?

7. Code coverage myth.
int sum(int a, int b) {
return a * b; // Should be a + b
}

8. Code coverage myth.
test_result test_sum() {
int result = sum(9, 5);
if (result > 0) {
return success;
}
return fail;
}

9. Code coverage myth.
test_result test_sum() {
int result = sum(9, 5);
if (result > 0) {
return success;
}
return fail;
}
• test_sum() => Success. Coverage: 100%

10. Code Coverage
is just a way to measure ratio of
executed code

11. Mutation Testing:
breaking the code and
checking that the failure is
detected by a test

12. Mutation Example
int sum(int a, int b) {
return a + b;
}
Becomes:
int sum(int a, int b) {
return a * b;
}

13. Kill the Mutant!

14. Kill the Mutant!

15. History
• 1971: Idea by Richard Lipton.
• 1981: Implementation by Timothy Budd, Yale
Univ.
• 2004: Hardware verification by Certess
(Synopsys).

16. The competent programmer,
most bugs due to small syntax

17. Coupling effect,
faults can cascade

18. Some Mutation Operators
• Statement deletion
• Statement duplication or insertion,
e.g. goto fail;
• Replacement of boolean subexpressions with true and false
• Replacement of some operations with others,
e.g. + with *, - with /, > with >=, == and <=
• Replacement of variables with others from the same scope

19. Who uses this today?
Java, Dot Net… Swift?

20. Do it by hand!

22. Fuzzing

23. Fuzzing
can be seen as
Mutation Testing.

24. Fuzzing:
using invalid, unexpected,
or random data
as input to a program

25. 1950
taking punched cards
from the trash as
input to programs

26. 1983
« The Monkey » by Steve
Capps:
testing Mac OS with
random user inputs

27. Infinite Monkey Theorem

28. Infinite Monkey Theorem

29. But remember to write tests!
(before you write code)
It will save you time!

30. References
• LLVM-Based Mutation Testing System. RFC.
https://lowlevelbits.org/llvm-based-mutation-testing-system.-request-for-comments
• Wikipedia
https://en.wikipedia.org/wiki/Mutation_testing
https://en.wikipedia.org/wiki/Fuzzing
• The blog from our friends
https://blog.xebia.com/mutation-testing-how-good-are-your-unit-tests/
https://blog.octo.com/mutation-testing-un-pas-de-plus-vers-la-perfection/

31. Thank You
Mathieu Godart – Sêmeia