Having a reliable test suite is incredibly useful when making changes to an existing codebase, both big and small. Mutation testing frameworks run tests against slightly-changed source code in order to detect whether the tests are actually checking the different paths of logic through the application. The aim is to improve the robustness of your test suite, and give you confidence that you aren't introducing any unintended changes. This presentation gives an overview of mutation testing, along with worked examples in JavaScript of how it catches gaps and improves test coverage.

1. z
Mutation Testing
Testing your tests to test that they test what you think
they test
Stephen Leigh
Test Manager
Sky Betting and Gaming

2. z
Why do we write unit tests?
- Fast feedback if code is not behaving in the expected way
- Good for debugging, reducing cost of defects
- A form of documentation
- Encourages better design principles
The most relevant reason for the purposes of mutation testing:
- To ensure that any changes we make don’t accidentally
break the functionality of the existing codebase

3. z
Why do we write unit tests?
“To ensure that any changes we make don’t accidentally break the
functionality of the existing codebase”
To make sure our changes don’t alter any existing functionality, we
need to be sure our unit tests are exercising this functionality
appropriately.
This is what mutation testing can help with.

4. z
What is mutation testing?
A mutant is a small change in an application’s source code, e.g.:
Source Code
const isPositive = num => num > 0;
Mutant Source Code
const isPositive = num => num >= 0;
Mutation test tools create a large number of these mutants,
then run unit tests against the edited code

5. z
Mutation examples
Code Possible mutations
if (x === 3) if (x >= 3)
if (x <= 3)
if (x !== 3)
if (true)
if (false)
if (x === 3) {
k++
}
if (x === 3) {}
if (x === 3) k--
return {token: “c38bf32”} return {}
return null
return {token: “”}

6. z
Killing mutants
When unit tests are run against the mutant, one of three things
happens:
- At least one unit test fails. This means the mutant has been ‘killed’
and therefore the part of the code that has been changed is
properly covered.
- All unit tests pass. This means the mutant has ‘survived’, and the
changed functionality is not covered by tests.
- Infinite loop/runtime error. This usually means that the mutation
isn’t something that could actually happen, and counts as a kill.
As you might expect, generating hundreds of mutants is very
computationally expensive. Fortunately, it’s all automatically handled
by the mutation testing framework.

7. Simple example
Source Code
function isPositive(num) {
if (num > 0) {
return true;
} else {
return false;
}
}
Pseudo-unit tests
testPositive() {
assertTrue(isPositive(6))
}
testNegative() {
assertFalse(isPositive(-3))
}

8. Simple example
Mutated Source Code
function isPositive(num) {
if (num >= 0) {
return true;
} else {
return false;
}
}
Pseudo-unit tests
testPositive() {
assertTrue(isPositive(6))
}
testNegative() {
assertFalse(isPositive(-3))
}

9. Simple example
Mutated Source Code
function isPositive(num) {
if (num >= 0) {
return true;
} else {
return false;
}
}
Pseudo-unit tests
testPositive() {
assertTrue(isPositive(6))
}
testNegative() {
assertFalse(isPositive(-3))
}
testZero() {
assertFalse(isPositive(0))
}
Mutant killed!

10. More complex example
Source Code
function handleLogin(request, response)
const {username, password} =
if (!username) {
return response.status(400)
.json({reason: ‘ERR_NO_USERNAME’})
}
if (!password) {
return response.status(400)
.json({reason: ‘ERR_NO_PASSWORD’})
}
}
Pseudo-unit tests
const testRequest = {
body: {} // no username or password
fields
}
const mockResponse = () => ...
testNoUsername() {
handleLogin(testRequest, mockResponse);
expect(mockResponse.calls.single.toBe(400)
);
}
testNoPassword() {
handleLogin(testRequest, mockResponse);
expect(mockResponse.calls.single.toBe(400)
);
}

11. More complex example
Mutated Source Code
function handleLogin(request, response)
const {username, password} =
if (!username) {
return response.status(400)
.json({reason: ‘ERR_NO_USERNAME’})
}
if (false) {
return response.status(400)
.json({reason: ‘ERR_NO_PASSWORD’})
}
}
Pseudo-unit tests
const testRequest = {
body: {} // no username or password
fields
}
const mockResponse = () => ...
testNoUsername() {
handleLogin(testRequest, mockResponse);
expect(mockResponse.calls.single.toBe(400)
);
}
testNoPassword() {
handleLogin(testRequest, mockResponse);
expect(mockResponse.calls.single.toBe(400)
);
}

12. More complex example
Mutated Source Code
function handleLogin(request, response)
const {username, password} =
if (!username) {
return response.status(400)
.json({reason: ‘ERR_NO_USERNAME’})
}
if (false) {
return response.status(400)
.json({reason: ‘ERR_NO_PASSWORD’})
}
}
Pseudo-unit tests
const testRequest = {
body: {} // no username or password
fields
}
const mockResponse = () => ...
testNoUsername() {
handleLogin(testRequest, mockResponse);
expect(mockResponse.calls[0].toBe(400));
}
testNoPassword() {
const testRequestUsername = {
body: {
username: “abc”
}
};
handleLogin(testRequestUsername,
mockResponse);Mutant killed!

13. z
Mutation testing: advantages and disadvantages
Advantages
• More reliable metric than line coverage – actually ensures your unit tests are testing
what they should be, and that they follow all possible lines of logic (cyclomatic
complexity)
• Catches many small and easy-to-miss programming errors, as well as holes in unit
tests that would otherwise go unnoticed
Disadvantages
• Extremely computationally expensive – runs can take several hours, making it
unsuitable to run as part of a standard release process
• Requires brainpower to sort ‘junk’ mutations or unimportant survivors from useful
catches. May be an unfavourable signal-to-noise ratio. In these cases it’s potentially
a more useful process to compare over a period of time, to ensure surviving
mutations don’t start going up dramatically.

14. z
Suggested Tools
- JavaScript: Stryker
- Java/Kotlin: PITest
Questions?