The document discusses systematic unit testing and describes it as an explicit procedure for choosing and creating test cases, executing tests and documenting results, evaluating results, and deciding when testing is complete. It provides examples of unit tests using PHPUnit and discusses best practices for writing independent, isolated tests and generating test reports. Code coverage is presented as one metric for determining when testing is complete, but not a perfect measure on its own.

1. S Y S T E M AT I C U N I T
T E S T I N G
S C O T T G R A N T

3. A B O U T M E
• Scott Grant (hi!)
• Startup CTO in Kingston, Ontario
• Formerly @ Electronic Arts, Google
• Formerly Adjunct Prof. @ Queen's University
• Prof. James R. Cordy (CISC327: Software Quality
Assurance)

4. A B O U T M E G A M A N 2
• Developed and published by
Capcom in 1988 for the Nintendo
Entertainment System
• In 200X, Dr. Wily and his eight evil
Robot Masters threaten Mega Man,
Dr. Light, and the world

5. W H AT I S S Y S T E M AT I C
T E S T I N G ?

6. W H AT I S S Y S T E M AT I C T E S T I N G ?
• An explicit discipline or procedure (a system) for:

7. W H AT I S S Y S T E M AT I C T E S T I N G ?
• An explicit discipline or procedure (a system) for:
• choosing and creating test cases

8. W H AT I S S Y S T E M AT I C T E S T I N G ?
• An explicit discipline or procedure (a system) for:
• choosing and creating test cases
• executing the tests and documenting the results

9. W H AT I S S Y S T E M AT I C T E S T I N G ?
• An explicit discipline or procedure (a system) for:
• choosing and creating test cases
• executing the tests and documenting the results
• evaluating the results, possibly automatically

10. W H AT I S S Y S T E M AT I C T E S T I N G ?
• An explicit discipline or procedure (a system) for:
• choosing and creating test cases
• executing the tests and documenting the results
• evaluating the results, possibly automatically
• deciding when we are done (enough testing)

11. U N I T T E S T S
• Units are, preferably, the smallest testable part of an
application
• Unit tests are designed to test small pieces of the
code (units) independently

12. U N I T T E S T S
• Units are, preferably, the smallest testable part of an
application
• Unit tests are designed to test small pieces of the
code (units) independently
function sum( $a, $b ) {
return $a + $b;
}

13. U N I T T E S T S
• Units are, preferably, the smallest testable part of an
application
• Unit tests are designed to test small pieces of the
code (units) independently
function sum( $a, $b ) {
return $a + $b;
}
sum( 1, 1 ) == 2

14. U N I T T E S T S
• Units are, preferably, the smallest testable part of an
application
• Unit tests are designed to test small pieces of the
code (units) independently
function sum( $a, $b ) {
return $a + $b;
}
sum( 1, 1 ) == 2
$this->assertEquals( 2, sum( 1, 1 ) );

15. A S S E R T I O N S
• In many ways, the heart of a unit test

16. A S S E R T I O N S
• In many ways, the heart of a unit test
• Checks for a predefined constraint about the program
state
• "After sum is called with the arguments 1 and 1, it will
return the value 2."

17. A S S E R T I O N S
• In many ways, the heart of a unit test
• Checks for a predefined constraint about the program
state
• "After sum is called with the arguments 1 and 1, it will
return the value 2."
• If the constraint does not hold, the test fails
• The test asserts that the code is correct for a particular
context if the assertion holds

18. P H P U N I T
• Provides a framework for creating and executing unit
tests easily, quickly, and reproducibly
• Based on atomic unit tests with assertions
• Open-source!
• https://github.com/sebastianbergmann/phpunit

19. 1 . C H O O S I N G A N D
C R E AT I N G T E S T C A S E S

20. W H AT M A K E S A G O O D T E S T ?

21. W H AT M A K E S A G O O D T E S T ?
• Independent: Each test needs to run independently
from other tests and environments

22. W H AT M A K E S A G O O D T E S T ?
• Independent: Each test needs to run independently
from other tests and environments
• Fast: To have useful tests and to be able to run them
as often as possible, tests need to be fast

23. W H AT M A K E S A G O O D T E S T ?
• Independent: Each test needs to run independently
from other tests and environments
• Fast: To have useful tests and to be able to run them
as often as possible, tests need to be fast
• Repeatable: You should be able to run a test as many
times as you want with the same result

24. W H AT M A K E S A G O O D T E S T ?
• Up to date: Tests are written once, but code can be
changed or extended

25. W H AT M A K E S A G O O D T E S T ?
• Up to date: Tests are written once, but code can be
changed or extended
• Short: Tests should be just a few lines--easy to read
and understand

26. W H AT M A K E S A G O O D T E S T ?
• Up to date: Tests are written once, but code can be
changed or extended
• Short: Tests should be just a few lines--easy to read
and understand
• Resilient: Once written, tests shouldn't change until
the behaviour of tested class/method changes

28. T E S T S I N I S O L AT I O N

29. T E S T S I N I S O L AT I O N
• Before each test, initialize the appropriate state

30. T E S T S I N I S O L AT I O N
• Before each test, initialize the appropriate state
• Next, perform the test

31. T E S T S I N I S O L AT I O N
• Before each test, initialize the appropriate state
• Next, perform the test
• One test! One!!

32. T E S T S I N I S O L AT I O N
• Before each test, initialize the appropriate state
• Next, perform the test
• One test! One!!
• After each test, clean up after yourself

33. T E S T S I N I S O L AT I O N
• Before each test, initialize the appropriate state
• Next, perform the test
• One test! One!!
• After each test, clean up after yourself
• There are more tests after this one, we don't want to
corrupt them with our test data

34. function is_even( $x ) {
if ( 0 == $x % 2 ) {
return true;
}
return false;
}
function is_odd( $x ) {
// ??
}

35. class Test_WCTO extends WP_UnitTestCase {
public function test_even_with_even() {
$this->assertTrue( is_even( 2 ) );
}
public function test_even_with_odd() {
$this->assertFalse( is_even( 1 ) );
}
}

36. C H O O S I N G & O R G A N I Z I N G T E S T S
• The experimental model of software testing gives us
two important principles for our test plan:

37. C H O O S I N G & O R G A N I Z I N G T E S T S
• The experimental model of software testing gives us
two important principles for our test plan:
• Test inputs should be chosen to carefully isolate
different causes of failure (the experimental
variables)

38. C H O O S I N G & O R G A N I Z I N G T E S T S
• The experimental model of software testing gives us
two important principles for our test plan:
• Test inputs should be chosen to carefully isolate
different causes of failure (the experimental
variables)
• Test cases should be ordered such that each test
only assumes features to be working that have
already been tested by a previous test

39. function is_odd( $x ) {
return ! is_even( $x );
}

40. function is_odd( $x ) {
$remainder = $x % 2;
if ( $remainder = 1 ) {
return true;
}
return false;
}

41. 2 . E X E C U T I N G T H E T E S T S A N D
D O C U M E N T I N G T H E R E S U LT S

42. T E S T R E P O R T S
• Output of test execution should be summarized in a
readable report

43. T E S T R E P O R T S
• Output of test execution should be summarized in a
readable report
• Test reports should be designed to be concise, easy to
read, and to clearly point out failures or unexpectedly
changed results

44. T E S T R E P O R T S
• Output of test execution should be summarized in a
readable report
• Test reports should be designed to be concise, easy to
read, and to clearly point out failures or unexpectedly
changed results
• Test results should be in a standardized form for easy
comparison with future test executions

45. PHPUnit 4.5.0 by Sebastian Bergmann
and contributors.
Configuration read from /tmp/
wordpress/src/wp-content/plugins/two-
factor/phpunit.xml
.....................................
................
Time: 11.37 seconds, Memory: 71.50Mb
OK (53 tests, 90 assertions)

46. PHPUnit 4.4.0 by Sebastian Bergmann.
Configuration read from /home/travis/build/scotchfield/arcadia/
test/phpunit_travis.xml.dist
...............................................................
...............................................................
......................................................F........
....................................
Time: 2.52 seconds, Memory: 9.50Mb
There was 1 failure:
1) TestArcadiaUser::test_get_user_by_name_simple
Failed asserting that false is not false.
/home/travis/build/scotchfield/arcadia/test/tests/include/
test_user.php:43
FAILURES!
Tests: 225, Assertions: 292, Failures: 1.

48. 3 . E VA L U AT I N G T H E R E S U LT S ,
P O S S I B LY A U T O M AT I C A L LY

49. T E S T R U N S U C C E S S
• When has a test run successfully completed?

50. T E S T R U N S U C C E S S
• When has a test run successfully completed?
• When all of the unit tests pass!

51. T E S T R U N S U C C E S S
• When has a test run successfully completed?
• When all of the unit tests pass!
• When does a unit test pass?

52. T E S T R U N S U C C E S S
• When has a test run successfully completed?
• When all of the unit tests pass!
• When does a unit test pass?
• When all of the assertions hold!

53. T E S T R U N S U C C E S S
• When has a test run successfully completed?
• When all of the unit tests pass!
• When does a unit test pass?
• When all of the assertions hold!
• When does an assertion hold?

54. T E S T R U N S U C C E S S
• When has a test run successfully completed?
• When all of the unit tests pass!
• When does a unit test pass?
• When all of the assertions hold!
• When does an assertion hold?
• When that individual piece of expected behaviour
captured in the assertion holds!

55. S A F E T Y B L A N K E T
• When a programmer makes a change to the code (any
change), they can run the test suite

56. S A F E T Y B L A N K E T
• When a programmer makes a change to the code (any
change), they can run the test suite
• If all of the tests pass, and we believe the test suite
covers enough functionality, we're fairly confident
that the code works

57. S A F E T Y B L A N K E T
• When a programmer makes a change to the code (any
change), they can run the test suite
• If all of the tests pass, and we believe the test suite
covers enough functionality, we're fairly confident
that the code works
• If a test fails, we might have broken something (or
exposed a bug!), and can fix it before checking in

58. http://cybermoonstudios.deviantart.com/art/Mega-Man-2-Robot-Masters-194915393

59. A U T O M AT E D T E S T I N G
• A continuous build machine can run unit tests after
every commit

60. A U T O M AT E D T E S T I N G
• A continuous build machine can run unit tests after
every commit
• Useful even if a programmer isn't proactively
running unit tests

61. A U T O M AT E D T E S T I N G
• A continuous build machine can run unit tests after
every commit
• Useful even if a programmer isn't proactively
running unit tests
• Catch bugs as they're introduced

62. A U T O M AT E D T E S T I N G
• A continuous build machine can run unit tests after
every commit
• Useful even if a programmer isn't proactively
running unit tests
• Catch bugs as they're introduced
• Or provide confidence that the code isn't going to
cause problems

64. 4 . D E C I D I N G W H E N W E A R E
D O N E ( E N O U G H T E S T I N G )

65. W H E N A R E W E D O N E ?
• When we've achieved 100% code coverage

66. W H E N A R E W E D O N E ?
• When we've achieved 100% code coverage
• Not an ideal statistic

67. W H E N A R E W E D O N E ?
• When we've achieved 100% code coverage
• Not an ideal statistic
• Does not check for input, output, or functionality
coverage

68. W H E N A R E W E D O N E ?
• When we've achieved 100% code coverage
• Not an ideal statistic
• Does not check for input, output, or functionality
coverage
• Hard to achieve in practice

69. W H E N A R E W E D O N E ?
• When we've achieved 100% code coverage
• Not an ideal statistic
• Does not check for input, output, or functionality
coverage
• Hard to achieve in practice
• However, it's a clear and unambiguous goal

70. C O D E C O V E R A G E
• PHPUnit has a set of --coverage-* options

71. C O D E C O V E R A G E
• PHPUnit has a set of --coverage-* options
• Generate code coverage reports in various formats

72. C O D E C O V E R A G E
• PHPUnit has a set of --coverage-* options
• Generate code coverage reports in various formats
• Checks which lines of code are covered by at least
one unit test

73. C O D E C O V E R A G E
• PHPUnit has a set of --coverage-* options
• Generate code coverage reports in various formats
• Checks which lines of code are covered by at least
one unit test
• In addition, functions and classes

74. C O D E C O V E R A G E
• PHPUnit has a set of --coverage-* options
• Generate code coverage reports in various formats
• Checks which lines of code are covered by at least
one unit test
• In addition, functions and classes
• Does not check for input, output, or functionality
coverage

75. https://github.com/scotchfield/roller

76. https://github.com/scotchfield/roller

77. https://github.com/scotchfield/arcadia

78. https://github.com/scotchfield/arcadia

80. A systematic approach helps inform where new
tests are needed and provides one way to know
when a test suite is complete.
@ S C O T C H F I E L D
@ S G R A N T
S C O T T @ S C O O TA H . C O M