A presentation about tests antipatterns. What are typical problems connected with testing. Why and how we should avoid slow tests, obscure tests, overusing mocks, manual tests, fragile tests, erratic tests, eager tests, verbose tests, mystery guests in tests, test code duplication and high test maintenance cost. And how can we write tests clearly in Behavior-Driven-Development (BDD) style.

1. Tests
Antipatterns
by Maciej Przewoźnik

2. Agenda
Slow Test
Overuse of mocks
Obscure Test

3. Agenda (2)
Manual Test
Fragile Test
Erratic Test
Eager Test

4. Agenda (3)
Verbose Test
Mystery Guest
Test Code Duplication
High Test Maintenance Cost

5. Slow Test

6. Slow Test
Stabilizing a very slow test
Individual slow tests
Executing all tests is too long

7. Slow Test
Stabilizing a very slow test
Individual slow tests
Executing all tests is too long

8. - -

9. Slow Test - stabilizing a very slow
test
Don't steer a rover on Mars from Earth
Execute a test ...
8 minutes later: failure
Fix & rerun
10 minutes later: failure
Fix & rerun
10 minutes later: failure

10. Slow Test - stabilizing a very slow
test
What to do in the meantime?
GTD: don't multitask
CPU cache vs "brain cache"
Context switch cost

11. Slow Test - stabilizing a very slow
test
What to do in the meantime?
Solution: use the fastest machine
Have the machine in the company
Other tests on slower machines

12. Slow Test - stabilizing a very slow
test
What to do in the meantime?
For performance or stress tests with big
amount of data
Solution: use smaller data sets for
stabilization first
Then run with bigger datasets

13. Slow Test - stabilizing a very slow
test
Solution: use REPL environment to
stabilize it faster
Real-eval-print loop
A failure shouldn't invalidate previous results!

14. Slow Test - stabilizing a very slow
test
Solution: stabilize parts separately
Given a test:
Generate a big file.
Consume it.
Don't remove the file if failure occurs in (2).
(to be continued)

15. Slow Test
Stabilizing a very slow test
Individual slow tests
Executing all tests is too long

16. Slow Test - individual tests are
slow
What is a slow test?
a developer gets irritated or bored waiting for
finish
is tempted to get a break, surf the Internet,
chat, or walk around

17. Slow Test - individual tests are
slow
What is a slow test?
Sometimes more than a second, sometimes a
millisecond
Not long individually, but all tests execution is
too long
Or a group of tests is too long

18. Slow Test - individual tests are
slow
Impact:
Developers stop running them after every code
change
Developers wait for a coffee break, lunch or a
meeting to run them
Delayed feedback, loss of "flow"
Decreases productivity and job satisfaction

19. Slow Test - individual tests are
slow
Cause: Slow dependent component (1)
Solution: Fast Fake Objects
Real DB → Fake DB (e. g. In Memory DB)
Real Cloud → In memory cloud
Filesystems: HDD, SSD → RAM disk:
mount ­t tmpfs ­o size=200m tmpfs /mnt/tmp

20. Slow Test - individual tests are
slow
Cause: Slow dependent component (2)
Example: Backup applications
Solution: Recorded test
Record operations done by backup
applications
Warning: Recorded test may be a Fragile Test

21. Slow Test - individual tests are
slow
Cause: Slow environment or layer
Solution: Separate tested logic from
environment:
Hardware/GUI/HTTP layer
Threads, processes
Clouds, etc.
Kernel code

22. Slow Test - individual tests are
slow
Cause: Over-engineered fixture
Long setup of many complicated objects
Even if most of them are not needed
Or using full product in tests
Solution: General Fixture → Minimal
Fixture

23. Slow Test - individual tests are
slow
Cause: Manual tests
Slow by definition
Even slower if repeated
"Please check that the system just works"
before a release
Automated tests may get written anyway later

24. Slow Test - individual tests are
slow
Cause: Manual tests
Often under pressure or under "give-it-to-me-
now driven-development"
No future returns
May be worthless after just a single commit
Lack of tests automation will slow test
development later

25. Slow Test - individual tests are
slow
"Just Say No to More End-to-End Tests"
Testing pyramid
Manual tests
End-to-end tests
Integration tests
Component tests
Fast Unit Tests

26. Slow Test
Stabilizing a very slow test
Individual slow tests
Executing all tests is too long

27. Slow Tests - executing all tests is
too long
In some big IT companies:
build time: one minute
release to production: 8 minutes
not just a release – a release to production!
What problems may it address?

28. Slow Tests - executing all tests is
too long
Impact: Developers context-switching,
worse productivity
tests start ...
15 min: a test fails
fix & rerun
20 min: a test fails

29. Slow Tests - executing all tests is
too long
Impact: Integrating code slower
Bigger release queue
Slower features delivery, missing deadlines

30. Slow Tests - executing all tests is
too long
Impact: Production big fixes are
delayed
A customer requires a fix, we need to test it
Release a quick fix or a well tested fix?

31. Slow Tests - executing all tests is
too long
Cause: Too much overlap between
tests
Test1: [ DATA LOADING ][doSth][check]
Test2: [ DATA LOADING ][doSth][check]
Test3: [ DATA LOADING ][doSth][check]

32. Slow Tests - executing all tests is
too long
Cause: Too much overlap between
tests
Use Shared Fixture?
Shared setup for many tests
Warning: may get closer to Erratic Tests,
High Test Maintenance Cost

33. Slow Tests - executing all tests is
too long
Cause: Too Many Tests
Too many tests run too frequently
Solution: subsets of tests can be run at
the time
All tests within larger period of time
Prerelease tests: all with fast fakes
Postrelease tests: with real objects

34. Slow Tests - executing all tests is
too long
Cause: Too Many Tests
The system is too large
Solution: break it into independent
subsystems

35. Slow Tests - executing all tests is
too long
Cause: Slow Machines
Only one integration server
Too slow integration servers
Insufficient parallelization: look at "top",
"vmstat"

36. Overuse of (strict)
mocks

37. Overuse of mocks
Impact:Tests can be harder to
understand
The extra code detracts from the tester authors
intent

38. Overuse of mocks
Impact: Tests are harder to maintain
Focus: implementation vs public interface
Focus: implementation vs behavior
Extra code for mock behavior needed
Over-specified software
Fragile

39. Overuse of mocks
Impact: Less assurance that the code
is working properly
It's hard to guarantee that the mocks behave as
real implementations
Especially over time

40. Overuse of mocks
When the real object cannot be used:
They are too slow
Their setup is too complex
They charge money, etc.

41. Overuse of mocks
Alternatives:
Fake objects
Hermetic local servers (for cash transactions,
etc.)

42. Consider the
following bad code ...

43. class TeaHouse(object):
def __init__(self, cash_service, tax_service):
self.cash_service = cash_service
self.tax_service = tax_service
self.money_total = 0
def order_cup(self, credit_card):
cup = Tea()
self.cash_service.begin_transaction()
result = self.cash_service.charge(
credit_card, cup.price)
if result:
self.money_total += cup.price
self.cash_service.commit_transaction()
self.tax_service.register_transaction(
cup.price
)
else:
self.cash_service.cancel_transaction()

44. def should_compute_doubled_price_for_two_cups():
cs = mock(CashService)
cs.expect_call("begin_transaction").times(2)
cs.expect_call("charge").times(2).repeatedly(
Return(Cup())
)
cs.expect_call("end_transaction").times(2)
ts.mock(TaxService)
ts.expect_call("register_transaction").times(2)
t = TeaHouse(cs, ts)
c = CreditCard(money=usd(30))
t.set_tea_price(usd(10))
t.order_cup(c)
t.order_cup(c)
assert t.money_total == usd(20)

45. Overuse of mocks
We actually don't care for interaction with cash
service and tax service here

46. Overuse of mocks
Alternative: use a fake
def should_compute_doubled_price_for_two_cups():
t = TeaHouse(
FakeInMemoryCashService(), FakeTaxService()
)
c = CreditCard(money=usd(30))
t.set_tea_price(usd(10))
t.order_cup(c)
t.order_cup(c)
assert t.money_total == usd(20)

47. Overuse of mocks
Alternative: use a stub or a non-strict
mock:
def should_compute_doubled_price_for_two_cups():
t = TeaHouse(
StubCashService(always_ok=True), StubTaxService()
)
c = CreditCard(money=usd(30))
t.set_tea_price(usd(10))
t.order_cup(c)
t.order_cup(c)
assert t.money_total == usd(20)

48. But in this particular
situation, even better
is ...

49. One function, one
responsibility

50. Overuse of mocks
class TeaHouseOrders:
def __init__(self):
# no cash_service and no tax service!
self.orders = []
def order_cup(credit_card):
self.orders.append(TeaOrder())
return
def total_price():
return sum([o.price() for o in self.orders])

51. Overuse of mocks
class TeaHouseCash:
def charge(price):
self.cash_service.begin_transaction()
result = self.cash_service.charge(cc, price)
if result:
self.cash_service.commit_transaction()
else:
self.cancel_transaction()
return self.cups

52. Overuse of mocks
def should_compute_doubled_price_for_two_cups():
t = TeaHouseOrders()
t.set_tea_price(usd(10))
t.order_cup()
t.order_cup()
assert t.total_price() == usd(20)
Neither mocks nor fakes needed!
Important concept: separate description
from execution

53. Obscure Test

54. Obscure Test
Impact:
Harder to understand
Harder to maintain
Developers will not read them as
documentation
May lead to Can result in a Buggy Test, and
then Production Bugs

55. Obscure Test
Good Tests:
Tests should act as documentation
And also a self-verifying executable
specification

56. Obscure Test
Is it a good test?
def should_have_the_same_permissions_after_copy():
fs = FileSystem("r­w")
f = File("a.txt", "rwxrwxrwx")
fso = FileSystemOps(fs)
fso.copy(f, "b.txt")
assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))

57. Obscure Test
Is it a good test?
def should_have_the_same_permissions_after_copy():
fs = FileSystem("r­w", "no­auto", 620, false,
true, true, 8096, 10*1024*1024*1024, 1000000)
f = File("a.txt", "rwxrwxrwx", 64, "xyzxyz",
"2016­12­12", "2016­12­12", "2016­12­12")
fso = FileSystemOps(fs, true, false, 1024, 11)
fso.copy(f, "b.txt")
assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))

58. Obscure Test
Cause: Irrelevant information
which values affect the outcome?
There can be a hundred constants in a test
code

59. Obscure Test
Cause: Too much information
Solution: introduce higher-level test
language
def file_copy_permissions_should_be_as_original():
given_a_filesystem()
given_a_file_with_permissions("rwxrwxrwx")
when_a_file_is_copied()
then_the_new_file_has_permissions("rwxrwxrwx")

60. Obscure Test
Cause: Too much information, Eager
Test
The test verifies too much functionality in a
single
Test Method and therefore is hard to
understand

61. Obscure Test
Cause: Too much information, Eager
Test
def test_my_filesystem():
fs = FileSystem("r­w", "no­auto", 620, false, true,
true, 8096, 10*1024*1024*1024, 1000000)
assert(fs.isReadOnly() == false)
f = File("a.txt", "rwxrwxrwx", 64, "xyzxyz",
"2016­12­12", "2016­12­12", "2016­12­12")
assert(f.getPermissions() == "rwxrwxrwx")
assert(f.getCreationDate() == "2016­12­12")
fso = FileSystemOps(fs, true, false, 1024, 11)
fso.copy(f, "b.txt")
assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))
assert(fso.size("a.txt") == fso.size("b.txt")

62. Obscure Test
Cause: Too much information, Eager
Test
Often a Fragile Test
And a High-Maintenance Test

63. Obscure Test
Cause: Too much information
Not caring about clean code
No refactoring
"These are only tests"
"just do it in-line" mentality

64. Obscure Test
Cause: Too little information – Mystery
Guest
Cause and effect not clear
Something is hidden from the Test Method

65. Obscure Test
Cause: Too little information – Mystery
Guest
Doesn't exist because is copied to root?
def test123():
fs = createFileSystem()
f = new File("a.txt", "rwxrwxrwx", 64*MB, "xyzxyz",
"2016­12­12", "2016­12­12", "2016­12­12"
)
fso = new FileSystemOps(fs, true, false, 1024, 11)
fso.copy(f, "/b.txt")
assert(fso.exists("b.txt") == false)

66. Obscure Test
Cause: Too little information – Mystery
Guest
No, because createFileSystem creates read-only
filesystem:
def createFileSystem():
return new FileSystem(
"r­o", "no­auto", 620, false, true, true,
8096, 10*1024*1024*1024, 1000000
)
)

67. Obscure Test
Cause: Setup-action-verify phases not
clear
def test123():
fs = createFilesystem()
f = createFile("a.txt")
fso = createFso()
f2 = fso.copy(f, "b.txt")
s1 = fso.size("a.txt")
s2 = fso.size("b.txt")
assert(s1 == s2)
Do we test "size" here or "copy"?
Solution: Given-when-then

68. Obscure Test
Cause: Setup-action-verify phases not
clear
Solution: Given-when-then:
given_filesystem()
given_file()
given_file_system_operations()
when_the_file_is_copied()
then_the_new_file_size_is_as_original()

69. Obscure Test
Cause: Overcomplicated Fixture
class TestFSFixture:
def createRealOnlyFS():
def createRWFS():
def createJournalingFS():
def createTransactionalFS():
def createNFS():
def createHDFS():
def createNoAcFs():
def createTwoFilesystems():
def createPerfOptFS():
def createBuggyFS():
def createSlowFS():

70. Obscure Test
Cause: Overcomplicated Fixture
The fixture is too big and too complex
To understand the test, reading the fixture may
be necessary
Solutions:
Divide a General Fixture into smaller Fixtures
Use Minimal Fixture
Use Facades

71. Obscure Test
Cause: Indirect Testing
Testing one component through other
components
Example: Execute a product in debug mode to
see if log rotation works
Root Cause: software not designed for
testability

72. Sources
Gerard Meszaros - xUnit Test Patterns
testing.googleblog.com
Paul Chiusano and Rúnar Bjarnason - Functional
Programming in Scala

73. THE END