We've all been there: debugging problems in a test case and silently screaming into the dark. (Sometimes not even silently.) Poor test case design can cost you significant time and effort let alone impact the quality of your application or product. Testing is vitally important, but so is having a test suite you can use effectively and can rely on. This session will take you through the top ten rules for writing effective and reliable testcases.
The new kids on the block such as Cloud or Docker and general "Infrastructure as Code" style solutions may make you believe old rules are just old. This talk will make you think again. Knowledge gained from personal experience is always best. Learn from these old masters how to design great test cases and maybe you'll never have to visit the dark side again.
2. #DevoxxUS @spoole167 @stuartmarks
Steve Poole : IBM
JVM Developer
DevOps practitioner
Developer Advocate
Stuart Marks : Oracle
Principal MTS
Java / OpenJDK Core Libraries
3. #DevoxxUS @spoole167 @stuartmarks
âI get paid for code that works, not for testsâ
How to maximize your effort
and protect your investment
in tests and testing now with
added
Cloud
4. #DevoxxUS @spoole167 @stuartmarks
1. Think before you act
2. Make your tests understandable
3. Keep your tests âsmall and simpleâ
4. Test one thing only
5. Fast tests only
6. Absolute repeatability
7. Independent tests only
8. Provide diagnostic data on failure
9. No hard-coding of your environment
10.No extraneous output
5. #DevoxxUS @spoole167 @stuartmarks
1. Think before you act
What are
you testing?
Why are you
testing?
Planhttps://www.flickr.com/photos/ayurvedicmedicines/
6. #DevoxxUS @spoole167 @stuartmarks
2. Make your tests
understandable
Comments
Expected behaviour
Aid debug
Diagnostics
https://www.flickr.com/photos/83633410@N07/
7. #DevoxxUS @spoole167 @stuartmarks
3. Keep your tests
âsmall and simpleâ
Separate test logic / setup
Much easier to debug
Use setup / teardown
https://www.flickr.com/photos/9266144@N02/
8. #DevoxxUS @spoole167 @stuartmarks
4. Test one thing only
One scenario per test
Enables fast debug
Obvious why test failed
https://www.flickr.com/photos/ryantron/
9. #DevoxxUS @spoole167 @stuartmarks
5. Fast tests only
Run unit tests often as possible
Maintain quality bar
Quick results
https://www.flickr.com/photos/mcleod/
10. #DevoxxUS @spoole167 @stuartmarks
6. Absolute repeatability
Non-deterministic
tests are a headache
Fix intermittent tests
immediately
No value, waste of resource
Must trust all tests
https://www.flickr.com/photos/fdecomite/
11. #DevoxxUS @spoole167 @stuartmarks
7. Independent tests only
Must run in any order
Run subset, faster results
No dependencies
https://www.flickr.com/photos/sheila_sund/
12. #DevoxxUS @spoole167 @stuartmarks
8. Provide diagnostic
data on failure
Use message in asserts
Make it simple to debug
Reference input data
Record test environment info
https://www.flickr.com/photos/davidbaker/
13. #DevoxxUS @spoole167 @stuartmarks
9. No hard-coding of
your environment
No ports, IP addresses,
data files, databases
Use config files, system
properties or mock objects
Portable tests
https://www.flickr.com/photos/pug50/
14. #DevoxxUS @spoole167 @stuartmarks
10. No extraneous output
A passing test is a silent test
Too much output = confusion
Use option, config file to
turn on debug, save output
https://www.flickr.com/photos/3-bs/
15. #DevoxxUS @spoole167 @stuartmarks
Pirate rules
Guidelines only
Some may seen obvious.
Some may seem pedantic
Until you inherit a testsuiteâŠ
Thank you.
https://www.flickr.com/photos/gapic/
Steve: To say why we are doing. Intro
Why do we write tests. I donât need to my code is good enough. I only get paid for code.
Stuart: Platitudes -> things are âbetterâ if we write tests. With not much complexity there was more knowledge in the code and tests than in my head. 200 lines of code
Your brain is not big enough to keep even a simple code in your head.
Pirate Rules (guidelines only)
Can we agree on rules vs guidelines ?
This list is the top ish(10) from professional testers and developers. Most of our pain has come from from personal experience in breaking these rules..
Our experiences only.
Valueable. Its not exclusive, just our opnion. And we donât have time for more..
Non of these are unimportant
Steve â intro and substance
---
Your first job is to capture the expected behaviour of your system so you can continue to show you have not broken bevaiour..
You cannot test quality in
But without tests you cannot assert âqualityâ
you only have some much time to capture your systems behavior in tests - donât waste it
--
Stuart â your opinion âon testing in relation to qualityâ
How do you measure this?
Test coverage -> Donât just test getters and setters
Its not valueless but unless youâre looking at conformance donât get too bent out of shape about coverage.
Improving Test coveage without bugs is not necessary a good thing.
JDK has jtreg. SQE has separate test suite (closed for no good reason) too much effort to open source the framework and tests.
Is adding more tests to extend coverage a good idea? Not necessary - time spend on arguing how the code is supposed to work Do new tests have bugs as often as code has bugs?
Stuart: tests are an equally important part of a code base as the production code.
Tests need to be maintained and enhanced alongside the production code, not left as an afterthought.
Q: What happens if your tests are not understandable?
Should tests be easier to understand than the code their testing?
Tests should be coded and commented well. A test suite is the inverse of the code. Its codifing expected behaviour so write it down what your think the behavior is.
When and a test fails you donât want to be spending time figuring out if itâs the test. How would you report a bug if you canât work out what the test.
Steve: tests should have a well defined setup
Can you find the actual line of testing in a large test suite
Why are we saying this - how often have you had to step a 1000times
Or debug a nested loop inside a nested loop?
Or inserted code to allow you to set a break point.
If you have to hack the test its too big
clean separate between getting to to the actual cost.
Q: âyou are in full agreement?â
Stuart: setup â operation â verification â teardown
Donât do multiple operations; they can spoil the fixture for subsequent op/verify steps.
Separate into multiple tests.
Q: âdo you agree?â
Steve: Testing is domain / time context
Weâd all like all our tests to run all the time. We canât do that so when backing off chose wisely.
Running the whole testsuite is useful. Long tests encourage you to not run tests. You let more bugs through
Even if part of a CI run..
Long running test suites are evil. They encourage corner cutting. The make you think about not running testsâŠ
Q: âStuart â what happens if your tests are not fast?â
War story about test suite run monthly
Stuart: An ideal to strive for but possibly impossible to achieve in practice. Nonetheless v important!
(ask for show of hands)
Noise level of intermittent test failures makes result analysis harder, masks real failures.
Q: âSteve how does this work in a cloud environment?â
Cloud: reliable system built from unreliable components. This manifests itself in intermittent test failures.
How to distinguish "valid" failure (failure that can be handled by the larger system) vs an "invalid" failure? Do we care?
"Offensive programming" â deliberate injection of failures. Useful for system tests.
Random without a seed. Sleeping to wait for stuff to happen.
Repeatability with timeouts is difficult.
âChaos Monkeysâ
Steve: War story - junit test ordering. Exposed unexpected test interdependencesâŠ
I want to one day run one of these tests singualry for debug purposes. I donât want to run 50 others first!
Run your tests in random order?
Q: â has this ever happened to you?
Stuart: Yes. Don't write tests that have persistent side effects. CORBA test war story.
Stuart: Reduce debugging time. Exception stack traces usually help here, but not always.
Consider: asserting that right exception was thrown at the right time.
Timeout war story.
Q: whatâs your experience with this?
Steve : War story FFDC
works for me syndrome
+ Cloud: Remote tests are difficult remove env is always different and un reproducable locally..
Steve: War story âthe App server test env and 1yrâ
+ Cloud needs you not to do this.. Every instance could be on a different box, somewhere else in the world
Reduce opportunities to be effected by using mock objects.
Q: âcan you think of any reasons why you would want to hard code your environment?â
Stuart: Barely. Hard to avoid hard-coding 100%. Example: test that the default port is 80.
Stuart: Essential if you have 10,000 unit tests. Even saying âpassedâ can generate too much output.
But for complex system tests, esp with timeouts, sometimes verbose logging is necessary.
Q: You like verbose output donât you Steve?
Steve: if you have too much output - > you invest in systems to analyse the log and look for the real âtruthâ
Steve: Just like the movie says â these are all guidelines. We think you should think before you ignore the rules. Wait until you inherit your first test suite..
Q: Stuart âany closing thoughts?