We’ve all been there: debugging problems in a test and silently screaming into the dark. Poor test cases can cost you time and effort and degrade the quality of your application or product. Testing is vitally important, but so is having a test suite you can use effectively and can rely on. This session takes you through the top 10 rules for writing effective and reliable test cases. The new kids on the block such as Cloud and Docker and general “infrastructure as code”-style solutions may make you believe that old rules are just old. This presentation will make you think again. Knowledge gained from personal experience is always best. Learn from two old masters how to design great test cases, and maybe you’ll never have to visit the dark side again.
2. #JavaOne @spoole167 @stuartmarks
Steve Poole : IBM
JVM Developer
DevOps practitioner
Developer Advocate
Stuart Marks : Oracle
Principal MTS
Java / OpenJDK Core Libraries
3. #JavaOne @spoole167 @stuartmarks
“I get paid for code that works, not for tests”
How to maximize your effort
and protect your investment
in tests and testing now with
added
Cloud
4. #JavaOne @spoole167 @stuartmarks
1. Think before you act
2. Make your tests understandable
3. Keep your tests “small and simple”
4. Test one thing only
5. Fast tests only
6. Absolute repeatability
7. Independent tests only
8. Provide diagnostic data on failure
9. No hard-coding of your environment
10.No extraneous output
5. #JavaOne @spoole167 @stuartmarks
1. Think before you act
What are
you testing?
Why are you
testing?
Plan
https://www.flickr.com/photos/ayurvedicmedicines/
7. #JavaOne @spoole167 @stuartmarks
2. Make your tests
understandable
Comments
Expected behaviour
Aid debug
Diagnostics
https://www.flickr.com/photos/83633410@N07/
8. #JavaOne @spoole167 @stuartmarks
3. Keep your tests
“small and simple”
Separate test logic / setup
Much easier to debug
Use setup / teardown
https://www.flickr.com/photos/9266144@N02/
9. #JavaOne @spoole167 @stuartmarks
4. Test one thing only
One scenario per test
Enables fast debug
Obvious why test failed
https://www.flickr.com/photos/ryantron/
10. #JavaOne @spoole167 @stuartmarks
5. Fast tests only
Run unit tests often as possible
Maintain quality bar
Quick results
https://www.flickr.com/photos/mcleod/
11. #JavaOne @spoole167 @stuartmarks
6. Absolute repeatability
Non-deterministic
tests are a headache
Fix intermittent tests
immediately
No value, waste of resource
Must trust all tests
https://www.flickr.com/photos/fdecomite/
12. #JavaOne @spoole167 @stuartmarks
7. Independent tests only
Must run in any order
Run subset, faster results
No dependencies
https://www.flickr.com/photos/sheila_sund/
13. #JavaOne @spoole167 @stuartmarks
8. Provide diagnostic
data on failure
Use message in asserts
Make it simple to debug
Reference input data
Record test environment info
https://www.flickr.com/photos/davidbaker/
14. #JavaOne @spoole167 @stuartmarks
9. No hard-coding of
your environment
No ports, IP addresses,
data files, databases
Use config files, system
properties or mock objects
Portable tests
https://www.flickr.com/photos/pug50/
15. #JavaOne @spoole167 @stuartmarks
10. No extraneous output
A passing test is a silent test
Too much output = confusion
Use option, config file to
turn on debug, save output
https://www.flickr.com/photos/3-bs/
16. #JavaOne @spoole167 @stuartmarks
Pirate rules
Guidelines only
Some may seen obvious.
Some may seem pedantic
Until you inherit a testsuite…
Thank you.
https://www.flickr.com/photos/gapic/
Editor's Notes
Questions for the end or on twitter..
Steve:
People say “ I don’t need to write tests my code is good enough.” “I only get paid for code”
Your brain is not big enough to keep even simple code in your head
Good tests are essential – poor tests are worse than useless.
This talk is about showing why you need to write tests and why in the world of cloud and containers its becoming even more important
This talk is about getting the best out of the time you spend writing tests.
Pirate Rules (guidelines only)
Can we agree on rules vs guidelines ?
This list is the top ish(10) from professional testers and developers. Most of our pain has come from from personal experience in breaking these rules..
Our experiences only.
Valueable. Its not exclusive, just our opnion. And we don’t have time for more..
Non of these are unimportant
Steve – intro and substance
---
Your first job is to capture the expected behavior of your system so you can continue to show you have not broken behavior.
Old platitudes: “You cannot test quality in”
But without tests you cannot assert ‘quality’
you only have so much time to capture your systems behavior in tests - don’t waste it
We’ve all seen large test suites that mostly don’t test anything useful : simple getters and setters please?
What happens if your tests are not understandable?
Should tests be easier to understand than the code their testing?
Tests should be coded and commented well. A test suite is the inverse of the code. Its codifing expected behaviour so write it down what your think the behavior is. Don’t assume your testcode is perfect – write comments to explain your intent
When an test fails you don’t want to be spending time figuring out if it’s the test. How would you report a bug if you can’t work out what the test is testing?
They say debugging is twice as hard as writing code - its easy to write tests that are much harder to understand than the code their testing. Debugging failing tests like that takes more brain power than anyone possesses.
Be nice to you future self trying to figure out whats wrong when your trying to deploy code at midnight
Steve: tests should have a well defined setup
Can you find the actual line of testing in a large test suite
Why are we saying this - how often have you had to step a 1000times
Or debug a nested loop inside a nested loop?
Or inserted code to allow you to set a break point.
If you have to hack the test its too big
clean separation between getting to to the actual code
Number of times a test has multiple asserts to ’help’ you work out what was wrong before running the test. But pre-failures mean your not getting close to your actual code.
Let other tests find those problems. - add help info into the logs not as asserts
Tests that test multiple things complicate how your measure progress. Soon you find you have blocking defects that are artificial – they’re not decessarly important to your users but they stop lots of tests running.
And of course its so much easier to run simple tests and understand the point of the test if its only got one assert
Steve: Testing is domain / time context
We’d all like all our tests to run all the time. We can’t do that so when backing off chose wisely.
Running the whole testsuite is useful. Long tests encourage you to not run tests. You let more bugs through
Even if part of a CI run..
Long running test suites are evil. They encourage corner cutting. The make you think about not running tests…
They make your developers less productive. I’ll just submit this one line code change for building over night…
Or I’ll have my private set of tests - I’ll just run the tests that I think are the right ones.
Things fail later (and cost more) when you could have got the developer to run the test – if only it ran faster..
Q: “Steve how does this work in a cloud environment?”
“Everything fails all the time in the cloud”
Cloud: reliable system built from unreliable components. This manifests itself in intermittent test failures.
How to distinguish "valid" failure (failure that can be handled by the larger system) vs an "invalid" failure? Do we care?
"Offensive programming" – deliberate injection of failures. Useful for system tests.
---
Random without a seed. Sleeping to wait for stuff to happen.
Repeatability with timeouts is difficult.
“Chaos Monkeys”
Steve: War story - junit test ordering. Broke then jit… Paralleise using clusters Exposed unexpected test interdependences… State, setup database , add records etc, then run tests. Later tests assumed state based on those .
I want to one day run one of these tests singualry for debug purposes. I don’t want to run 50 others first!
Run your tests in random order?
Steve : War story FFDC (any part of the code could have raised the error. Customer wanted it fixed…!
Clear simple information at failure to find code and culprit data
Note that you need to test that failure generation too.
Log(foo)
Log(foo.bar)
If(foo==null) …
works for me syndrome
+ Cloud: Remote tests are difficult remove env is always different and un reproducable locally..
Steve: War story “the App server test env and 1yr”
+ Cloud needs you not to do this.. Every instance could be on a different box, somewhere else in the world
Reduce opportunities to be effected by using mock objects.
Q: You like verbose output don’t you Steve?
Steve: if you have too much output - > you invest in systems to analyse the log and look for the real ‘truth’
Steve: Just like the movie says – these are all guidelines. We think you should think before you ignore the rules. Wait until you inherit your first test suite..
Q: Stuart –any closing thoughts?