1. Bad Tests, Good Tests
Tomek Kaczanowski
http://twitter.com/#!/devops_borat
2. Tomek Kaczanowski
• Developer
• Team lead
• Blogger
• http://kaczanowscy.pl/tomek
• Book author
• http://practicalunittesting.com
• Working at CodeWise (Krakow, Poland)
• ...we are hiring, wanna join us?
3. Why bother with tests?
• System works as expected
• Changes do not hurt
• Documentation
4. Tests help to achieve quality
Not sure when I saw this picture – probably
in GOOS?
5. What happens if we do it wrong?
• Angry clients
• Depressed developers
http://www.joshcanhelp.com
6. When I started out with unit tests, I was
enthralled with the promise of ease and
security that they would bring to my
projects. In practice, however, the
theory of sustainable software through
unit tests started to break down. This
difficulty continued to build up, until I
finally threw my head back in anger and
declared that "Unit Tests have become
more trouble than they are worth."
Llewellyn Falco and Michael Kennedy, Develop Mentor August 09
10. Before we begin
• All of the examples are real but were:
• obfuscated
• to protect the innocents :)
• truncated
• imagine much more complex domain objects
• Asking questions is allowed
• ...but being smarter than me is not ;)
11. We don't need no stinkin' asserts!
public void testAddChunks() {
System.out.println("*************************************");
System.out.println("testAddChunks() ... ");
ChunkMap cm = new ChunkMap(3);
cm.addChunk(new Chunk("chunk"));
List testList = cm.getChunks("chunk",null);
if (testList.isEmpty())
fail("there should be at least one list!");
Chunk chunk = cm.getActualChunk("chunk",null);
if (chunk.getElements().isEmpty())
fail("there should be at least one element!");
if (cm.getFinalChunkNr() != 1)
fail("there should be at least one chunk!");
// iterate actual chunk
for (Iterator it = chunk.getElements().iterator();
it.hasNext();) {
Element element = (Element) it.next();
System.out.println("Element: " + element);
}
showChunks(cm);
System.out.println("testAddChunks() OK ");
} Courtesy of @bocytko
12. Success is not an option...
/**
* Method testFailure.
*/
public void testFailure() {
try {
Message message = new Message(null,true);
fail();
} catch(Exception ex) {
ExceptionHandler.log(ExceptionLevel.ANY,ex);
fail();
}
}
Courtesy of @bocytko
13. What has happened? Well, it failed...
public void testSimple() {
IData data = null;
IFormat format = null;
LinkedList<String> attr = new LinkedList<String>();
attr.add("A");
attr.add("B");
try {
format = new SimpleFormat("A");
data.setAmount(Amount.TEN);
data.setAttributes(attr);
IResult result = format.execute();
System.out.println(result.size());
Iterator iter = result.iterator();
while (iter.hasNext()) {
IResult r = (IResult) iter.next();
System.out.println(r.getMessage());
...
}
catch (Exception e) {
fail();
}
}
Courtesy of @bocytko
14. What has happened? Well, it failed...
public void testSimple() {
IData data = null;
IFormat format = null;
LinkedList<String> attr = new LinkedList<String>();
attr.add("A");
attr.add("B");data is null - ready or not,
NPE is coming!
try {
format = new SimpleFormat("A");
data.setAmount(Amount.TEN);
data.setAttributes(attr);
IResult result = format.execute();
System.out.println(result.size());
Iterator iter = result.iterator();
while (iter.hasNext()) {
IResult r = (IResult) iter.next();
System.out.println(r.getMessage());
...
}
catch (Exception e) {
fail();
}
}
Courtesy of @bocytko
15. No smoke without tests
class SystemAdminSmokeTest extends GroovyTestCase {
void testSmoke() {
def ds = new org.h2.jdbcx.JdbcDataSource(
URL: 'jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;MODE=Oracle',
user: 'sa', password: '')
def jpaProperties = new Properties()
jpaProperties.setProperty(
'hibernate.cache.use_second_level_cache', 'false')
jpaProperties.setProperty(
'hibernate.cache.use_query_cache', 'false')
def emf = new LocalContainerEntityManagerFactoryBean(
dataSource: ds, persistenceUnitName: 'my-domain',
jpaVendorAdapter: new HibernateJpaVendorAdapter(
database: Database.H2, showSql: true,
generateDdl: true), jpaProperties: jpaProperties)
…some more code below
}
16. No smoke without tests
class SystemAdminSmokeTest extends GroovyTestCase {
void testSmoke() {
// do not remove below code
// def ds = new org.h2.jdbcx.JdbcDataSource(
// URL: 'jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;MODE=Oracle',
// user: 'sa', password: '')
//
// def jpaProperties = new Properties()
// jpaProperties.setProperty(
// 'hibernate.cache.use_second_level_cache', 'false')
// jpaProperties.setProperty(
// 'hibernate.cache.use_query_cache', 'false')
//
// def emf = new LocalContainerEntityManagerFactoryBean(
// dataSource: ds, persistenceUnitName: 'my-domain',
// jpaVendorAdapter: new HibernateJpaVendorAdapter(
// database: Database.H2, showSql: true,
// generateDdl: true), jpaProperties: jpaProperties)
…some more code below, all commented out :(
}
17. Let's follow the leader!
@Test
public class ExampleTest {
public void testExample() {
assertTrue(true);
}
}
18. Uh-oh, I feel lonely...
@Test
public class ExampleTest {
public void testExample() {
assertTrue(true);
}
}
19. Conclusions
• Automation!
• Running
• Verification
• Do not live with broken window
• And remember there is no one else to fix them but
you!
• It is a full time job!
• You should be informed why your test failed
• Master your tools
• …at least learn the basics!
20. Use of the real objects obscures the test
@Test
public void shouldGetTrafficTrend() {
//given
TrafficTrendProvider trafficTrendProvider
= mock(TrafficTrendProvider.class);
Report report = new Report(null, "", 1, 2, 3,
BigDecimal.ONE, BigDecimal.ONE, 1);
TrafficTrend trafficTrend = new TrafficTrend(report, report,
new Date(), new Date(), new Date(), new Date());
given(trafficTrendProvider.getTrafficTrend())
.willReturn(trafficTrend);
TrafficService service
= new TrafficService(trafficTrendProvider);
//when
TrafficTrend result = service.getTrafficTrend();
//then
assertThat(result).isEqualTo(trafficTrend);
}
21. Use of the real objects obscures the test
@Test
public void shouldGetTrafficTrend() {
//given
TrafficTrendProvider trafficTrendProvider
= mock(TrafficTrendProvider.class);
TrafficTrend trafficTrend = mock(TrafficTrend.class);
given(trafficTrendProvider.getTrafficTrend())
.willReturn(trafficTrend);
TrafficService service
= new TrafficService(trafficTrendProvider);
//when
TrafficTrend result = service.getTrafficTrend();
//then
assertThat(result).isEqualTo(trafficTrend);
}
23. ModelAndView from
Mock'em All! SpringMVC – a mere
container for data, without
@Test any behaviour
public void shouldAddTimeZoneToModelAndView() {
//given
UserFacade userFacade = mock(UserFacade.class);
ModelAndView modelAndView = mock(ModelAndView.class);
given(userFacade.getTimezone()).willReturn("timezone X");
//when
new UserDataInterceptor(userFacade)
.postHandle(null, null, null, modelAndView);
//then
verify(modelAndView).addObject("timezone", "timezone X");
}
24. Use the front door
@Test
public void shouldAddTimeZoneToModelAndView() {
//given
UserFacade userFacade = mock(UserFacade.class);
ModelAndView modelAndView = new ModelAndView();
given(userFacade.getTimezone()).willReturn("timezone X");
//when
new UserDataInterceptor(userFacade)
.postHandle(null, null, null, modelAndView);
a pseudocode but that is
what we mean
//then
assertThat(modelAndView).contains("timezone", "timezone X");
}
25. Mock'em All!
Public class Util {
public String getUrl(User user, String timestamp) {
String name=user.getFullName();
String url=baseUrl
+"name="+URLEncoder.encode(name, "UTF-8")
+"×tamp="+timestamp; Developer wants to check
return url;
}
whether timestamp is added
to the URL when this method
public String getUrl(User user) { is used
Date date=new Date();
Long time= date.getTime()/1000; //convert ms to seconds
String timestamp=time.toString();
return getUrl(user, timestamp);
}
}
26. Mock'em All!
Public class Util {
public String getUrl(User user, String timestamp) {
String name=user.getFullName();
String url=baseUrl
+"name="+URLEncoder.encode(name, "UTF-8")
+"×tamp="+timestamp;
return url;
}
public String getUrl(User user) {
Date date=new Date();
Long time= date.getTime()/1000; //convert ms to seconds
String timestamp=time.toString();
return getUrl(user, timestamp);
} @Test
} public void shouldUseTimestampMethod() {
//given
Util util = new Util();
Util spyUtil = spy(util);
//when
spyUtil.getUrl(user);
//then
verify(spyUtil).getUrl(eq(user), anyString());
}
27. Dependency injection
Use the front door will save us
@Test
public void shouldAddTimestampToGeneratedUrl() {
//given
TimeProvider timeProvider = mock(TimeProvider.class);
Util util = new Util(timeProvider);
when(timeProvider.getTime()).thenReturn("12345");
util.set(timeProvider);
//when
String url = util.getUrl(user);
//then
assertThat(url).contains("timestamp=12345");
}
33. One feature at a time
@Test
public void shouldRedirectToSigninPageWhenRegistrationSuceeded () {
...
}
@Test
public void shouldNotifyAboutNewUserRegistration() {
...
}
Hint: forget about methods
40. Do not make me learn the API!
server = new MockServer(responseMap, true,
new URL(SERVER_ROOT).getPort(), false);
41. Do not make me learn the API!
server = new MockServer(responseMap, true,
new URL(SERVER_ROOT).getPort(), false);
private static final boolean RESPONSE_IS_A_FILE = true;
private static final boolean NO_SSL = false;
server = new MockServer(responseMap, RESPONSE_IS_A_FILE,
new URL(SERVER_ROOT).getPort(), NO_SSL);
42. Do not make me learn the API!
server = new MockServer(responseMap, true,
new URL(SERVER_ROOT).getPort(), false);
server = new MockServerBuilder()
.withResponse(responseMap)
.withResponseType(FILE)
.withUrl(SERVER_ROOT)
.withoutSsl().create();
server = MockServerBuilder
.createNoSSLFileServer(responseMap, SERVER_ROOT);
44. Test methods names are important
• When test fails
• Relation to focused tests
45. Test methods names are important
@Test
public void testOperation() {
configureRequest("/validate")
rc = new RequestContext(parser, request)
assert rc.getConnector() == null
assert rc.getOperation().equals("validate")
}
47. “should” is better than “test”
• Starting test method names
with “should” steers you in
the right direction.
http://jochopra.blogspot.com/
• “test” prefix makes your test
method a limitless bag
where you throw everything
worth testing
http://www.greenerideal.com/
48. Test methods names are important
@Test
public void testQuery(){
when(q.getResultList()).thenReturn(null);
assertNull(dao.findByQuery(Transaction.class, q, false));
assertNull(dao.findByQuery(Operator.class, q, false));
assertNull(dao.findByQuery(null, null, false));
List result = new LinkedList();
when(q.getResultList()).thenReturn(result);
assertEquals(dao.findByQuery(Transaction.class, q, false), result);
assertEquals(dao.findByQuery(Operator.class, q, false), result);
assertEquals(dao.findByQuery(null, null, false), null);
when(q.getSingleResult()).thenReturn(null);
assertEquals(dao.findByQuery(Transaction.class, q, true).size(), 0);
assertEquals(dao.findByQuery(Operator.class, q, true).size(), 0);
assertEquals(dao.findByQuery(null, null, true), null);
when(q.getSingleResult()).thenReturn(t);
assertSame(dao.findByQuery(Transaction.class, q, true).get(0), t);
when(q.getSingleResult()).thenReturn(o);
assertSame(dao.findByQuery(Operator.class, q, true).get(0), o);
when(q.getSingleResult()).thenReturn(null);
assertSame(dao.findByQuery(null, null, true), null);
}
49. Test methods names are important
@Test
public void shouldReturnNullListWhenDaoReturnsNull {
when(q.getResultList()).thenReturn(null);
assertNull(dao.findByQuery(Transaction.class, q, false));
assertNull(dao.findByQuery(Operator.class, q, false));
assertNull(dao.findByQuery(null, null, false));
}
public void shouldReturnEmptyListWhenDaoReturnsIt {
List result = new LinkedList();
when(q.getResultList()).thenReturn(result);
assertEquals(dao.findByQuery(Transaction.class, q, false), result);
assertEquals(dao.findByQuery(Operator.class, q, false), result);
assertEquals(dao.findByQuery(null, null, false), null);
}
public void shouldReturnNullSingleResultWhenDaoReturnsNull {
when(q.getSingleResult()).thenReturn(null);
assertEquals(dao.findByQuery(Transaction.class, q, true).size(), 0);
assertEquals(dao.findByQuery(Operator.class, q, true).size(), 0);
assertEquals(dao.findByQuery(null, null, true), null);
}
public void shouldReturnSingleResultReturnedByDao {
when(q.getSingleResult()).thenReturn(t);
assertSame(dao.findByQuery(Transaction.class, q, true).get(0), t);
when(q.getSingleResult()).thenReturn(o);
assertSame(dao.findByQuery(Operator.class, q, true).get(0), o);
when(q.getSingleResult()).thenReturn(null);
assertSame(dao.findByQuery(null, null, true), null);
}
50. Assertion part is freaking huge
public void shouldPreDeployApplication() {
// given
Artifact artifact = mock(Artifact.class);
when(artifact.getFileName()).thenReturn("war-artifact-2.0.war");
ServerConfiguration config
= new ServerConfiguration(ADDRESS, USER, KEY_FILE, TOMCAT_PATH, TEMP_PATH);
Tomcat tomcat = new Tomcat(HTTP_TOMCAT_URL, config);
String destDir = new File(".").getCanonicalPath() + SLASH + "target" + SLASH;
new File(destDir).mkdirs();
// when
tomcat.preDeploy(artifact, new FakeWar(WAR_FILE_LENGTH));
//then
JSch jsch = new JSch();
jsch.addIdentity(KEY_FILE);
Session session = jsch.getSession(USER, ADDRESS, 22);
session.setConfig("StrictHostKeyChecking", "no");
session.connect();
Channel channel = session.openChannel("sftp");
session.setServerAliveInterval(92000);
channel.connect();
ChannelSftp sftpChannel = (ChannelSftp) channel;
sftpChannel.get(TEMP_PATH + SLASH + artifact.getFileName(), destDir);
sftpChannel.exit();
session.disconnect();
File downloadedFile = new File(destDir, artifact.getFileName());
assertThat(downloadedFile).exists().hasSize(WAR_FILE_LENGTH);
}
51. Just say it
public void shouldPreDeployApplication() {
// given
Artifact artifact = mock(Artifact.class);
when(artifact.getFileName())
.thenReturn(ARTIFACT_FILE_NAME);
ServerConfiguration config
= new ServerConfiguration(ADDRESS, USER,
KEY_FILE, TOMCAT_PATH, TEMP_PATH);
Tomcat tomcat = new Tomcat(HTTP_TOMCAT_URL, config);
// when
tomcat.preDeploy(artifact, new FakeWar(WAR_FILE_LENGTH));
// then
SSHServerAssert.assertThat(ARTIFACT_FILE_NAME)
.existsOnServer(tomcat).hasSize(WAR_FILE_LENGTH);
}
52. Asserting using private methods
@Test
public void testChargeInRetryingState() throws Exception {
// given
TxDTO request = createTxDTO(RequestType.CHARGE);
AndroidTransaction androidTransaction = ...
// when
final TxDTO txDTO = processor.processRequest(request);
// then
List<AndroidTransactionStep> steps
= new ArrayList<>(androidTransaction.getSteps());
AndroidTransactionStep lastStep = steps.get(steps.size() - 1);
assertEquals(lastStep.getTransactionState(), CHARGED_PENDING);
assertEquals(txDTO.getResultCode(), CHARGED);
}
53. Asserting using private methods
@Test
public void testChargeInRetryingState() throws Exception {
// given
TxDTO request = createTxDTO(RequestType.CHARGE);
AndroidTransaction androidTransaction = ...
// when
final TxDTO txDTO = processor.processRequest(request);
// then
assertState(request, androidTransaction,
CHARGED, CHARGE_PENDING, AS_ANDROID_TX_STATE,
ClientMessage.SUCCESS, ResultCode.SUCCESS);
}
62. Awaitility
@Test
public void updatesCustomerStatus() throws Exception {
// Publish an asynchronous event:
publishEvent(updateCustomerStatusEvent);
// Awaitility lets you wait until
// the asynchronous operation completes:
await().atMost(5, SECONDS)
.until(costumerStatusIsUpdated());
...
}
http://code.google.com/p/awaitility/
63. What do you really want to test?
@Test
public void shouldAddAUser() {
User user = new User();
userService.save(user);
assertEquals(dao.getNbOfUsers(), 1);
}
64. You wanted to see that the number increased
@Test
public void shouldAddAUser() {
int nb = dao.getNbOfUsers();
User user = new User();
userService.save(user);
assertEquals(dao.getNbOfUsers(), nb + 1);
}
Because:
1) This is closer to what you wanted to test
2) There is no assumption about the database “users” table being empty
65. The dream of stronger, random-powered tests
public void myTest() {
SomeObject obj = new SomeObject(
randomName(), randomValue(), ....);
// testing of obj here
}
Does it make your test stronger?
66. The dream of stronger, random-powered tests
public void myTest() {
SomeObject obj = new SomeObject(
randomName(), randomValue(), ....);
// testing of obj here
}
Does it make your test stronger?
...or does it only bring confusion?
Test failed
Expected
SomeObject(„a”, „b”, ....)
but got
SomeObject(„*&O*$NdlF”, „#idSLNF”, ....)
67. Random done wrong
public void myTest() {
SomeObject obj = new SomeObject(
a, b, c, productCode());
// testing of obj here
}
private String productCode(){
String[] codes = {"Code A", "Code B",
"Code C", "Code D"};
int index = rand.nextInt(codes.length);
return codes[index];
}
68. Asking for troubles...
LoggingPropertyConfigurator configurator = mock(...);
BaseServletContextListener baseServletContextListener =
= new BaseServletContextListener(configurator)
@Test public void shouldLoadConfigProperties() { Should load some
baseServletContextListener.contextInitialized();default config
verify(configurator).configure(any(Properties.class));
}
@Test(expected = LoggingInitialisationException.class)
Should load this
public void shouldThrowExceptionIfCantLoadConfiguration() {
specific file
System.setProperty("logConfig", "nonExistingFile");
baseServletContextListener.contextInitialized();
}
69. Asking for troubles...
LoggingPropertyConfigurator configurator = mock(...);
BaseServletContextListener baseServletContextListener =
= new BaseServletContextListener(configurator)
@Test public void shouldLoadConfigProperties() {
baseServletContextListener.contextInitialized();
verify(configurator).configure(any(Properties.class));
}
@Test(expected = LoggingInitialisationException.class)
public void shouldThrowExceptionIfCantLoadConfiguration() {
System.setProperty("logConfig", "nonExistingFile");
baseServletContextListener.contextInitialized();
}
@Before
public void cleanSystemProperties() {
...
}
70. Test-last? No!
• makes people not write tests at all
• makes people do only happy path testing
• tests reflect the implementation
71. Always TDD?
For six or eight hours spread over the next few weeks I
struggled to get the first test written and running.
Writing tests for Eclipse plug-ins is not trivial, so it’s not
surprising I had some trouble. [...] In six or eight hours
of solid programming time, I can still make significant
progress. If I’d just written some stuff and verified it by
hand, I would probably have the final answer to whether
my idea is actually worth money by now. Instead, all I
have is a complicated test that doesn’t work, a pile
of frustration, eight fewer hours in my life, and the
motivation to write another essay.
Kent Beck, Just Ship, Baby
72. There is so much more to discuss…
• Integration / end-to-end tests which are not parametrized
(so they all try to set up jetty on port 8080),
• Tests which should be really unit, but use Spring context
to create objects,
• Tests with a lot of dependencies between them (a
nightmare to maintain!),
• Tests which are overspecified and will fail whenever you
touch the production code,
• Tests with monstrous objects-creation code,
• Tests which run slow,
• Tests which try to cover the deficiencies of production
code and end up being a total mess,
• Tests which verify methods instead of verifying
responsibilities of a class,
• Happy path tests,
• etc., etc.
73. Treat tests as the first class citizens
• do it everyday or forget about it • make tests readable using matchers,
• use the right tool for the job builders and good names
• and learn to use it! • test behaviour not methods
• do not live with broken windows • be pragmatic about the tests you write
• respect KISS, SRP, DRY (?) • TDD always?
• write good code, and you will also write • what is the best way to test it?
good tests unit/integration/end-to-end ?
• or rather write good tests and you • automate!
will get good code for free • always concentrate on what is worth
• code review your tests testing
• do more than happy path testing • ask yourself questions like: 'is it
really important that X should send
• do not make the reader learn the API,
message Y to Z?'
make it obvious
• use the front door – state testing before
• bad names lead to bad tests
interaction testing (mocks)
74. Thank you!
You can learn more about writing
high quality tests by reading my
book – „Practical Unit Testing
with TestNG and Mockito”.
You can also participate in
writing of my
new (free!) e-book devoted to
bad and good tests.
Hinweis der Redaktion
Not really bad but imperfect
No corporate bullshit of any kind – it is just us and the code
This is not a talk about the pros and cons of writing tests so lets only mention it. DESIGN
Many big shops rely solely on automated tests, there are new job positions for people who specialize in this area. We will rather concentrate on the internal quality.
With the power there always comes responsibility.
This is an excerpt from the article where these guys they describe their journey towards good tests. And they tell a story how it all started with a glory, than there was this crash, so they stopped writing tests, then again it occured that without tests they are not progressing very well, because of all regression bugs. And the conclusion is *we should treat our tests as the first class citizens*. And in this presentation I would like to take a look at various tests and try to treat them a little better than their original authors did.
We write more test code than production code. Let us not waste the time we spent on writing them!
Real in this sense that they were written by people for real (production code, open-source projects, job interviews). We will start with some really ugly tests.
There are may things to admire here – e.g. use of Syso, lack of assertions (replaced with if’s and fail), verification by looking
I love everything about it. From the informative javadoc to the great logic.
Love the method name. What is really cool is the catch-all
If you look closer, you will see that it fails, indeed
Sad story about the project. Do not live with broken windows.
Sad story about the project. Do not live with broken windows.
Another sad story of the project - maybe no management support, lack of skills, lack of time? It falls into the same category as with autogenerated tests presented before. |We started and it will somehow work. Similarly when I joined one project, I noticed that some jobs on CI server were not run for a long time. So I sat down and run all of them one by one. Like ~50% of them failed, for various reasons. So now we move to some more subtle bugs. The test we will see from now on, they all work, and in general they fulfill the first requirement – that is they test something
Now to the mocks.
The major issue with this test is that it creates real objects which are irrelevant to the tested scenario! This is bad because the test becomes very fragile this way. Any change in constructor of the Report or TrafficTrend classes and the tests needs to be updated. Another downside of this test is that it distracts the reader from the testing scenario by providing too many details which are not important at all. For example, what is "" parameter and does it matter that the there are four identical dates passed to the second constructor?
The major issue with this test is that it creates real objects which are irrelevant to the tested scenario! This is bad because the test becomes very fragile this way. Any change in constructor of the Report or TrafficTrend classes and the tests needs to be updated. Another downside of this test is that it distracts the reader from the testing scenario by providing too many details which are not important at all. For example, what is "" parameter and does it matter that the there are four identical dates passed to the second constructor?
You should rarely (if ever) mock such a simple entities! If it is not a service (if it does not offer any valuable responsibility) then as a rule of thumb do not mock it!
You should rarely (if ever) mock such a simple entities! If it is not a service (if it does not offer any valuable responsibility) then as a rule of thumb do not mock it! Everytime you verify you take a look into an object and check what it is doing internally. This is anti-OO so do not do that without a good reason.
Simply create such a simple entity. Pseudocode, but apart from this, a very nice state test! State test does not break objects integrity, interactions test do.
Tests are often written poorly because of the bad code. A classic example, one that I see very often, is related to time. Ok, so we have these two methods. The first one is pretty nice. It takes two parameters and constructs an URL using them. Nothing fancy. In fact there were much more parameters added to the URL. The second one is more tricky. It takes only one parameter and constructs the second one using new Date(), then it invokes the first method. Please note that both methods return an URL (a String), so there is something we could test. Question: how can I test this method and make sure it returns url with timestamp?
Test method name makes it clear that this test is very much related to this concrete implementation of class. Spying it is rarely used and shouldn't be without a good reason. Verification of interactions instead of assertion on result.
The solution is to refactor the production code. And then you can assert on the output instead of verifying the behaviour.
This is definitely a good advice for unit tests, but rather not valid for integration and end-to-end tests. This is something I see very often to be broken. I will present it on a very simple examples.
A typical example of such case. Data provider gives data, test gives algorithm. The problem – doing too many things at once. And you can see it by looking at the name of this test method.
Run separately, so you have a very detailed information about the results.
Look at the names, they all make sense here – at least to me.
Nothing wrong with this test, it is valid and important. And – always as whether the code under test is not doing too much.
We know pretty well how to make our code readable. We choose good method names, we care about variables, we refactor so the code is clean. What irritates me, is that sometimes it would require a little effort to improve the readability of our code, and we do not do it.
This provides data to test methods.
This provides data to test methods.
This provides data to test methods.
This provides data to test methods.
This provides data to test methods.
So there is this mock server, which is parametrized and reused between many tests. Cool thing, definitely.
Or maybe you need a builder?
do not make the reader learn the api, make it obvious
We go back to SRP here.
When I see such names, then I already know that there is something wrong with tests.
A tiny example.
Contains expected result and the action which should trigger such behaviouir
Contains expected result and the action which should trigger such behaviouir
An example in all its glory.
An example in all its glory.
There is a difference between creating an assert and putting it into private methods.
At some point we realise that: Then part is too big Other tests have exactly same assertions only statuses differ
There is a difference between creating an assert and putting it into private methods.
Haven't we hoped for our tests to be a living documentation?
There is a difference between creating an assert and putting it into private methods.
It is not only about testing framework
Or use matchers – they also provide nice API
It is not only about testing framework
If they are important, then we should test all 4 cases. If it is not important than we should disregard it.
If they are important, then we should test all 4 cases. If it is not important than we should disregard it.
From what I have seen so far people who use random in tests will pay for it dearly. There is a probability that some values will never be tested ! If they are important, then we should test all 4 cases. If it is not important than we should disregard it.
Flickering tests – one time it is green, another it is read without apparent reason. Usually this is because of some external source is not reliable, or because of some concurrency issues. But sometimes it us who reall ask for trouble. Order of execution is not guaranteed Global state modifications
Flickering tests – one time it is green, another it is read without apparent reason. Usually this is because of some external source is not reliable, or because of some concurrency issues. But sometimes it us who reall ask for trouble. Order of execution is not guaranteed Global state modifications
Not much chance you will find some bugs this way.
From my personal experience – Amazon Elastic Beanstalk