Yesterday, I opened some tickets because some core tests failed while I run them with the new testsuite shell. As I didn’t attach any patches to the tickets I got “attacked” on twitter by some people from the core team, and I even got a nice mail from gwoo (the project manager of CakePHP) that I should stop submitting tickets…
Anyway, I think this is a good opportunity to think about failing tests (with tests I primarily mean unit tests).
Even though “failing tests” may sound quite negative, this is not the case, at least if you practice test driven development (or a similar approach). There, failing tests are an integral part of the process. You start with a failing test, and then you implement the functionality so that the test no longer fails. Or if you make a change, and you break something by accident, then a failing test informs you about it. So, a failing test is quite positive.
On the other hand, failing tests also have a dark side.
The longer you wait to fix them, the higher the “costs”. If you fix a failing test immediately, then the costs are almost zero: you know what you changed, and so you can easily fix it. But if you wait a while, then the code is no longer present in your mind and it takes much more effort to fix the failing test. A side effect of this waiting is that it encourages others to do the same. If X doesn’t fix the tests he has broken, why should I care whether my tests run? As another side effect it may undermine the motivation of bug reporters to contribute tests if they see the project members don’t care about tests…
A special “behavior” of failing tests is that they don’t tell you whether they fail because of an incorrect test case or a bug in the code you test (in practice, this is sometimes obvious, sometimes not, depending on various factors). And they don’t tell you whether they only fail in your environment or everywhere. So, this means, if I run your tests and some of them fail, then all I can say may be: test X fails in my environment (the reason for saying “in my environment” is that I assume you don’t give me broken code, i.e. code with tests you know are failing). As it is an environment-specific issue, I will inform you about it, so it can get fixed. But if you provide broken code, then everyone using it will bother you with the same issue, at least theoretically ;-)
I think it is obvious what helps to deal with those dark sides: don’t provide code with tests you know are failing, it’s that simple!
I hope it wasn’t too confusing, and sorry about the rant at the beginning, but sometimes it’s necessary to vent one’s anger ;-)