(No) Tickets, please!

Published on and tagged with bug tracking  communication

In the last few days I noticed a new “pattern” on CakePHP’s bug tracker: tickets got closed with the resolution “needmoreinfo” and the explanation “We require test cases for all bugs opened against stable releases” (see tickets 5943, 5946, and 5951).

Sure, if you have a project you are free to define your own rules, and you can define a rule as shown above. However, such a rule has its price: you are annoying exactly those people who want to help you to make a better product…

28 comments baked

  • klevo

    Very sad to see this. Especially those 3 tickets are absolutely clear about where the problem is. There are people that just don’t do unit testing yet. It’s their big problem, but they’re still capable of reporting bugs. Plus adding a test case to even a simple problem could take hours.

  • Abhimanyu Grover

    lol.. yea. It happens a lot, and that is why I sometimes dont feel that CakePHP in a whole is a community effort. Tickets without unit tests are closed as if developers think that we’re making a request to add fix to the code, when many people like me just create tickets to actually help the community. Lead developers should understand that people who submits a ticket are actually investing a part of their time to report, and then see their effort wasted by the end, when ticket is closed and marked with “needmoreinfo”.

  • Mike

    This could also be interpreted as the devs trying to raise the standard for the project as a whole. Providing errors accompanied by failed tests shows a high degree of professionalism imho. Forcing end-developers to write tests also educates them about the deep inner-workings of cake thus increasing the number of people who could contribute valuable core-level enhancements.

    I don’t know if I really believe anything I just wrote.. but it’s a thought. btw keep up the good work with your blog. I’ve gotten a lot of questions answered from here :)

  • bakytn

    It happened to me too I was confused:

    https://trac.cakephp.org/ticket/5816

  • Rob Weaver

    My only add to this would be that there should be a clear link to guidance on setting up a test case for this. I had one a while back that I opened, only to never reopen due to my inability to find any guidance on how to write a test case.

    If you are asking more from the community, I think it is important to help the community to learn how to help you solve the problem.

    P.S. – this is not in any way a slam, I’ve been way more than happy with the level of support and response from the core developers and the Cake team in general. The only thing I have seen is that finding help getting over the initial learning curve was somewhat challenging.

  • Troy Schmidt

    I have to say that the “requirement” for unit testing really does bother me as well. I spent many many hours before Cake even released a stable version working on it and the inner workings. Tracking through lines of the core files to find errors and problems. So I was working with CakePHP before unit testing even started. Sadly I haven’t gotten into unit testing yet.

    I do understand that before tickets were being created left and right and without people researching existing tickets, not marking as enhancements, etc. So while this does keep tickets with unit testing to real tickets, it eliminates any of the other real help that could be out there.

    My favorite ticket was one that was closed by Nate three times over the course of a month and finally fixed by Nate himself in the end. It had something to do with the new find(shortcuts) methodology that they recently implemented. It was very interesting.

  • dragon

    Heh, sounds very familiar. If u want to see big kahuna’s answer, check this ticket: https://trac.cakephp.org/ticket/5045

    It even got converted to an enhancement, even thou it it clearly a bug (Cake adds a question mark after link if it can not find file). It’s very sad.

    I think it is a bigger problem for CakePHP than it seems.

  • alkemann

    I think none of you understood the nate’s answer. The key word is “stable”. In other words, you will be allowed to make willy nilly tickets on 1.3 when that milestone goes into development, but stable versions have all passing test, and you just saying it isnt so, just dont cut the mustard.

    Ps. This is just my personal observation / oppinion and I am not a cake core developer.

  • Rafael Bandeira

    I think we have to remember that, (a) trac is not a troubleshooter, it’s a bug tracking system: if there’s a bug, prove it, (b) the dev team isn’t by any means in need to solve problems that they don’t have/can’t see/can’t reproduce and, as nate said once, they don’t own the community or whatever anything, so why can’t you just help them to do their freetime job, part time occupation and hobby work that is to provide us developers with a good frame for us to work with.

    It might a little arrogant from me to say that but, the framework community isn’t made by “users” as people say, it’s made by “developers” and as such they should care more about learning stuff and using them properly. Why can’t you just go there and try to learn how to get things done in a way that you can help the people that are helping you? I couldn’t write a test case on the first bug I reported, I couldn’t even make a diff on windows, but I took like 10 minutes talking with my uncle Google, and I could provide the team with a valuable material to see the error, to get a working test and a working diff. Anyway, if you are not contributing with the project, and are not liking the result, move along.

    In other side there’s still the fact that bug reports can be easily understood by the reporter, but sometimes it just means nothing for the people who’s reading it, even more if one has spent uncountable hours planning, designing and coding that supposedly wrong piece of code, and has all expected behaviors covered in all passed test cases.

    What’s wrong with taking some time to help who’s helping you. It’s like helping them to help you, it’s not asking that much.

  • Jonathan Snook

    I understand where the sentiment comes from. It “feels” like a passive-aggressive way to avoid work. I understand that this isn’t the intention of the core team. They don’t have time to investigate every single bug filed, especially since many provide little useful information that would point the core devs in the right direction. However, there is still a high barrier of entry to filing a bug report when a test case is required. It’s also frustrating to go through the amount of work to put together a test case when it doesn’t get accepted in the end.

    What would likely help mitigate some of this feeling is the wording on closed tickets that “needmoreinfo”. Something like, “It’s time consuming to validate bug reports. Please create a test case and re-open this ticket. This will save us time and make the framework stronger.” Make it boiler plate.

  • freenity

    Actually yes, it’s annoying, of course devs work is not troubleshoot, but user’s work is not debug the core, making tests and fixing bugs. It is a very nice contribution if they do it, but I don’t thing it should be mandatory.
    What do you think of this? http://freenity.wordpress.com/2008/12/29/asserttags-a-bug/
    They closed it but there is still a bug / or API is wrong….

  • brian

    slightly of topic, but is learning to write test cases a very generic thing or are there specific ways to do it for Cake? I did some hunting and found some general articles on test driven development, but if I find what I think is a bug, do I end up need to create a model, view, controller for the test case and send all that code or what? My apologies for asking this in the venue of a blog comment thread.

  • Xr

    The way to submit a good bug report is to submit a reproducible way of triggering the problem, all right. Forcing the use of unit testing is overkill and will eventually slow down the overall development and reduce community input.

    I’m convinced unit testing is a good idea, and I’m using it in a few projects. However, I find it very impractical in PHP, and I’ve seen my real data go away more than once (I had made backups). Of course, it was all my fault but the tediousness of the process means I won’t dig into it before long.

    If you want to see a good example of a bug tracking system that works with enough information but no really strict prerequisites to submitting a bug, just check Debian’s. 500k bugs and counting. Surely the devs didn’t do all the work, did they ?

    This elitist attitude is symptomatic of CakePHP. There aren’t a lot of successful FOSS projects whose developers are so aggressive. The day a Cake dev will say a user is “full of shit” (remember Theo de Raadt ?), I will drop the software and will start pushing people to do the same. Hopefully I’ll never have to.

    tl;dr: Lower your expectations: you can’t ask so much professionalism from users.

  • Martin Westin

    I am of the opinion that requiring testcases is acceptable in general.
    However:
    • Clear guidelines as to how to write and submit them is required.
    • Exceptions should be made when an existing testcase is just a single-line edit away from catching the bug. (and similar small changes)

    I am also one of the many that have had problems trying to make a testcase that is up the the standards required to satisfy the core team. I still have no idea how to make the right kind of diff file that trac likes. My svn-diff files just looks like any old text-file when uploaded.

    I guess the google group or some other forum will now become tha place for post like:
    Can someone help me make a testcase for this bug I found?

  • teknoid

    I think it is very fair to require a test case to prove a bug.

    Especially if there is already a test case showing that the “bug” is actually non existent.

    Consider that “tough love” or whatever, but innocent until *proven* guilty is a better approach, imo, than vice versa.

    So what happens if you don’t know how to write a test case? Ask at the google group, jump on the IRC channel or email one of the devs.
    You’d be very surprised at how helpful people can be…

    And even if that fails, open a ticket and clearly state your problem *and* ask someone to help you write a test case. Just because you opened a ticket and cannot write a test case, someone else just might help you out (or disprove the issue) as it has happened many, many times in the past.

  • Brendon Kozlowski

    I have had help in creating a test case, creating a diff, and supplying a patch; thanks to mark_story that patch was also applied. However, when working with another “bug” (assumed) in the model, the methods and properties I needed to test were not instantiated in the test case automatically since I’m not using the model class, I’m using the test case schemas. When calling the method from the schema, there is no error. When calling from a plain-jane baked project, the issue exists. I’m having problems creating a test case to show this specific issue, and have almost given up and decided that modifying the core to fix *my* issue (and breaking 4 other test cases) is almost justifiable since no one on IRC was able to help, and god forbid I post in in Google Groups with a real request for technical help on the core. :P

    I’m sure I’ll eventually try to tackle it again, but after wasting a full week trying to get a simple test case working, I’m sure you can understand my hesitation to try again with “needmoreinfo” after explicitly stating that I was unable to come up with a test case (during RC4, not stable). :( Heck, I’m almost willing to PAY someone to write the damned test so I can learn.

  • cakebaker

    @all: Thanks for your comments!

    @klevo: Exactly, not everyone is into testing. I would even guess that’s the majority of CakePHP users…

    @Abhimanyu: I think it is a typical lose-lose-lose situation: the bug reporter gets discouraged to contribute, the bug isn’t fixed, and the developer doesn’t get the help he is looking for…

    @Mike: Sure, receiving only tickets with failing tests is a developer’s dream. However, requiring tests is an additional barrier for people to contribute. And I really doubt you can educate people by using “force”, they simply won’t open tickets anymore…

    @bakytn: To me your ticket sounds more like an optimization (“don’t perform an unnecessary query”) than a bug, and so it is indeed a bit strange to get asked to provide some sample code/a test case.

    @Rob: You can find some info about testing in the cookbook: http://book.cakephp.org/view/160/Testing

    @Troy: Yes, watching trac can be quite funny sometimes ;-)

    @dragon: Yes, I have seen that answer :)

    @alkemann: That’s quite illogical imho ;-) Users of the stable release (which comes without tests btw) should open tickets with tests, whereas for the users of the development branch it is ok to open “willy nilly” tickets…

    @Rafael: Yes, trac is not for troubleshooting but for bug tracking. However, I don’t think bug tracking is about proving bugs. If you can prove a bug by writing a test, then do it. But if you are, for whatever reason, not able to provide a test, then it is ok to “just” describe the bug as good as you can.

    Yes, it’s true, the cake devs don’t own the community anything, however, it is their product, and so it should be in their own interest to know about as many bugs as possible and to fix them occasionally.

    I think you are mixing roles with your saying the community consists of “developers”. Sure, most people in the community are some kind of developers, but their main role is “CakePHP user”, because they use CakePHP and don’t care much about its inner workings. There are some people like you, who are interested in those inner workings, but you can’t expect everyone to be like that. And contributing is not only writing tests/patches, but also writing bug reports. A bug report is as valuable as a test. Even an invalid ticket may contain a hint of something you can improve (maybe a doc block is not that precise?).

    It’s funny you mention: “What’s wrong with taking some time to help who’s helping you. It’s like helping them to help you, it’s not asking that much.” This not only applies to the bug reporters, but also to those who are handling the tickets ;-)

    @Jonathan: I doubt a nicer wording would be more helpful, it is still the same message: “I don’t care what effort you put into the ticket, it doesn’t follow our rules, stupid!”

    I think Seth Godin described it very good in his article “Teaching people a lesson”: “Instead of worrying so much about establishing good habits among transient customers [or in our case, bug reporters], perhaps it’s worth figuring out what works best for both sides and doing that instead.”

    @freenity: It seems like this issue got resolved :)

    @brian: Learning to write tests is both: on the one hand you have to learn some generic concepts and on the other hand you have to learn how to use those concepts with a concrete implementation (i.e. in your case cake). It is similar to learning your first programming language: you have to learn for example the concept of “loops”, plus how to write a loop in PHP, if you are learning PHP.

    No, usually you do not have to create model, view, controller, if you write a test for the core. If you intend to write a test for a specific class (e.g. the Model class), then it helps to look at the tests which are already there, to see how tests are written for this specific class.

    Hope that answers your questions!

    @Xr: Yes, I agree with you that Debian’s bug tracking system is a good example for a system with low barriers for submitting bugs. And with “reportbug” they even provide a tool to make it easier to report problems.

    Yes, the attitude of some core developers could be “improved” ;-)

    I fully agree with your latest sentence: “Lower your expectations: you can’t ask so much professionalism from users.”

    @Martin: Yes, clear guidelines would be helpful, independent of whether test cases are required or not. However, I’m not sure you should have exceptions if you really want to require test cases. I already see the discussions on trac: “That’s an exception” – “No” – “But…” – “NO!” ;-)

    I can’t help you with the svn-diff issue, personally I always use the “create patch” feature of Eclipse…

    @teknoid: Well, your “tough love” concept can also be applied in the following way: “A ticket is helpful until proven not to be helpful”. It really depends on your point of view, whether you see it from a developer’s or bug reporter’s point of view.

    Sure, your expectations can be that users will do x, y, z, to get a bug fixed. But the reality is not like that. Some will meet your expectations. Many won’t. Now you can be “angry” about those people for not meeting your expectations and try to “force” them to meet your expectations, or you could be thankful even if they “only” contributed a bug report. It’s your choice how you see it :)

    @Brendon: Yes, that’s frustrating…

  • teknoid

    @cakebaker

    “A ticket is helpful until proven not to be helpful” … i respectfully disagree. In many projects we’ve gotten tickets that were not only a waste of everyone’s time, but seriously stirred up trouble.
    (I’m sure you’ve seen a few of those).

    Now, the “attitude” with which some tickets are closed are a different story, but that’s not my right to judge.

    Requiring a test case does slow things down from a business process perspective and obviously gets some people upset. However, the ultimate result is a better end-product and, that outweighs some hurt feelings, IMO.

  • Martin Westin

    @cakebaker
    I can see the line-drawing being a problem but it does feel like a big waste of effort to write a NEW test case when an existing one does not do its job and should be fixed.

    It the function testSomething() (or its fixtures and other data) turns out to not cover every eventuality I think it is bad to require the creation of new fixtures and the function testSomethingAgainButCorrectlyThisTime() instead of correcting the existing code.

    The question was more: Should a new test case and new fixtures be created to illustrate a bug in a test case or a fixture?

    I had one of these situations myself. A destructive bug that was (IMHO with poor attitude) closed and re-closed on the grounds of “policy” with little regard for a the fact that it actually destroyed data in peoples applications.

    There was an existing test to cover bug, but it was not prefect. One (1) line added to the test’s fixture revealed the bug and showed that the suggested patch fixed the bug and did not cause any other tests to fail. The accepted test eventually ended up being 30-40 lines of code identical to the existing test except that the corresponding fixture had that extra line.

    The bug was caught early (not by me, I found it a day later) but was open a lot longer that necessary (again IMHO). In the mean time causing several threads in the google group where people was wondering where their data had gone.

    I am not looking to bash on the core team, but I can see how the above may sound like it.
    I just want to illustrate that this door swings both ways. Requiring tests will definitely weed out a lot of “nonsense” but it will also slow down the time it takes for real bugs to be fixed.

    And I am currently in no position to submit any tests… since my cake/tests dir for 1.2 stable is almost empty. Anyone know where all the tests went? :)

  • cakebaker

    @teknoid: Yes, there are tickets which waste everyone’s time, but I don’t think you can eliminate those tickets completely, even if you require tests.

    I agree with you that requiring tests will slow down things, but I don’t think it will result in a better end-product. I don’t see a connection between the quality of the end-product and the requirement of providing bug reports with tests.

    @Martin: No, that would be a bit overkill to provide a test for a test ;-)

    And regarding an almost empty cake/tests directory: the release doesn’t contain the core tests, see the announcement: “For this release, we have removed the test files from the build [..]”.

  • Martin Westin

    I only found the line: “Test suite now integrated into the framework” and at first though that they sure had integrated it well since I could not find the files :)

  • Abhimanyu Grover

    @cakebaker

    “For this release, we have removed the test files from the build [..]“

    That is quite funny!

  • cakebaker

    @Martin: *g*

    @Abhimanyu: It seems like the left hand doesn’t know what the right hand does ;-) However, I think it is a good idea to ship the release without tests as probably most users of the release don’t need them anyway.

  • Matt Huggins

    I basically stopped submitting bugs after I realized how much extra work was required on my end. I thought I submitted a ton of info already, but it was marked as closed without any work because I didn’t provide a test case, diff file, etc. I did it for one issue that I needed to have fixed, but it was too much effort for me to want to go through it again.

  • cakebaker

    @Matt: Yes, that’s not really encouraging to contribute and I really hope the cake devs will rethink the way they respond to tickets…

  • Brandon P

    Dear CakePHP,
    Don’t bit the hand that feeds you.

    Sincerely,
    Everyone who is just trying to help,

  • cakebaker

    @Brandon: Well said :)

  • CakePHP Digest #5 | PseudoCoder.com

    […] (cakebaker) declaration of his framework free agency (on twitter too). But then his first post after this announcement was about CakePHP. Since that post launched a fairly long comment thread let’s skip right […]

Bake a comment




(for code please use <code>...</code> [no escaping necessary])

© daniel hofstetter. Licensed under a Creative Commons License