The Quest for a Good Bug Tracker

Let’s start with the documentation aspect.  We generated 30 plus pages of assumptions, clarifications and questions from an 8 page document we received from the client.  The document we received is the crux of the project so it doesn’t surprise me that it generated that much.  In fact, once the client answers the questions it’ll most likely generate yet another set of questions.

It’s hard wrapping our minds around the agile process.  All of our documents are ‘living’ which means that from sprint to sprint they might (and most likely will) change.  Documenting the documentation changes is becoming important.  From my engineering days this is completely backwards.  Requirements documents were set in stone and were the bible and it took an act of God to change them.  So in the end, all of us are adapting.

Rally
Our agile tool, Rally, has some built-in defect tracking software.  We are now generating defects (i.e. bugs) in our prototypes and have to start tracking them.  Rally’s own set of defect tracking is severely limited in a number of ways.  It just doesn’t seem to follow the life cycle of a bug properly:  report-assign/feedback-fix-evalulate fix-close.  Their default states just don’t follow that and while I’m sure we could have added our own status’ I think our biggest limitation was the interface.

The interface just isn’t very rich and the Rally interface is just setup wrong for our needs.  They have the ability to add custom fields which is nice but their inline edit doesn’t always work the way you would like it to.  So you end up using the editor screen which is clunky:  it doesn’t expand to fit the available screen space and that means you end up scrolling to get to your required fields.  Slow and painful.

So I went looking for alternatives.

Bugzilla
Our Subversion host automatically gives us Bugzilla and Rally has an integration with Bugzilla.  This was my first stop.  I’m not sure why Bugzilla is considered one of the best bug trackers around.  I (and everyone else on the team) found it very hard to use.  Maybe it’s because all of us are Mac users and we just found it confusing.  I was tasked with setting it up and found setup to be very cumbersome.  The people entering bugs didn’t like it either and then the integration piece didn’t work so well in that there’s really two components of any bug (the fix by the developer and the verification of the fix by QA) that Rally doesn’t create automatically.  So rather than beat our heads against the wall any longer I did more research.

TestTrack Pro
I’ve used this tracking system before.  Unfortunately we’re using hosted servers (i.e. hosted) and there doesn’t seem to way to get this to work for us.  Everything I’ve seen on their website says that you need a dedicated machine for the server.  I even tried finding a hosted TTP and couldn’t.  Since I didn’t have a lot of time to set it up I gave up on this solution.  Perhaps when our QA team comes on board we’ll revisit this solution since Rally has an integration tool for it but it’s also a bit pricey and before I commit to it again I’d want to use it for a while.

phpBugTracker
I’ve never had good luck with this tool so I didn’t bother with it.  It’s been very buggy in my experience.  Other developers I know have had good success so your milage may vary.

Mantis
We’ve used Mantis for a long time.  The other team members have used Mantis on their other projects as well so in the long run this really wasn’t much of a contest.  When we made the decision to switch I had it installed and setup in under an hour.  The project setup seems to be especially easy to understand and use.  The Change Log is very handy for documenting changes in any particular build and we’ve found that the email process works well for us.

I must admit that the MyView page takes a while to get used to but once you ‘get it’ it makes life easy.  With one glance you can see issues assigned to you, unassigned issues, recently changed issues, issues you submitted and issues that you’re monitoring.

Maybe I like Mantis because its interface is compact and doesn’t require a lot of scrolling and clicking.  The interface just seems to be in the right spots for me.

I would have liked to taken a look at FogBugz (Real Software uses this) and others but time just didn’t permit.  Frankly, now that we’re up and running on Mantis it’ll have to be something compelling to get us to change.

I mentioned this in one of my RB Developer columns, but I think it’s worth repeating.  A good bug tracking system that is accessible by your clients, as well as your team, is very important.  It’s important not only from a development standpoint but from a sales standpoint.  When you’re making your pitch to the client you should mention that you have a bug tracking system and that they’ll have access to it and they’re expected to use it (rather than email and phone calls).  You’re helping the client see that you have a process for your software development.

What are your thoughts on bug tracking systems?

What The Hell Were They Thinking?

In December 2007 I wrote in my wish list that I wanted them to start using a modern bug tracking system.   Let’s be honest, the old one sucked so I wish everyone would stop moaning that they liked it.  We put up with it, but once you use a real bug tracking system instead of the one they wrote and half implemented, you’ll realize how bad it was.

With that said, those making the decision to kill access to the old were smoking something.  Not everyone is using the most modern version of REALbasic.  Heck, there are more than a few developers still using RB 5.5 so removing access to it was a very bad decision.  Okay, so now they’ve brought it back as read only.  That should have been the plan all along.

FogBugz may be awesome.  I sure can’t tell from RS’ comments.  Can’t search for bug reports and workarounds?  I can’t believe that Joel Spolsky would implement a bug tracking system that you couldn’t search.  Come on, the guy wrote a good portion of Microsoft Excel.  He knows his stuff.  He’s good.  And his bug reporting system doesn’t have public vs private bug reports?  I can do that in my free Mantis bug tracker.

So yeah, RS dropped the ball on this one big time.  So instead of the sh**storm they’re dealing with now, this simple forum post (or one like it) would have prevented some of the outrage:

To All REALbasic Developers:
It has become increasingly obvious over the past several years that the Feedback System (i.e. the Feature Request/Bug Submittal System) for REALbasic and REAL SQL Server is not adequate for our internal needs, nor the needs of our customers.  The Feedback System does not have many of the modern features that we, and our customers need and are now demanding.
After a lengthy evaluation process, we have chosen to use FogBugz, an industry leading bug tracking and reporting system.  We feel that using FogBugz will allow us to meet the increasing demands of not only our customers but our internal developers as well.  Starting April 22, 2008, FogBugz will be the only method of submitting bugs and feature requests to Real Software for all of our products.
This change is not without some drawbacks.  Since the feedback system contains many years of bugs that are no longer relevant we have decided that none of the existing feedback system reports will be migrated to FogBugz.  There is so much old data, however, that we felt that it would cause more problems than it solved.
In addition, the current FogBugz implementation does not allow a public search of bugs, nor is there a way to add yourself to an existing feature request or bug and therefore we will have to write some additional tools and reporting mechanisms into the system.  We understand that these are must have features for many of our users so please be  assured that we are trying to implement these features with the help of the FogBugz developers as quickly as possible.  Unfortunately, we do not currently have a timeframe for these to be added back in.
In the meantime, the old Feedback System will remain in place in read-only mode so REALbasic users using older versions may still search and view existing feedback reports.
We understand that this will place hardship on some of you that depend upon existing functionality of the feedback system and we apologize.  If you have any questions or concerns, I will try to answer any and all questions in the forums.  Thank you for the patience and understanding.
Sincerely,
Someone Thinking Ahead

Real Software, we (us developers in the real world) depend upon you to make tough decisions on our behalf in order to make your products better.  However, you still have to keep in mind that we’re developers too and we have to make tough decisions for our products as well.  This week’s situation has the appearance that you are out of control or, even worse, aren’t fully thinking through the implications of your decisions and how they affect the user base.