Good Bug Reports

Bugs happen.  It doesn’t matter how much experience you have, or how much you test your code, a user will do something that you didn’t expect and your application behaves improperly.  Perhaps it throws an exception and reports it to the user and continues working.  Maybe your application quits on exceptions.  Either way, you have an unhappy customer that is reporting back to you.

“It’s crashing,” is a common phrase I hear in the Xojo forums and from our own users.  So the first question I usually have is “is it really crashing or is the application reporting an exception?”  The difference being, of course, one is controlled by me and the other is the application goes *poof* and there’s nothing controlled about it.  With the former I usually have some data that can help me figure out what’s going on and the second is a bad sign that it might be a plugin or library issue that I’ll have to track down.

What is their operating system and version?  

It’s important to know if they are running on Mac, Windows, or Linux and what version it is.  For Windows and Linux it’s important to know if it’s 32-bit or 64-bit.  You may end up having to send them instructions to determine what version they’re running depending on the end-users skill level.

What version of the Application/Library/Control are they using?

Applications are relatively easy to get version numbers from since the user can get this from multiple locations (usually).  Controls and libraries are a different matter and you, as the developer, should make this easy to find with documentation and constants in the code.

We’ve all had an email where a disgruntled customer says, “It doesn’t work!”  If you have multiple products this makes for an awkward return email asking which product they’re actually talking about.

Can they replicate the problem?  If so, what are the steps?

Customers often think that simply telling you about a problem is good enough.  Sometimes it is but usually I need more information.  The more detail the better.  If it’s a sequence issue I will sometimes ask for a video showing it in action.

Can they send you the error log?

If you create error logs it is helpful to get those.  To get the error Stack of an exception is useful and can sometimes tell you exactly what to fix if you’re lucky (especially with small/short methods!).

You may have to tell the user not only where they are but how to get the location.  Getting to the Application Support folder on Mac and Windows isn’t hard but it’s also not easy.  On MacOS the users Library folder is invisible by default.  On Windows this location is buried multiple layers deep.

Can they send you an example project or file?

With developer products it’s handy to get a small example project demonstrating the issue.  I can’t tell you how often simply asking for this solves the problem because they find their own mistake.  Even if it doesn’t you now have a good example project demonstrating an issue you didn’t know about!

With applications and utilities getting their data file is good.  There is nothing like working with real data to discover issues.

Users aren’t often helpful when it comes to asking for help with your products.  What other things do you ask customers when dealing with bugs?

4 thoughts on “Good Bug Reports

  1. My own Xojo-project-framework, my point of starting a project, takes care about logging as much error-details as possible, also stack, method-names etc. Next I log the customer’s OS- and environment parameters at first startup and at every change that might occur when the user is updating his OS or hardware.
    The logging goes into a local configuration- and application file (sqlite) which is created and populated with initial parameters and values at first startup, or if not existing at launch. When the user quits the program on a normal way, and errors have occurred during this session, a dialog will appear listing all logged details of the session and if there is still a connection to the application-production database, a button to upload all this information to the sysop. To have a clear overview about what was going on, every session has it’s own unique session-id logged.
    Once created all this, it’s a pleasure for all projects having detailed information even before the end-user is calling with the message “it’s just not working”.

  2. Good bug reports are worth their weight in gold. It’s been my experience that good bug reports are also often the difference between getting your bug fixed or not, especially in projects that are more resource constrained. These are all very good tips towards writing a good bug report!

    The only other thing I would add is in regards to sending in an example: by default, try to send in a *minimal* example that demonstrates the issue. Only submit more complicated examples as a last resort.

    It can be frustrating to spend time doing this “busy” work, but 1) it will show you where you’ve made a mistake in your code, if you’ve made one, and 2) if it really is a bug, a minimal example is often *so* much easier to work with when doing bug triage that your bug is more likely to be fixed. Spending that bit of time and effort to find a minimal example is very rarely a waste.

    • I can’t tell you how many times I’ve been all mad about some ‘bug’ and started a small example project only to find that it really was my code.

      But, on the other hand it’s also frustrating to start a small example project to have it work but have no idea why it’s working in the small project but not in the large project. Is it a cache issue? Something with the file format changing? Or something about the larger project?

      • Yeah, dev tools should never be “mysterious” to their users. That’s one of the ways you lose trust in the dev tool. Don’t get me wrong, I can screw up a makefile like you wouldn’t believe (build systems are not fun): but those mistakes are *mine* and not my tool’s 99 times out of 100.

Comments are closed.