What Makes a Good Bug Report?

They might not realize it due to my surly programmer attitude, but I love my testers.  I especially love a good bug report.  If a bug report is good, it can direct me almost exactly to the code I need to fix.  Some of the best reports allow me to diagnose a problem before I even look at the code.  With the right information about when, and under what conditions, a bug occurs, I can figure out solutions without so much as turning on my PC.  Even when I don’t know why a bug is happening, I can, at the very least, direct the bug report to the person most likely to know what’s going on.  Good bug reports can save incredible amounts of development time and, ultimately, lead to a better game.

With that in mind, here are a few of the features that I think compose a Good Bug Report (GBR™)

A Good Bug Report has a Crystal Clear Description

If I can’t figure out what the problem is, I can’t fix it.  Bugs that paint an unambiguous picture of what is going wrong, and what the correct behavior should be, get fixed faster.  Every moment I have to spend explaining my interpretation on a tester’s prose to someone else is a moment spent not fixing the actual bug.

This is an area where a tester’s ability to read their own report from the perspective of receiving the bug is invaluable.  Being able to establish the context for when and how a bug occurs can go a long way.  If a bug is hard to describe, a screenshot is crucial.  If it has something to do with animation or a complex sequence of events, video can be invaluable.  The key is to be able to answer the question: “Will someone who’s never seen the actual bug before know what I’m talking about?”

A Good Bug Report has a Solid Repro

Repros, the steps required to reproduce a bug, are crucial to fixing a bug.  By providing the minimum path to reproducing a bug, a good repro helps rule out irrelevant information and narrow the area of code a programmer needs to check to find the cause of a bug.  It also allows a programmer to make sure their changes have actually fixed the bug by running through the repro before and after their changes.

Naturally, it can be hard for a tester to pick out what information is irrelevant so erring on the side of too much information is a good idea.  However, it’s often possible to weed out irrelevant information by trying multiple variations of repros which leads me to the next point.

A Good Bug Report Checks Variations of the Bug

They’re not supposed to do it, but it happens: programmers copy-paste code to implement different, but similar, portions of games.  This often leads to the case where a bug may be fixed in one instance of the copy-pasted code but left in all other instances.  A tester that understands this can be extremely useful by testing likely variations of a bug and ruling them in/out.  For example, finding out that a bug repros only in multi-player but not in single-player game modes, whether a foot sliding problem occurs only for the dog enemy type or all quadruped enemies, or if a crash occurs only when starting a campaign with no save data.

If it’s a Crash Report, a Good Bug Report has Logs and Dump Files

Not all crashes can be reproduced 100% of the time.  These are often the most insidious bugs since it’s often difficult to know for sure whether or not they’ve been fixed.  Against intermittent crash bugs, the best weapon at a programmer’s disposal is forensic analysis of logs and state dumps from the time of the crash.  Outside of final shipping builds, almost all games produce copious amounts of log data and modern devkits usually have some facility for outputting the state of the machine into a (rather hefty) file.  Without this sort of data, a report about a crash that seems to occur randomly is about as useful as having someone come into the room and say: “The build crashes sometimes…”  Yes, good to know that something is wrong, but without any additional information there’s only so much developer’s can do to fix the problem.

There’s never enough time to implement everything we want in a game.  Hell there’s usually not enough time to fix every bug that comes down the line.  That’s why Good Bug Reports, which can save hours (or even days with offsite testing) of confirmation, are so valuable.  The purpose of a Good Bug Report is not to just report a bug, but to fix it.  Testers that understand this are some of the best assets a development team can have.

 

Share this Article:
  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Yahoo! Buzz
  • Twitter
  • Google Bookmarks
  • Print