The Coolest Research Ever!

This morning I attended "Looking for Bugs in All The Right Places", a talk by Elaine Weyuker.  She presented what she described as 'the coolest research ever' - and she's right, her work is awesome.  This is probably one of the best talks I've been to at any GHC, ever.

Weyuker introduced the goals of her research as finding the parts of a large software system which are likely to contain the largest numbers of bugs in the future.  At first, I was worried about job security!  But when it comes down to it, if possible this would allow testers to:

• Find bugs faster
• Get higher quality code for lower cost
• Help Devs know when to do design and code reviews and what to re-implement
• Focus on a particularly risky part of the code

The first step in the project was to verify that bugs are non-uniformly distributed among files.  Generally from our experience as Testers we know this - but it's still important to verify it so that the dataset can be well understood.

One of the most interesting findings is that there are only five simple predictors of whether a file will contain bugs or not:

• Size of file (the larger, the more bugs)
• Number of changes to the file in the previous two releases (it turns out some changes can improve a file, but others can make it even worse.  We Testers know this all too well and are paranoid about any changes being made.)
• Number of bugs in the file in the previous release
• Age of the file (number of releases the file has been in the system)
• Languages the file is written in (if you use many different languages, there are more likely to be bugs when the code interacts)

A surprising result is that the author of the code is not a predictor.  One reason is that good programmers tend to get more difficult assignments, resulting in riskier code and potentially more bugs.  Poorer programmers tend to have more critical code reviews and their code is more scrutinized. 

Weyuker studied nine systems and found that 83% of the bugs in a given system can be found in 20% of the files.  She was asked if she considered the severity of the bugs found, which is the question I was really curious about too!  She said that they did consider severity in their study but found that it was very inaccurately reported (at least in the databases they looked at); roughly 1% of bugs were reported as severity one, while others were downgraded to severity two for political reasons.  At the same time, severity three bugs often get filed as severity two bugs (I can believe this as passionate testers want their bugs fixed and might think a higher severity will convince the powers at be to do so).  This made me think about how important it is to build 'doing the right thing' into any bug filing and triage processes a company creates to ensure that games aren't played with bugs to either hide or promote issues.

At the end of a talk I asked Weyuker whether they considered the type of bugs (i.e. UI vs. performance bugs) in her study.  She said hey didn't in this particular work but had done other research that did include how the bugs were found.  One thing I've been thinking about lately is how to find bugs that aren't necessarily functional problems but rather UI defects - what if font sizes are off, there is a 2 pixel offset, etc.  I think an interesting future direction for this work would be to see if it can be used to predict which files are likely to contain which kinds of bugs.