Coverage Analysis and what it means for you
Introduction
In my last post, I talked about how much bugs cost. In this post, I’m going to talk about how we can remove them. Removing bugs has always been a multi-faceted problem. Any bug removal strategy has to contain a number of separate attacks on the problem of bugginess.
Formal inspections have been shown to be the most effective way of removing bugs, both in terms of bugs discovered per hour of testing and the cost of finding those bugs, but formal inspections are only part of the solution.
In this post, I’m going to talk about another technique for rooting out bugs; namely code coverage.
What is code coverage testing?
When you write code you should always have a plan of how you’re going to test that code. One question you can ask about your test suite is “How many code paths do my test cases cover?” Or perhaps, more precisely “What percentage of my code is exercised by the test suite?”
Coverage analysis allows you to answer those questions. Most well used languages have at least one coverage analyser.
These coverage tools work by injecting instrumentation code in to your code which records when each statement in your program is executed. When the test suite is executed against this augmented code, a report is produced of all the statements executed when your test suite is run.
Here’s a screen-shot of the NCover tool I use at work:
The blue bits show the statements that were covered by the test suite. The red bits shows the code that was not covered by the test suite.
If I get 100% coverage does that mean my program is bug free?
No, this is the poisonous side of coverage testing. It’s very easy to turn it in to a game where the goal is to get 100% coverage. This is generally not a productive use of your time. The whole point of doing coverage analysis is to try and find untested areas of your code with the goal of finding bugs.
It’s a well supported fact that 80% of the bugs reside in 20% of the code. If you go for 100% coverage of your code base you’re going to spend a lot of money testing the modules that are not very buggy. You could take the same resources you’re using to achieve 100% coverage and put them in to properly testing your buggy modules and you would remove a lot more bugs.
Another problem is that coverage analysis has is that it can only test code that is there. There are a great number of errors that are caused by code that is missing; code that doesn’t exist.
For example, consider the following function written in Python. It tries to compute the roots of a quadratic equation of the form ax2 + bx + c:
def roots(a, b, c): root = (-b + math.sqrt(b**2 + 4*a*c)) / a return root
This code contains a few bugs. First, the inputs a, b, c are not checked to see if they’re numbers. Second, even if ‘a’ is a number the program will crash if the value of ‘a’ is zero. These are both precondition errors and in a language like Python, with its duck-typing, we might let the first of these slide. However, not checking whether ‘a’ is zero is definitely a bug. There is an even more serious error: there are two roots to a quadratic equation and the code only gives you one of them.
All the defects I’ve pointed out are defects of omission. There is code missing from the program that means an error can occur in some circumstances. If you were blindly trying to get 100% code coverage, you’d probably miss all these defects. You can get a 100% coverage of this routine by just passing a=1, b=2, c=3 to the algorithm and you’re done.
So what does coverage analysis actually tell me?
Coverage analysis tells you the areas of code that didn’t get executed by your test suite. This means that your test suite is deficient in someway as it doesn’t cover all the functionality.
It is important to use your coverage analyser as a tool to inform the development of further tests in your suite and not as an end to itself.
My approach to writing tests is to not use the coverage analyser on my first attempt at writing a test suite. After composing my first lot of tests, I then remove any bugs I’ve found and then run the coverage analyser.
This will normally throw up a few more features that need testing. I then code up these tests and check how I did again. Assuming I covered all the important scenarios, I then stop and move on to other types of testing.
The skill to correctly using a coverage analyser is knowing when to quit. If you quit too early, you’ll leave too much code untested, if you quit too late then you’re just wasting your time that could be spent on more focused, effective testing.