Don't test to remove bugs!

Testing is to defect location as trial division is to finding primes. Just as it's very slow to show a number is prime by doing trial division it is very expensive to expose bugs in your program simply through blind testing.

Yet I've found that a lot of people do think that this sort of blind testing is the best way to find and remove bugs from their program.

Just like there are mathematical short-cuts that allow you to find primes quickly, defects have properties you can exploit that help you find them faster.

There are a number of useful facts about software defects:

• 80% of the defects come from 20% of the code.
• High cyclometric complexity is a good predictor of defects.
• Even if your test coverage was 100%, roughly 35% of defects emerge from missing logic paths.
• Another 40% of defects come from the execution of a unique combination of logic paths.
• There are errors that most programmers tend to make.
• The longer you leave a bug, the more expensive it is to fix it.

These facts paint a pretty important picture; there is a much more cost effective way to a lower defect rate.

I've come to the opinion that the biggest generator of defects in a piece of software is the design of the program. The fact that 80% of the bugs live in 20% of the code suggests that this must be true. What's special about that 20%? Why is it so buggy?

It's because this code contains bad abstractions, low cohesion, high cyclometric complexity and tight coupling. It's badly designed!

Fortunately, these facts give us a clear strategy for reducing defects. All we have to do is improve the design of the program. The better your design, the simpler your code will be. The simpler your code will be, the less bugs it will contain.

The simplification of one abstraction may result in the simplification of other dependent abstractions. Success often breed success and you can quickly find you have a radically simpler program and consequentially far fewer defects.

It is clear that there are common defects that all developers make. They tend to be simple things, such as not checking a pointer for null before dereferencing it, or using an out of range index on an array. This implies that defect check-lists are a good way to keeping common defects out. It also implies that there may be ways to automate detection of these defects at check-in.

If high cyclometric complexity indicates the presence of bugs, perhaps you should ban checkins of routines that score highly.

So if the purpose of testing is not to find defects then what is it for? It's for confirming that the specification has been implemented correctly.

Test-Driven-Development (TTD) doesn't produce high-quality code because it finds bugs; although that is a nice side effect. It increases the quality of the code because in order to unit-test code you have to make sure your code has low coupling, high cohesion and low cyclometric complexity. You have to have a reasonably good design.

When you're writing your tests in TTD, you should at the very minimum have one test per function point in your specification. In reality you probably need an order of magnitude more tests.

This is the purpose of testing; to ensure the requirements have been fully implemented.

2008-08-21 21:27:55 GMT | #Programming | Permalink