Test coverage serves as one of the great lightning rods in the world of software development. First, people ask whether it makes for a good metric at all. Then they ask, if you want to use it as a metric, should you go for 100 percent coverage? If not, what percentage should you go for? Maybe 42 percent, since that’s the meaning of life?
I don’t mean to trivialize an important discussion. But sometimes it strikes me that this one could use some trivializing. People dig in and draw battle lines over it, and counterproductive arguments often ensue. It’s strange how fixated people get on this.
I’ll provide my take on the matter here, after a while. But first, I’d like to offer a somewhat more philosophical look at the issue (hopefully without delving into overly abstract navel-gazing along the lines of “What even is a test, anyway, in the greater scheme of life?”)
What Does “Test Coverage” Measure?
First of all, let’s be very clear about what this metric measures. Many in the debate — particularly those on the “less is more” side of it — quickly point out that test coverage does not measure the quality of the tests. “You can have 100 percent coverage with completely worthless tests,” they’ll point out. And they’ll be completely right.
To someone casually consuming this metric, the percentage can easily mislead. After all, 100 percent coverage sounds an awful lot like 100 percent certainty. If you hired me to do some work on your car and I told you that I’d done my work “with 100 percent coverage,” what would you assume? I’m guessing you’d assume that I was 100 percent certain nothing would go wrong and that I invited you to be equally certain. Critics of the total coverage school of thought point to this misunderstanding as a reason not to pursue that level of test coverage. But personally, I just think it’s a reason to clarify definitions.