The key to evaluating your test suite effectiveness

One key aspect of unit testing is identifying how many unit tests you should have for your project. Keeping too many tests will lead to an increased maintenance effort. Having too little tests will obviously lead to important bugs being missed. So how can you find this sweet spot in the middle, where your test suite brings you maximum value? In the end, judging this effectiveness of your test suite comes down to one single question, which I will break down for you in this article: What is the value of each test case in comparison to its costs? So, how can you answer this question in practice?

1Assessing the value of a test case

First, you have to find a way to identify the value of each of your test cases. But what is the value of a test case? In the end, this is all about its potential to find a bug in the future (isn’t this the reason why you are testing your software?). This potential can be measured by simply counting the number of bugs that have already been found by this or similar tests, assuming bugs are often introduced in the same part of your software (There is a bunch of literature on this, see e.g. Predicting Bugs from History). Alternatively, you can also use more commonly accepted metrics such as code coverage or mutation score, or even combine several of these metrics, as shown here: How to Automate Regression Testing TODAY.

2Assessing the cost of a test case

Besides this value, each test case also has a certain amount of costs that have to be paid. One part of these costs is paid on each test run. E.g. the longer the execution time of a test, the more time you spend waiting. The largest portion of the costs of a test case however have to be paid during maintenance. After each change in your software, usually several unit tests start failing. Even if you’re “just” doing refactoring – changing the structure of your software will most likely kill a lot of test cases. Adapting these test cases over and over again, will be a whole lot of effort. Especially if you first have to think about what these tests actually do, and why they start failing now. A primitive way of calculating these costs is counting the lines of code of your unit tests, as these lines have to be maintained. However, minimizing the lines of code is not necessarily the most efficient solution. It’s rather about how maintainable these lines of code are written – so to which extent you applied clean unit testing techniques (as described e.g. here: Clean Unit-Testing – die Kunst, wartbare Unit-Tests zu schreiben). To judge to which extent you comply to these best practices, and thus assess how costly it will be to maintain your unit tests, ca be done using checklists such as this one here: The Ultimate Checklist for Better Unit Testing.

3For which test cases does the value pay off the costs?

Once you know the value and cost of each test case, you compare these values for each test case. Based on this comparison, you can easily see which tests have a value which pays off the costs that you have to pay for them over time (value > costs). If this is not the case, you know that you have to get rid of the corresponding test cases. How you weigh the value of a test (calculated in step 1) in contrast to its costs (calculated in step 2) of course depends on the requirements of your particular project. E.g. in a setting where labour is cheap and you can outsource tasks, time you have to spend on maintenance might be less relevant than if you as a senior developer have to take care of your unit tests yourself.

But that’s a story for another day (most probably a follow-up article on this blog).

Image by jcomp on Freepik