One key aspect of unit testing is identifying how many unit tests you should have for your project. Keeping too many tests will lead to an increased maintenance effort. Having too little tests will obviously lead to important bugs being missed. So how can you find this sweet spot in the middle, where your test suite brings you maximum value? In the end, judging this effectiveness of your test suite comes down to one single question, which I will break down for you in this article: What is the value of each test case in comparison to its costs? So, how can you answer this question in practice?

1Assessing the value of a test case

First, you have to find a way to identify the value of each of your test cases. But what is the value of a test case? In the end, this is all about its potential to find a bug in the future (isn’t this the reason why you are testing your software?). This potential can be measured by simply counting the number of bugs that have already been found by this or similar tests, assuming bugs are often introduced in the same part of your software (There is a bunch of literature on this, see e.g. Predicting Bugs from History). Alternatively, you can also use more commonly accepted metrics such as code coverage or mutation score, or even combine several of these metrics, as shown here: How to Automate Regression Testing TODAY.

2Assessing the cost of a test case

Besides this value, each test case also has a certain amount of costs that have to be paid. One part of these costs is paid on each test run. E.g. the longer the execution time of a test, the more time you spend waiting. The largest portion of the costs of a test case however have to be paid during maintenance. After each change in your software, usually several unit tests start failing. Even if you’re “just” doing refactoring – changing the structure of your software will most likely kill a lot of test cases. Adapting these test cases over and over again, will be a whole lot of effort. Especially if you first have to think about what these tests actually do, and why they start failing now. A primitive way of calculating these costs is counting the lines of code of your unit tests, as these lines have to be maintained. However, minimizing the lines of code is not necessarily the most efficient solution. It’s rather about how maintainable these lines of code are written – so to which extent you applied clean unit testing techniques (as described e.g. here: Clean Unit-Testing – die Kunst, wartbare Unit-Tests zu schreiben). To judge to which extent you comply to these best practices, and thus assess how costly it will be to maintain your unit tests, ca be done using checklists such as this one here: The Ultimate Checklist for Better Unit Testing.

3For which test cases does the value pay off the costs?

Once you know the value and cost of each test case, you compare these values for each test case. Based on this comparison, you can easily see which tests have a value which pays off the costs that you have to pay for them over time (value > costs). If this is not the case, you know that you have to get rid of the corresponding test cases. How you weigh the value of a test (calculated in step 1) in contrast to its costs (calculated in step 2) of course depends on the requirements of your particular project. E.g. in a setting where labour is cheap and you can outsource tasks, time you have to spend on maintenance might be less relevant than if you as a senior developer have to take care of your unit tests yourself.

But that’s a story for another day (most probably a follow-up article on this blog).

Image by jcomp on Freepik

We use cookies to give you the best online experience. By agreeing you accept the use of cookies in accordance with our cookie policy.

Privacy Settings saved!
Privacy Settings

When you visit any web site, it may store or retrieve information on your browser, mostly in the form of cookies. Control your personal Cookie Services here.

GetResponse, Google Analytics

We use LinkedIn Insight for marketing purposes. You can disable these cookies.

We use Google Analytics for marketing purposes. You can disable these cookies.
  • __utmz
  • __utma
  • _ga
  • _gat

We use GetResponse for marketing purposes. This service cannot be disabled, otherwise the website functions will be limited.

Decline all Services
Accept all Services
Get Free Access Now to
9 eBooks!
All about Automated Software Testing
Proven experts
Learn to save up to 75% of your test efforts
Get Free Access Now!
Get Access Now! & Save 50%
Personal Trainer FREE Nutrition Custom Workout App
Get Access Now!
eBook Download
Enter your details to get your free ebook!
All about Automated Software Testing
Download Free Ebook
Lorem ipsum dolor sit amet, consectetur adipiscing