How to evaluate the value of your test suite

Writing unit tests is easy. Most unit tests however do not provide actual value to your software. They don’t actually help you improve software quality, but simply require a lot of effort to create and maintain these tests. Evaluating the actual value that your test suite in practice is usually quite tricky, and requires a whole bunch of experience. To make this easier for you, I’ve broken it down into three questions you can answer in order to identify the value of your test suite:

What are the chances that a failing test finds a defect?

If one test in your test suite fails, you might assume that this is because there is a defect in your software. However, there are many other reasons for a test to fail. Maybe your requirements have changed, and you changed them in your software, but did not adapt the tests? Maybe you did a refactoring, and most of your tests don’t even compile any more? Maybe you have some test cases marked as “update”, but have never actually done the update necessary for them to pass again? Each of these reasons mean that once at least one test in your test suite fails, (i) you have to start investigating what’s the actual source of the failure and (ii) you have to fix your test in order to produce correct results again in the future (or the more likely option: start ignoring failing test after all). On the other side, if you can be sure that the failure is caused by a defect, you can start debugging to find the defect right away! In other words, the more likely it is that a failure in your test suite is ONLY triggered by a defect, the higher the value of your test suite!

How easy can you find a defect once a test fails?

Even if a failing test is triggered by a defect, the test itself should give you maximum guidance to fix the defect as fast as possible. However, in many cases, it’s already difficult enough to understand what the test case actually does – Don’t even try to get started on finding out which line of the test case is actually is actuall executing which parts of your software (side-effects included), and which values are used as test inputs. Maybe you can still achieve these tasks right after writing your unit test – but when returning again after months, or trying to understand a test that someone else has written – no chance. Don’t worry – just like you have clean code practices for your software, there are also clean unit testing practices to ensure readability and adaptability of your tests, to easily find the source of a test failure: Clean Unit-Testing – die Kunst, wartbare Unit-Tests zu schreiben.

So the better your test code, the easier it is for you to find the defect causing a bug, thus the higher the quality of your test suite!

How many defects can you potentially find?

Obviously, the more defects a test suite can find, the higher its value. The interesting part here is that you can actually estimate this bug finding potential. There are different test case evaluation metrics (e.g. coverage and mutation scores) that can be calculated for each test case, and then aggregated for the whole test suite. One of these metrics is code coverage, which uses the assumtion that the more lines of code you cover with a test case, the more likely it is that a defect is found. Of course this requires that you also use meaningful data to test these lines, which you can ensure when you use the correct methods (they are called boundary value analysis and equivalence class partitioning, and described here: 2 methods that help you save 60 % of your effort in Unit Testing) for defining your tests in the first place. Additionally or alternatively, you can also use mutation score, which artificially introduces errors in your software by altering the code (e.g. change <= to <) and measures how many of these errors are found by your test.

So what is the value of your test suites? Did answering these three questions help you improve this value? Let me know in the comment sectin below!

Image by vectorjuice on Freepik