The 3 ultimate steps to define your perfect regression test suite

Regression testing is the key value of unit testing, and the #1 enabler of agile unit testing. A good regression test suite provides you with the confidence that frequent changes did not cause any serious damage to your existing code base. A bad regression test suite however does not provide any value, as it either (i) contains too few tests to actually give some insights into your software quality, or (ii) it contains so many tests that they are not feasible to maintain, meaning that when they fail, the reason is that they most likely are simply outdated.

In this article, I’ll show you three easy steps to define your perfect regression test suite.

1Evaluate the effectiveness of each unit test

In order to judge which test cases are effective enough to maintein them within your regression test suite, you need some produce some data that allows you to compare test case effectiveness. To do so, you can use existing test evaluation metrics such as code coverage or mutation scores (read here for an intro to mutation testing What is Mutation Testing? or dedicated tool support: PiTest Basic Concepts.

However, all of these test suites have their advantages and drawbacks. For example, code coverage gives pretty instable results (see this article for some examples: Limits at which code coverage fails catastrophically), but is very computed quite easily. Whereas mutation testing is seens as the gold standard for evluating the value of unit tests, it takes very long to calculated mutation scores. Other metrics such as the execution time of a test case can be relevant as well, even though they do not regard the actual bug finding potential of a test case.

In order to overcome these tradeoffs and get the best of all worlds, different metrics can be combined into a single test suite effectiveness metric. If you calculate each metric on your test suite, you can e.g. simply sum up the values to get a combined metric. In order to account for the relevance of each metric within your project, you can also assign weitghts to each metric. The resulting formula could look something like this: regression effectiveness = 0.3execution time + 0.4line coverage + 0.4*muation score. By applying this combined metric to all unit tests, they can be ranked based on their effectiveness.

2Define an effectiveness threshold

Using the metric calculated above, you can simply define an effectiveness threshold that needs to be achieved by each regression test (e.g. test code effectiveness > 5). By throwing away any test that does not meet this threshold, you can ensure a minimum quality of each unit tests in your regression suite. Alternatively, you could also define a maximum number of regression tests, and simply choose to only include the most efficient test cases into your regression suite.

3Continuously assess and improve

Throughout your development project, you should regularly apply steps 1 and 2 on your test suite to decide (i) which newly developed test cases should be included into your regression suite, and (ii) which tests in your regression suite should be sorted out after some time. You should also keep an eye on your effectiveness threshold, and maybe adapt it once you realize that your regression test suite becomes too small or too big.

These were my 3 simple and easy-to-follow steps to define your perfect regression test suite. I hope that they will help you become a more efficient tester, and thus a better software developer. If they did help you, don’t hesitate to share your experience in the comments section below.

Image by vectorjuice on Freepik