Regression testing is the key value of unit testing, and the #1 enabler of agile unit testing. A good regression test suite provides you with the confidence that frequent changes did not cause any serious damage to your existing code base. A bad regression test suite however does not provide any value, as it either (i) contains too few tests to actually give some insights into your software quality, or (ii) it contains so many tests that they are not feasible to maintain, meaning that when they fail, the reason is that they most likely are simply outdated.

In this article, I’ll show you three easy steps to define your perfect regression test suite.

1Evaluate the effectiveness of each unit test

In order to judge which test cases are effective enough to maintein them within your regression test suite, you need some produce some data that allows you to compare test case effectiveness. To do so, you can use existing test evaluation metrics such as code coverage or mutation scores (read here for an intro to mutation testing What is Mutation Testing? or dedicated tool support: PiTest Basic Concepts.

However, all of these test suites have their advantages and drawbacks. For example, code coverage gives pretty instable results (see this article for some examples: Limits at which code coverage fails catastrophically), but is very computed quite easily. Whereas mutation testing is seens as the gold standard for evluating the value of unit tests, it takes very long to calculated mutation scores. Other metrics such as the execution time of a test case can be relevant as well, even though they do not regard the actual bug finding potential of a test case.

In order to overcome these tradeoffs and get the best of all worlds, different metrics can be combined into a single test suite effectiveness metric. If you calculate each metric on your test suite, you can e.g. simply sum up the values to get a combined metric. In order to account for the relevance of each metric within your project, you can also assign weitghts to each metric. The resulting formula could look something like this: regression effectiveness = 0.3execution time + 0.4line coverage + 0.4*muation score. By applying this combined metric to all unit tests, they can be ranked based on their effectiveness.

2Define an effectiveness threshold

Using the metric calculated above, you can simply define an effectiveness threshold that needs to be achieved by each regression test (e.g. test code effectiveness > 5). By throwing away any test that does not meet this threshold, you can ensure a minimum quality of each unit tests in your regression suite. Alternatively, you could also define a maximum number of regression tests, and simply choose to only include the most efficient test cases into your regression suite.

3Continuously assess and improve

Throughout your development project, you should regularly apply steps 1 and 2 on your test suite to decide (i) which newly developed test cases should be included into your regression suite, and (ii) which tests in your regression suite should be sorted out after some time. You should also keep an eye on your effectiveness threshold, and maybe adapt it once you realize that your regression test suite becomes too small or too big.

These were my 3 simple and easy-to-follow steps to define your perfect regression test suite. I hope that they will help you become a more efficient tester, and thus a better software developer. If they did help you, don’t hesitate to share your experience in the comments section below.

Image by vectorjuice on Freepik

We use cookies to give you the best online experience. By agreeing you accept the use of cookies in accordance with our cookie policy.

Privacy Settings saved!
Privacy Settings

When you visit any web site, it may store or retrieve information on your browser, mostly in the form of cookies. Control your personal Cookie Services here.

GetResponse, Google Analytics

We use LinkedIn Insight for marketing purposes. You can disable these cookies.

We use Google Analytics for marketing purposes. You can disable these cookies.
  • __utmz
  • __utma
  • _ga
  • _gat

We use GetResponse for marketing purposes. This service cannot be disabled, otherwise the website functions will be limited.

Decline all Services
Accept all Services
Get Free Access Now to
9 eBooks!
All about Automated Software Testing
Proven experts
Learn to save up to 75% of your test efforts
Get Free Access Now!
Get Access Now! & Save 50%
Personal Trainer FREE Nutrition Custom Workout App
Get Access Now!
eBook Download
Enter your details to get your free ebook!
All about Automated Software Testing
Download Free Ebook
Lorem ipsum dolor sit amet, consectetur adipiscing