Stop Defining your Regression Tests – Do this instead!

In this article, I’ll show you how easy it can be to keep your test suite in a manageable size – without having to go through the process of defining regression tests. But why is this relevant for your daily work? Because the true value of your unit tests lies in repeatedly executing them after each check-in. This ensures that your code changes did not cause any harming side effects. Now, you can even ship your code without having to fear that it will break down in production (or have a bit more confidence at least).

However, executing all of your unit tests after every check-in is not feasible either. Why? Maintaining unit tests takes a lot of effort. The more tests you have in your test suite, the more difficult it is to maintain them. So try to keep them as small as possible.

That is why regression testing was invented. You can mark those tests that you want to maintain long-term using some kind of tagging (e.g. test-case annotations in your favorite programming language). Based on this tag, you can execute those test on every check-in, to ensure a certain functional of the adapted code. However, deciding for each test whether it should be marked as regression test or not can be quite tricky. Once you’ve done this, the work is however not over. You have to continuously re-decide whether it still makes sense to keep and maintain each of your regression tests. This involves regularly browsing through your test suite, and doing all the hard decisions over and over again.

But today, I’ll show you a way to never have to go through this annoying pain of defining regression tests again. Just do this instead:

1Continuously Prioritize your unit tests

Instead of binarily and manually deciding whether a test should be maintained or not, you can simply rank your tests based on their effectiveness. To measure this effectiveness, there are already metrics that can be automatically calculated with the right tools. With these metrics, you can e.g. calculate the amount of code that is covered by an individual unit test (i.e. code coverage) or how likely it is that they detect a defect in the future (i.e. mutation score).Now, you have some solid data basis that makes it a ton easier to decide for each test whether it should still be used as regression test or not.

2Leverage the Automation Potential of Continuous Priorization

With this idea of continuous priorization in mind, let’s talk about leveraging its actual value. Now, you can reduce the number of decisions you have to make form hundreds for each check-in – to one single decision for the whole project. How? Simply automate the priorization of unit tests themselves. No need to weigh execution times against code coverage or mutation score to decide on the effectiveness of a test case. Just use a genetic algorithm that takes over here automatically (see Search Algorithms for Regression Test Case Prioritization). Now, the decision you have to make is simply the minimum effectiveness that your test suite should have – the genetic algorithm will find a set of regression tests that satisfies this constraint with a minimum number of unit tests involved. The best thing: you take one single decision. This decision applies to all of your unit tests, throughout the lifespan of your project. If you are not happy with this decision – you can obviously change it at any time – but you don’t have to – this is the true relief in regression testing!

That was my take on avoiding the pain in defining regression tests – let me know your take on it in the comments section below!

Vector created by Freepik – www.freepik.com