The 3 Golden Rules of Regression Testing

Running a unit test once helps you find bugs in your code without having to run your software. This can be nice – but the actual value of a unit test lies in running it over and over again – automatically. So no need to think about what potential side effects your code changes might have. Define a good set of regression tests, and you have this certainty that a check in did not cause any side effect that broke your whole software. But to leverage this value, there are certain rules you have to stick to for creating a reliable set of regression tests that is easy to maintain. In this article, I share with you my 3 golden rules of successfully applying regression testing in practice.

1Keep your test suite as small as possible

Whereas it is good to thoroughly test your code after you have written it, for regression testing, you want to focus on those tests that actually matter. It simply becomes infeasible to keep running hundreds of test cases for each class that you are writing, once your software code starts expanding. Therefore, once your code is thoroughly tested and deployed, you should focus on testing the part that might still break in the future, and use the tests that cover the most critical part of this code. You can manually decide on the criticality of each unit test, and e.g. annotate the most important ones as “regression tests” in your CI tool. Or you could also run a tool that tells you the impact without any effort on your side (besides clicking a button). There is some theoretical work on how to do that (for more info, see Machine Learning Applied to Software Testing: A Systematic Mapping Study), but also actual tools that (e.g. devmate) that are actively working on support for test case priorization and reduction.

2Without readability, you are lost

If a regression test that you have created several months ago now suddenly fails, the key factor that determines the value of this informaiton is in the end the readability of your test code. At that time, you most likely don’t remember why you have created that particular test case (or did not even write it yourself). So you need to revisit the test code, and find out as easily as possible where you have to look for the source of the test failure (is it maybe based on a requirement that recently changed? or which part of the code should you have a deeper look at for a potential bug?). If you don’t understand what’s going on in your test any more, you are simply lost in gaining any value from it. But luckily, there are some best practices for this that you can easily stick to to make sure your tests stay readable, and you also understand them after years of writing them (my favorite is the arrange-act-assert pattern: The AAA Test Pattern explained with C# and NUnit)

3Maintainability makes the effort

One small change in the code, and hundreds of unit tests do not compile any more. Every developer who is doing serious unit testing has seen this problem dozens of times alredy. Maintaining your test code is where the actual effort lies – especially when it comes to regression testing! The more readable your code, the easier it is obviously to maintain it. But there are also other principles (e.g. the FIRST rule: Clean Unit-Testing, What it takes to make good Unit-Tests) that influence how easy it is to change your tests after you have done a change in the code. Luckily, we have compiled the ultimate checklist for creating high-quality unit test code for you here: The Ultimate Checklist for Better Unit Testing.

Css vector created by vectorjuice – www.freepik.com