5 things you can learn from executing your test cases

I believe that most developers do not understand the full potential of unit testing. I also believe that writing unit tests does not have any sense before you know how to do it right (I’ve written about that in another post already).

Most people think that you can only write unit tests to find bugs in your implementation (which is unnecessary, because you do not create bugs, anyways).

Therefore, most articles about unit testing that are out there won’t help you, because they focus on how to create the most efficient, most beautiful, and most maintainable test cases.

In this article, I will be a bit more pragmatic. I want to give 5 particular insights you can gain from executing your test cases. This should show you the many ways that you can benefit from actually writing unit tests.

Check them out and let me know what you think in the comments section below!

 

If you implement something, and you run your tests, you should immediately see the impact of your changes on different (all?) aspects of your system. If a test begins to fail (meaning it didn’t fail before), you know that the corresponding system part has been affected.

 

If you expect some parts of the system to be affected by your code changes, but the unit tests for these system parts still pass, you know that you have to add some test cases there. This way, you can identify blind spots (parts of the system that you expect to be covered, but aren’t) in your tests. Can these blind spots exist, even if you have 100 % code coverage? The short answer is yes, but I’ve also written a more detailed post about this if you are interested.

The idea of code refactoring is to improve the code structure of a system, without changing its functionality. The main focus is thereby the improvement of maintainability. But what if after the refactoring, some unit tests start to fail (because of an incorrect behavior of the tested system part, not because interface structures have changed)? You know that you have done refactoring wrong!

 

By following test results over several execution runs, you can actually derive a lot of information on the quality of particular tests. (1) If test results change often between different executions (from fail to pass, and back to fail, …), the value of your test case is very high. You can usually find a lot of information about the defects/impacts of system changes/refactorings in these test cases. (2) If tests pass most of the time, you know that the underlying system parts are quite stable. The value of the test cases is therefore lower than in the previous case, but it is still important to identify these stable system parts). (3) If several test cases always fail at the same time, and then also go back to pass at the same time again, chances are high that they are redundant. The value of these redundant tests cases is therefore comparably low, and you can try to reduce them.

 

Usually, you expect a test case to fail, or maybe not even be executable any more, if the behavior and structure of the tested system is changed (intentionally or unintentionally). But what if the test still executes? And what if the result is even positive? Classic example: you expect an error message, but do not specifiy which one. And suddenly, your service method does not respond with the expected server error (e.g. 500), but throws a 404/408 client error, because the whole service isn’t available any more. However, these cases can still be identified. Usually, executing a test takes a similar amount of time over several test runs (this timespan is often neglectable when it comes to unit tests). If this execution time increases between test runs, this is often because the system has to work longer than usually (e.g. because the test client tries to contact the server several times). In some cases, test execution might even decrease, because the execution path is terminated earlier due to exception handling. Both ways, the change can be detected by having a look at the execution times of your test cases (or at least identifying outliers).