Current research shows that the way we think of testing is wrong. Building user interface tests on top of service or integration tests, which are again built on top of unit tests, is proven to be inefficient when being applied to modern software projects. In this article, we describe the issues that come with the traditional way of separating test cases and come up with an alternative solution. Before that, however, we start with an introduction to the concept of the test pyramid that led to the current way we think of test case types.

The Test Pyramid

The principle of the test pyramid was initially proposed by Mike Cohn in his Book “Succeeding with Agile” [1]. Although several aspects of this traditional test pyramid are often criticized, the concept of dividing test cases into different layers has become a fundamental aspect of test automation. The basic idea is that the test pyramid consists of different types of layers, in which each layer represents a particular type of test case. The level of the layer in the pyramid indicates the speed of test execution and the required isolation of the system part to be tested by a particular test case. On the bottom of the pyramid, there are unit tests that run fast and strongly isolate the tested system parts. On the top there are UI tests that are slow and require little isolation. Between those two layers, integration or service tests are situated (the naming depends on the used definition of the test pyramid). The layers of the pyramid also indicate the number of test cases needed for a particular layer.
When creating test cases, the testing pyramid is ascended from its bottom to the top. After each unit is fully tested, the correct integration of several units can be tested to find defects that do not show up on a unit level. After this integration is tested, additional defects are exploited by running test cases against a user interface. As most defects should already be found at an earlier stage and test cases on a higher layer of the pyramid only have to cover cases that are not checked on a lower level, the number of required test cases decreases when ascending the testing pyramid.

The problem with the current separation of test types

The separation of test cases into unit tests, integration tests and user interface tests is an essential concept of test automation. However, after its initial publication, this separation (the core of the testing pyramid, regardless of any extensions that have been made over the years) has been unchanged for decades now, although software development has evolved tremendously in those years. In fact, current research [2] shows that executing integration tests on top of unit tests do not yield significant improvement in defect detection, compared to running unit tests alone. In other words, integration tests do not find defects that cannot be found by unit testing in modern software systems. If what this research shows is true, there is a high chance that integration and user interface tests are also not as different as they were decades ago, when it comes to defect detection. So, why still distinguishing between those types test cases at all, if unit tests alone might do the necessary job?

Next-generation test case definition

If traditional separations of unit test types are not applicable to modern software projects, what are the alternatives? A promising suggestion is to divide test cases by the types of faults that are intended to be detected [3]. A taxonomy of an example alternative separation is provided in the figure below.

On a top level, test cases can be divided into (i) verifying the correct handling of an error of the system under test, (ii) checking for the correct output based on some input and (iii) verifying the correctness of side effects in the system under test.
For the case of error handling, test cases can be further distinguished based on the granularity of the test case. In one case, it can be checked whether the correct type of exception is thrown. In another set of test cases, the correctness of the returned exception message can be thrown. In a third test case set, the correct response of the system under test to malicious input can be checked in general (e.g. shutdown of parts of the system, correct functioning of other parts, …).
Test cases checking for correct output can be further separated by whether primitive datatypes, complex datatypes or collections are either passed as input or expected as output of the tested part of the system.
When testing for side effects, test cases can be further divided into cases that require a particular setting to be present before test execution and cases that verify whether a particular setting is present after test execution. Further separation is possible based on whether a particular setting must be present in the same unit as the test is run against (e.g. some object variables of a class instance have changed), in a different unit in the same system (e.g. some object variables of a different class instance have changed), or in a completely different system (e.g. the content of a database has changed, after some action is performed on the frontend).

Obviously, this taxonomy is only a proposal that must be validated in practice. But even if those proposed separation types turn out to be a bad example, a new generation of test case definition will be necessary to avoid a mass of redundant test case definitions in modern software projects.


[1] “Succeeding with Agile: Software Development Using Scrum”, Book, available via https://www.amazon.com/gp/product/0321579364 (last accessed 2020-July-17)

[2] “Are Unit and Integration Test Definitions Still Valid for Modern Java Projects? An Empirical Study on Open-Source Projects”, Scientific Publication, available via https://www.swe.informatik.uni-goettingen.de/publications/are-unit-and-integration-test-definitions-still-valid-modern-java-projects-impirical (last accessed 2020-July-17)

[3] “Introduction to Software Testing”, Book, available via https://books.google.at/books?hl=de&lr=&id=58LeDQAAQBAJ (last accessed 2020-July-17)

 People vector created by pch.vector – www.freepik.com

1 Comment

Leave a Comment

We use cookies to give you the best online experience. By agreeing you accept the use of cookies in accordance with our cookie policy.

Privacy Settings saved!
Privacy Settings

When you visit any web site, it may store or retrieve information on your browser, mostly in the form of cookies. Control your personal Cookie Services here.

GetResponse, Google Analytics

We use LinkedIn Insight for marketing purposes. You can disable these cookies.

We use Google Analytics for marketing purposes. You can disable these cookies.
  • __utmz
  • __utma
  • _ga
  • _gat

We use GetResponse for marketing purposes. This service cannot be disabled, otherwise the website functions will be limited.

Decline all Services
Accept all Services
Get Free Access Now to
9 eBooks!
All about Automated Software Testing
Proven experts
Learn to save up to 75% of your test efforts
Get Free Access Now!
Get Access Now! & Save 50%
Personal Trainer FREE Nutrition Custom Workout App
Get Access Now!
eBook Download
Enter your details to get your free ebook!
All about Automated Software Testing
Download Free Ebook
Lorem ipsum dolor sit amet, consectetur adipiscing