Stages of Test Automation

In this article, we introduce three different stages of Test Automation, as originally proposed by Mike Cohn [1], and give characteristics of test cases in each stage.

As naming and definition of the individual stages differ in the literature, we briefly describe the intentions behind each sort of test cases before giving a detailed description of the characteristics and challenges of each stage.

In the Unit Testing stage, unit tests are implemented by the developer during implementation to ensure that the developed component satisfies expected functionality. After several of such components are finished, test automation can proceed to the Integration Testing stage. There, the correct interplay of these components is verified on a code level using integration tests. In the Frontend Testing stage, another layer is introduced. Automated frontend tests are not run against code anymore, but directly access the user frontend of a software. Compared to integration tests, the correct communication between the frontend and the already tested backend functionality is verified.

Automated Unit Testing (unit tests)

In this phase, unit tests are created. They are executed on the lowest abstraction level of a software system. Isolated units of the implemented code are tested. As those units are usually only machine-accessible, these unit tests can only be executed using test automation. Isolation of individual components allow to track the source of a test failure.

The biggest challenge in creating unit tests is to truly isolate the unit under test.

A unit within a software system usually depends on many other units. Data is read from another unit, or sent to another unit for further processing, or processing functionality is used from another unit. However, these dependencies have to be simulated to ensure an isolated view of the unit under test.
The tester has to ensure this isolation by:
1. manually providing test data
2. simulating dependencies by using mocks and stubs

Automated Integration Testing (integration tests)

Unit tests are a good way to verify functionality of the individual components of a system. However, when these components are integrated, new potential for test cases arise.

Therefore, integration tests are created in the integration testing phase. The correct integration can be checked in one of the following types of integration test cases:

1. side-effect tests: are executed using different units that are intended to interact with each other. Usually, an action is taken with unit a, and the corresponding side effect in unit b is verified (e.g. creating a new user and verifying whether it is available via a particular search procedure).
2. abstract input-output tests: are executed on a higher level of abstraction (e.g. against a service method). It is verified whether the expected result is returned (e.g. trying to create an invalid user and verify that the correct error message is returned).

The biggest challenge when creating automated unit tests is to manage the dependencies to different components or systems.
As the system under test is not only comprised of code that is fully controlled by the person writing the test code, changing requirements or unavailability of a component is more difficult to grasp and fix. This directly impacts the correct functionality of integration test cases.
Additionally, the interaction with different types of components increases complexity. The tester not only interacts with the code via common interfaces but also has to set up and interact with required components like data stores or webservices.
Ensuring reproducibility of test results is also more complex. As test cases are executed in a real environment, interferences with system interactions by other users (e.g. manually executed test cases) might directly influence the result of a test case.

The above-mentioned issues arise because of a lack of control of the tested software components. However, integration is not restricted only to components of a system. If several independently developed systems are integrated, that are maybe even developed by different companies, control over the system is even weaker. This further increases the impact of above-mentioned challenges on the complexity of creating correct automated integration tests.

Automated Frontend Testing (frontend tests)

This phase is built on top of the preceding ones. It uses frontend tests to test system behavior on a more user-centered level. Automated frontend tests abstract from the underlying code. Automated test cases of this stage are run directly against the frontend of a software system. Whereas unit and integration tests usually consider the system under test as a white box, in which the internal structure (i.e. the software conde) is taken into account for testing, frontend tests consider the tested software as a black box, where only input (via frontend interactions) and output (via frontend feedback) is taken into account. In other words, tests automatically execute actions that are usually performed by a user or by a tester when executing manual test cases. The underlying structure of the system is thereby ignored.

One big challenge when creating and executing automated frontend tests is to mimic user actions.

Interaction with the system cannot be triggered by directly calling internal code. Visual elements provided by the user interface have to be used. However, as these visual elements are usually intended for interaction by a human, accessing them is often challenging.

Conclusion

Considering those stages when writing test cases can help in writing better adaptable code because of a clearer purpose of created tests. Understanding the characteristics of each stage and the corresponding type of test cases also helps to better prepare for challenges that arise when creating tests.

written by Daniel Lehner