Barriers and limits in test automation

The potential of Test Automation is huge. Dependencies to requirement specifications change requests and project deadlines significantly impact the success of Test Automation. Insufficient fulfilment of prerequisites often provides huge barriers to the automated execution of test cases in software projects.

• Test Automation and requirement specifications:
  Test cases are written based on software-requirements. As automation of test cases requires a formal definition of execution steps and expected behavior, this information has to be derived from requirement specifications. As in many software projects, requirements are still specified in an ad-hoc manner, written in textual language, deriving automated test cases is often impossible without further information that has to be collected from stakeholders.
  
  
• Test Automation and change requests:
  Change requests imply that the requirements on a software system change. Therefore, the test cases that are based on these requirements have to be adapted as well. Adapting test cases requires manual effort of the test automator, reducing the efficiency gained by Test Automation. Changing requirements often also lead to changes in the software interfaces that are used by automated test cases. As a consequence, even more adaption is required.
  
  
• Test Automation and Project Deadlines:
  Making automated test execution work requires time. This time is often not available when deadlines approach. Change requests and new functionality are not documented to an extent in which automated test cases can be derived. Getting information required to create new automated test cases is not possible, because stakeholders are occupied with finishing their own work before the deadline. And even if required information is available, unplanned adaptations required to make executing automated test cases may extend project budget concerning time and money.

Challenges for test automators

Test Automation comes with the promise of improving efficiency and overcoming existing challenges in software testing, making the Test Automator the hero of both testers and managers. However, (i) insufficiently formulated requirements, (ii) frequent changes in requirements and software architecture, as well as (iii) time pressure when project deadlines approach make application of Test Automation in software development projects very difficult.

Test Automators spend a huge amount of their working time with (i) chasing stakeholders to gain information required to automate test cases and (ii) adapting their testing programs to keep them up to date.

This effort, weighed against the increased efficiency created by automated test execution, dramatically reduces profitability of Test Automation approaches, often making the test automator more of a burden than a hero in software development projects.

Approaches, such as Model Based Testing, Automated Test Case Generation or Automated End-To-end Testing, are required in order to overcome these challenges and make automation in software testing profitable.

What can be done to overcome these challenges?

To overcome time pressure when project deadlines approach, we can decouple the implementation of unit tests with the modeling of test cases and the underlying requirements. Model based testing approaches help to decouple these constraints. The Test-Model can be done with stake holders.

With devmate for example the test model can be created by the stakeholder alone without any programming knowledge needed.

Model based testing also give us the opportunity to generate test code from the model automatically. Which saves up to 75% of unit test development time.