Testing as a crucial part of software development

Testing is an important activity in software development. It aims at finding incorrect implementation of requirements before software is shipped and potentially tremendous results occur. Prominent examples of such errors are the crash of the Ariane 5 skyrocket [1], the loss of the Mars Climate Orbiter by NASA [2] or the Millenium Bug [3, 4].

According to official reports (e.g. [1], [5], [6]), all these errors together with their economic consequences could have been avoided by a more intense or structured testing effort. Increasing importance of testing on the one side and higher degree of complexity and difficulty of testing on the other side, lead to a number of challenges that software testers have to face. In this article, we elaborate on the causes and nature of these challenges and summarize them.

The Industry relies more and more on software systems to improve efficiency by avoiding redundancy and automating manual tasks. In current times, this development is not just limited to managing physical and human resources or knowledge (in 2017, 81 % of organizations were using ERP software to manage resources [7]). Trends like digital twins enable automation of whole production lines and many business models are based solely on technology.

However, at the same time, a growing number of parts in our personal lives rely on software systems. This means that errors in software systems do not just imply economical risks on the vendor side. In fact, applications like autonomous driving and smart homes integrate intelligent software into core areas of our lives, with dramatic impact on system failure. This makes testing a crucial activity to ensure both vendor survival and security in human lives. Another aspect that significantly impact testing is the growing size and complexity in modern software systems. Both increase the effort which is required for ensuring software quality using testing strategies.

But what are trends that influence these aspects?

1. Larger software:
Over the past decades, software has become more and more powerful. An illustrative example is Adobe Photoshop. Whereas the first version of the software released in 1990 had 100.000 Lines of Code, the Photoshop CS 6 released in 2012 included about 4.5 million Lines of Code [8].

2. Increasing application scenarios:
Although the above-mentioned increase of 4.500 % may already seem huge, growing functionality leads to an explosion of software size in almost every area. For example, the leading Operating System for computers in 2001, named Windows XP, was comprised of 45 million Lines of Code. Nowadays, one program on a computer (example: Office 2013) can already be this large. When looking into more advanced areas, size increases even further. For instance, the codebase of an average modern car consists of around 100 million lines of code [8, 9].

3. Software systems instead of standalone tools:
Nowadays, software is not solely based on a program that is installed by a user on his or her computer. Clients like desktop programs, mobile applications or web pages only form visual representations of functionality that is accessed from different backend applications. An illustrative (yet very simple) example here is Facebook. Besides the web page, Facebook offers several clients for different devices. Whereas the Android App comprises of 9 million Lines of Code, all of their clients sum up to a total of 62 million lines of code. All of these clients access the same backend functionality, which builds the core of the software. Additionally, clients may interact with each other directly (e.g. the Facebook App opens the Facebook Messenger App on the same device). All these interactions increase the complexity of the overall software system. Current leader concerning the size of its range of application that interact with each other is Google, whose software ecosystem consists of about 2 billion lines of code [10].

4. New types of software:
Trends like Artificial Intelligence and Open Systems that can be accessed by other systems further increase complexity in software development by changing the way software is implemented and applied to solve real-world problems.

Result: increasing effort for testing:
Obviously, increasing size of software also increases the number of test cases required to ensure software quality with testing. Furthermore, increasing complexity has an even more significant impact on testing effort. Although reduced assumptions on the usage of software improve application scenarios and reusability, potential for errors increases as well. As testing cannot rely on those assumptions, more and more cases have to be considered when defining and executing test cases. This easily makes the number of test cases explode. For example, if a software program can be accessed from different clients (different web browsers, different versions of mobile devices), functionality has to be tested for all of those test cases individually.