Define once, reuse always – the power of mutants in Software Testing

When defining test cases, a lot of manual effort is spent on repeatedly deriving tests based on predefined methods (e.g. Boundary Value Analysis), with the aim to maximize coverage of created test suites. As white-box evaluation metrics, such as code coverage, are based the syntax of the system under test (and are often criticized for this), methods for creating test cases also usually rely on syntactical information. Although including semantic data might provide benefits for the efficiency of created test cases, and evaluation of test cases based on semantics might lead to more representative results, implementing such approaches is quite challenging.

Mutation-Based Testing is an approach to overcome such challenges. In this article, we will discuss basic characteristics of Mutation-Based Testing, different applications of the general concept of Mutation to Software Testing, as well as their advantages.

Mutation Based Testing

The basic idea of mutants in software testing is to create variants of some information (so-called mutants). Those mutants are defined on a meta-level, so that they can be reapplied to a large number of cases. Mutants can be created based on formal grammar definitions [1] or models [2, 3]. Let’s look at an example. A typical functionality of a search engine is to merge search results. E.g. if a search string “Mutation Testing” is entered, results for “Mutation” are searched, results for “Testing” are searched, and then, the two result sets are merged. In this particular case, a merge is carried out based on the common subset (results that include both “Mutation” and “Testing”). For a search string such as “Mutation” “Testing”, a merge might be carried out based on the superset of the individual results (results that include at least one of the terms “Mutation” or “Testing”). These mentioned merge rules can be used for an “original” implementation of a search engine. A mutant of this search engine can be created by e.g. swapping the functionality for (using subset merging for “Mutation” Testing” and superset merging for “Mutation Testing”). This example mutation is also visualized in the figure below.

Applications of Mutation in Software Testing

The concept of Mutation can be applied to any software artefact in the development process. This includes code, specification, xml/json-files, but also test cases. However, the main distinction in the purpose of mutation can be made between inserting mutants into the system under test, and inserting mutants into test cases. A detailed description of those two types is given below:

• When mutating the system to be tested (on different levels of abstractions, such as code, or specification), the intention is usually to evaluate the efficiency of test cases. Therefore, system mutants are created by introducing faults into the original implementation (as demonstrated in the example of the previous section). The efficiency of a set of test cases is then indicated by calculating the percentage of detected mutants. This value is called the mutant score. For instance, if 150 faults are introduced into a system and 75 are detected by a certain set of test cases, their mutation score is 50 %. The higher this score, the better the efficiency of the test cases.

• When mutating test cases, the intention is usually to create a large set of test cases based on a small set of reference tests. Generic mutant definition allows to create a large set of test cases by re-applying mutation rules to several reference tests. In a search engine, an example mutation would be the negation of a search result. Every time a test case uses a search string, this test case can also be applied with the negation of this search string (e.g. search for terms including “mutation”, and search for terms that do not include “mutation”). This mutation rule can be applied to several search strings, but also to different parts of a system, or even different systems that provide search functionality. With metamorphic testing [4], even the result of such „mutated“ test cases can be calculated automatically from predefined rules.

Advantages compared to existing approaches

The semantic value introduced with system mutation makes mutation score a valuable alternative for white-box test case evaluation techniques that solely rely on syntactic information (e.g. code coverage). The semantics introduced by mutant definition even allow predictions of mutation score using machine learning, without requiring to actually execute mutants or test cases [5]. The additional effort for introducing mutants into a system to be tested also pays off when reusing mutant definitions for different parts of the system or even different systems.

However, the most outstanding advantage of mutation in software testing is when test cases are automatically created based on reusable mutation definitions. This allows a tester to focus on the most important and interesting aspects of her job, the invention of new kinds of test cases, sparing the endless time for repetitive test case definitions.