D E V M A T E

Loading

In software testing, test cases are currently defined by specifying expected output for a particular input. If this expected output does not match the actual output of a tested system, a fault is detected.

However, this usually requires the tedious and repetitive definition of input-output pairs for each case to be tested; and repetitive work usually cries for automation. 
So why not focus on describing what really matters and automate the rest (the remaining, repetitive tasks)?
Metamorphic testing [1] is one way to do that. Besides automating repetitive tasks when defining test cases, metamorphic testing is proven to find defects that were undetected with traditionally defined test cases [2], and allow application scenarios that cannot be covered by traditional testing [5, 8, 10].

In this article, we discuss (i) issues with current input-output based test case definition, (ii) describe metamorphic relations as an alternative way to define test cases and (iii) give some application scenarios for implementing those metamorphic relations in the context of metamorphic testing.

Current issues with Input-Output testing

In current software testing approaches, the behavior of a system under test is verified by adapting test inputs and creating assertions on the expected output of the system under test. Although the execution of and even the creation of code for such test cases can be automated, the definition of input and expected output must happen manually in traditional software testing.
With test suites for modern software systems including thousands of test cases, this work can become tedious, exhausting and expensive. Therefore, a lot of effort into automating this by e.g. using test oracles that describe a relationship between input and output of a test case on a meta level.

Automation is achieved by predicting expected outputs on new input data based on this oracle definition. Based on this prediction, a testing framework can automatically detect whether the result returned from the system under test is correct. However, the complexity of modern software systems usually makes it very hard for a human to define such oracles [1, 2, 3]. Especially if a huge amount of data is involved (so-called big data) [4] or inputs cannot be predetermined (e.g. in artificial intelligence) [5, 6], oracle functions are not applicable. This is also referred to as the oracle problem [7].
All of the challenges mentioned above call for a new way of defining test cases. The remainder of this article describes one option called metamorphic relations.

What are metamorphic relations?

Metamorphic relations provide an alternative way of defining test cases. In the traditional definition method (Figure 1, left), input-output pairs or oracle functions to derive test output from test input must be defined for every test case. In the alternative definition method described in this article, new test cases are derived from existing input-output pairs based on so-called metamorphic relations.
Those relations allow to automatically reason about different execution options of a system under test [2] (in other words, different test cases). Thereby, several test cases can be automatically derived by applying metamorphic relations to an existing input-output pair. Test inputs of a generated test case is derived by applying a metamorphic relation to a predefined test case, and expected test output is generated by applying such a relationship to the defined output of the predefined test case.
If a relationship between such a set of test cases is violated, a fault is detected.

                     

Figure 1: Concept of defining test cases based on input-output relations (left) and metamorphic relations (right).

The following example [2] illustrates metamorphic relations in the area of search engines.
Assume a search engine that returns a list of search results, based on some input conditions.
An example of such input condition (an example test case) can be that search results must include the term “metamorphic”. If the expected result is defined in more detail (e.g. searching for results that must include “metamorphic testing”), the actual results must be a subset of the results from the first test case. This relationship can be expressed as a metamorphic relation (input 1“metamorphic” -> input 2: “metamorphic testing”; result 1: List<x, y, z> -> result 2: subset(List<x,y,z>)).

Although using metamorphic relations provides additional effort for deriving test cases from a single predefined test case (definition of input-input relation and output-output relation, instead of a singular input-output relation in traditional test case definition), the reuse of such relations for several predefined test cases (e.g. for several search queries, or different search databases) or even across several tools and platforms (e.g. for different search machine implementations) provides huge automation potential, ultimately helping to overcome the challenges mentioned in the previous section of this article.

Application scenarios for Metamorphic Testing.

The method to apply metamorphic relations to a set of existing test cases for automation is usually referred to as metamorphic testing. In recent years, research around metamorphic testing had the focus on creating test cases of artificial intelligence (such as driverless cars [8], obstacle detection systems [9] or machine learning algorithms [5], [6]), or big data systems [4].

However, metamorphic testing was also shown to detect defects that were not found with traditionally defined test cases in software systems for different application scenarios [2], or allow for use cases that are difficult to cover with traditional test case definitions (such as quality assessment and comparison of search engines) [10]. Metamorphic testing can also be applied to different testing activities, such as performance testing [11].
The results obtained from those application scenarios, together with the automation potential of metamorphic relations described in this article, make metamorphic testing a promising alternative for traditional test case definitions.

References
[1] “Metamorphic Testing: A New Approach for Generating Next Test Cases”, Scientific publication, available via https://arxiv.org/abs/2002.12543 (last accessed 02 August 2020)
[2] “A Survey on Metamorphic Testing”, Scientific publication, available via https://www.researchgate.net/publication/296477118_A_Survey_on_Metamorphic_Testing (last accessed 02 August 2020)
[3] “Metamorphic Testing and Its Applications”, Scientific publication, available via https://pdfs.semanticscholar.org/815d/d5a682d6026d00ee640ada1d4860314adb5a.pdf (last accessed 02 August 2020)
[4] “Towards testing big data systems”, Scientific Publication, https://link.springer.com/article/10.1007/s12551-018-0492-6 (last accessed 02 August 2020)
[5] “Test your Machine Learning Algorithm with Metamorphic Testing”, article, available via https://medium.com/trustableai/testing-ai-with-metamorphic-testing-61d690001f5c (last accessed 02 August 2020)
[6] “Generating Biased Dataset for Metamorphic Testing of Machine Learning Programs”, Scientific publication, available via https://link.springer.com/chapter/10.1007/978-3-030-31280-0_4 (last accessed 02 August 2020)
[7] “The Oracle Problem in Software Testing: A Survey”, Scientific publication, available via http://www0.cs.ucl.ac.uk/staff/M.Harman/tse-oracle.pdf (last accessed 02 August 2020)
[8] “Metamorphic testing of driverless cars”, Scientific publication, available via https://dl.acm.org/doi/pdf/10.1145/3241979?casa_token=vgL3LCIzv_EAAAAA:0ULR0_mB3MEz_BXrCX2eruT0rB9KfYO_WyQWUwnUW520Y2r-yZFk8A_wrjbLI46I36Qq_r8VoP4O (last accessed 02 August 2020)
[9] “Metamorphic Testing for Object Detection Systems”, Scientific publication, available via https://arxiv.org/abs/1912.12162 (last accessed 02 August 2020)
[10] “QUALITY ASSESSMENT OF SEARCH ENGINES USING METAMORPHIC TESTING”, Scientific publication, available via https://pdfs.semanticscholar.org/ab63/f7f96e3e0a4d9655bfe4832317c75e177f64.pdf (last accessed 02 August 2020)
[11] “Performance metamorphic testing: A Proof of concept”, Scientific publication, available via https://www.sciencedirect.com/science/article/abs/pii/S095058491830017X (last accessed 02 August 2020)
Banner vector created by katemangostar – www.freepik.com

You are welcome!

Leave a Comment

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Want to experience the magic of devmate?
Join A Live Product Demo
Get Free Access Now to
9 eBooks!
All about Automated Software Testing
Proven experts
Learn to save up to 75% of your test efforts
Get Free Access Now!
Get Access Now! & Save 50%
Personal Trainer FREE Nutrition Custom Workout App
Get Access Now!
eBook Download
Enter your details to get your free ebook!
All about Automated Software Testing
Download Free Ebook
SUBSCRIBE
MY WEB
NEWSLETTERS
Lorem ipsum dolor sit amet, consectetur adipiscing