Toward Just-in-Time and Language-Agnostic Mutation Testing

Mutation Testing is a popular approach to determine the quality of a suite of unit tests. It is based on the idea that introducing faults into a system under test (SUT) should cause tests to fail, otherwise, the test suite might be of insufficient quality. In the language of mutation testing, such a fault is referred to as “mutation”, and an instance of the SUT’s code that contains the mutation is referred to as “mutant”. Mutation testing is computationally expensive and time-consuming. Reasons for this include, for example, a high number of mutations to consider, interrelations between these mutations, and mutant-associated costs such as the cost of mutant creation or the cost of checking whether any tests fail in response. Furthermore, implementing a reliable tool for automatic mutation testing is a significant effort for any language. As a result, mutation testing is only available for some languages.

Present mutation tools often rely on modifying code or binary executables. We refer to this as “ahead-of-time” mutation testing. Oftentimes, they neither take dynamic information that is only available at run-time into account nor alter program behavior at run-time. However, mutating via the latter could save costs on mutant creation: If the corresponding module of code is compiled, only the mutated section of code needs to be recompiled. Run-time information (like previous execution results selected by an initial test run) could help to determine the utility of a mutant. Skipping mutants of low utility could have an impact on mutation testing efficiency. We refer to this approach as just-in-time mutation testing.

In this paper, we provide a proof of concept for just-in-time and language-agnostic mutation testing. We present preliminary results of a feasibility study that explores the implementation of just-in-time mutation testing based on Truffle’s instrumentation API. Based on these results, future research can evaluate the implications of just-in-time and language-agnostic mutation testing.