In the mutation testing paradigm, test suite quality is measured by its ability to detect variant programs generated through application of random changes to an original program. In evolutionary biology however, neutral mutations that leave fitness unchanged are considered to be beneficial---improving the system's robustness and ability to discover evolutionary improvements. In this paper, we generate a population of variant programs from an original program by applying lightweight random mutations. We adopt biological terminology and refer to undetected variants as neutral, and the percentage of all variants that are neutral as mutational robustness. Although they are related to equivalent mutants in mutation testing, which are viewed as problematic, we show positive properties of neutral variants which are easily generated and can be used to protect software against unknown defects.
Even when mutations are restricted to statements executed by the test suit, we find that mutational robustness is high: 36.75% on a corpus of 22 programs including production software projects, the Siemens benchmark, and a few specially constructed programs. The results hold for mutations at the source code and assembly instruction levels, across various programming languages, and are not fully explained by test suite quality. We conclude that mutational robustness is an inherent property of software, existing even when a program is correct according to its specification. Rather than an indicator of test suite inadequacy, it is an opportunity to create useful proactive diversity. To demonstrate this, we generate and select diverse populations of neutral program variants. For a program with seven or more held-out latent bugs, we can, on average, construct and select seven neutral variants such that at least one fixes a latent bug.
http://arxiv.org/abs/1204.4224
