Generating complex and faulty test data through model-based mutation analysis

Testing the correct behaviour of data processing systems in the presence of faulty data is extremely expensive. The data structures processed by these systems are often complex, with many data fields and multiple constraints among them. Software engineers, in charge of testing these systems, have to handcraft complex data files or databases, while ensuring compliance with the multiple constraints to prevent the generation of trivially invalid inputs. In addition, assessing test results often means analysing complex output and log data. Though many techniques have been proposed to automatically test systems based on models, little exists in the literature to support the testing of systems where the complexity is in the data consumed in input or produced in output, with complex constraints between them. In particular, such systems often need to be tested with the presence of faults in the input data, in order to assess the robustness and behaviour of the system in response to such faults. This paper presents an automated test technique that relies upon six generic mutation operators to automatically generate faulty data. The technique receives two inputs: field data and a data model, i.e. a UML class diagram annotated with stereotypes and OCL constraints. The annotated class diagram is used to tailor the behaviour of the generic mutation operators to the fault model that is assumed for the system under test and the environment in which it is deployed. Empirical results obtained with a large data acquisition system in the satellite domain show that our approach can successfully automate the generation of test suites that achieve slightly better instruction coverage than manual testing based on domain expertise.