Scalability analysis of grammatical evolution based test data generation

Heuristic-based search techniques have been increasingly used to automate different aspects of software testing. Several studies suggest that variable interdependencies may exist in branching conditions of real-life programs, and these dependencies result in the need for highly precise data values (such as of the form i=j=k) for code coverage analysis. This requirement makes it very difficult for Genetic Algorithm (GA)-based approach to successfully search for the required test data from vast search spaces of real-life programs. Ariadne is the only Grammatical Evolution (GE)-based test data generation system, proposed to date, that uses grammars to exploit variable interdependencies to improve code coverage. Ariadne has been compared favourably to other well-known test data generation techniques in the literature; however, its scalability has not yet been tested for increasingly complex programs. This paper presents the results of a rigorous analysis performed to examine Ariadne's scalability. We also designed and employed a large set of highly scalable 18 benchmark programs for our experiments. Our results suggest that Ariadne is highly scalable as it exhibited 100% coverage across all the programs of increasing complexity with significantly smaller search costs than GA-based approaches, which failed even with huge search budgets.