Search-based DNN Testing and Retraining with GAN-enhanced Simulations

In safety-critical systems (e.g., autonomous vehicles and robots), Deep Neural Networks (DNNs) are becoming a key component for computer vision tasks, particularly semantic segmentation. Further, since DNN behavior cannot be assessed through code inspection and analysis, test automation has become an essential activity to gain confidence in the reliability of DNNs. Unfortunately, state-of-the-art automated testing solutions largely rely on simulators, whose fidelity is always imperfect, thus affecting the validity of test results. To address such limitations, we propose to combine meta-heuristic search, used to explore the input space using simulators, with Generative Adversarial Networks (GANs), to transform the data generated by simulators into realistic input images. Such images can be used both to assess the DNN accuracy and to retrain the DNN more effectively. We applied our approach to a state-of-the-art DNN performing semantic segmentation, in two different case studies, and demonstrated that it outperforms a state-of-the-art GAN-based testing solution and several other baselines. Specifically, it leads to the largest number of diverse images leading to the worst DNN accuracy. Further, the images generated with our approach, lead to the highest improvement in DNN accuracy when used for retraining. In conclusion, we suggest to always integrate a trained GAN to transform test inputs when performing search-driven, simulator-based testing.