TY - JOUR
T1 - Search-based automated testing of continuous controllers
T2 - Framework, tool support, and case studies
AU - Matinnejad, Reza
AU - Nejati, Shiva
AU - Briand, Lionel
AU - Bruckmann, Thomas
AU - Poull, Claude
N1 - Publisher Copyright:
© 2014 Elsevier B.V. All rights reserved.
PY - 2015
Y1 - 2015
N2 - Context: Testing and verification of automotive embedded software is a major challenge. Software production in automotive domain comprises three stages: Developing automotive functions as Simulink models, generating code from the models, and deploying the resulting code on hardware devices. Automotive software artifacts are subject to three rounds of testing corresponding to the three production stages: Model-in-The-Loop (MiL), Software-in-The-Loop (SiL) and Hardware-in-The-Loop (HiL) testing. Objective: We study testing of continuous controllers at the Model-in-Loop (MiL) level where both the controller and the environment are represented by models and connected in a closed loop system. These controllers make up a large part of automotive functions, and monitor and control the operating conditions of physical devices. Method: We identify a set of requirements characterizing the behavior of continuous controllers, and develop a search-based technique based on random search, adaptive random search, hill climbing and simulated annealing algorithms to automatically identify worst-case test scenarios which are utilized to generate test cases for these requirements. Results: We evaluated our approach by applying it to an industrial automotive controller (with 443 Simulink blocks) and to a publicly available controller (with 21 Simulink blocks). Our experience shows that automatically generated test cases lead to MiL level simulations indicating potential violations of the system requirements. Further, not only does our approach generate significantly better test cases faster than random test case generation, but it also achieves better results than test scenarios devised by domain experts. Finally, our generated test cases uncover discrepancies between environment models and the real world when they are applied at the Hardware-in-The-Loop (HiL) level. Conclusion: We propose an automated approach to MiL testing of continuous controllers using search. The approach is implemented in a tool and has been successfully applied to a real case study from the automotive domain.
AB - Context: Testing and verification of automotive embedded software is a major challenge. Software production in automotive domain comprises three stages: Developing automotive functions as Simulink models, generating code from the models, and deploying the resulting code on hardware devices. Automotive software artifacts are subject to three rounds of testing corresponding to the three production stages: Model-in-The-Loop (MiL), Software-in-The-Loop (SiL) and Hardware-in-The-Loop (HiL) testing. Objective: We study testing of continuous controllers at the Model-in-Loop (MiL) level where both the controller and the environment are represented by models and connected in a closed loop system. These controllers make up a large part of automotive functions, and monitor and control the operating conditions of physical devices. Method: We identify a set of requirements characterizing the behavior of continuous controllers, and develop a search-based technique based on random search, adaptive random search, hill climbing and simulated annealing algorithms to automatically identify worst-case test scenarios which are utilized to generate test cases for these requirements. Results: We evaluated our approach by applying it to an industrial automotive controller (with 443 Simulink blocks) and to a publicly available controller (with 21 Simulink blocks). Our experience shows that automatically generated test cases lead to MiL level simulations indicating potential violations of the system requirements. Further, not only does our approach generate significantly better test cases faster than random test case generation, but it also achieves better results than test scenarios devised by domain experts. Finally, our generated test cases uncover discrepancies between environment models and the real world when they are applied at the Hardware-in-The-Loop (HiL) level. Conclusion: We propose an automated approach to MiL testing of continuous controllers using search. The approach is implemented in a tool and has been successfully applied to a real case study from the automotive domain.
KW - Automotive software systems
KW - Continuous controllers
KW - Model-in-The-loop testing
KW - Search-based testing
KW - Simulink models
UR - http://www.scopus.com/inward/record.url?scp=84922619944&partnerID=8YFLogxK
U2 - 10.1016/j.infsof.2014.05.007
DO - 10.1016/j.infsof.2014.05.007
M3 - Article
AN - SCOPUS:84922619944
SN - 0950-5849
VL - 57
SP - 705
EP - 722
JO - Information and Software Technology
JF - Information and Software Technology
IS - 1
ER -