TY - GEN
T1 - Testing autonomous cars for feature interaction failures using many-objective search
AU - Abdessalem, Raja Ben
AU - Panichella, Annibale
AU - Nejati, Shiva
AU - Briand, Lionel C.
AU - Stifter, Thomas
N1 - Publisher Copyright:
© 2018 Association for Computing Machinery.
PY - 2018/9/3
Y1 - 2018/9/3
N2 - Complex systems such as autonomous cars are typically built as a composition of features that are independent units of functionality. Features tend to interact and impact one another's behavior in unknown ways. A challenge is to detect and manage feature interactions, in particular, those that violate system requirements, hence leading to failures. In this paper, we propose a technique to detect feature interaction failures by casting this problem into a search-based test generation problem. We define a set of hybrid test objectives (distance functions) that combine traditional coverage-based heuristics with new heuristics specifically aimed at revealing feature interaction failures. We develop a new search-based test generation algorithm, called FITEST, that is guided by our hybrid test objectives. FITEST extends recently proposed many-objective evolutionary algorithms to reduce the time required to compute fitness values. We evaluate our approach using two versions of an industrial self-driving system. Our results show that our hybrid test objectives are able to identify more than twice as many feature interaction failures as two baseline test objectives used in the software testing literature (i.e., coverage-based and failure-based test objectives). Further, the feedback from domain experts indicates that the detected feature interaction failures represent real faults in their systems that were not previously identified based on analysis of the system features and their requirements.
AB - Complex systems such as autonomous cars are typically built as a composition of features that are independent units of functionality. Features tend to interact and impact one another's behavior in unknown ways. A challenge is to detect and manage feature interactions, in particular, those that violate system requirements, hence leading to failures. In this paper, we propose a technique to detect feature interaction failures by casting this problem into a search-based test generation problem. We define a set of hybrid test objectives (distance functions) that combine traditional coverage-based heuristics with new heuristics specifically aimed at revealing feature interaction failures. We develop a new search-based test generation algorithm, called FITEST, that is guided by our hybrid test objectives. FITEST extends recently proposed many-objective evolutionary algorithms to reduce the time required to compute fitness values. We evaluate our approach using two versions of an industrial self-driving system. Our results show that our hybrid test objectives are able to identify more than twice as many feature interaction failures as two baseline test objectives used in the software testing literature (i.e., coverage-based and failure-based test objectives). Further, the feedback from domain experts indicates that the detected feature interaction failures represent real faults in their systems that were not previously identified based on analysis of the system features and their requirements.
KW - Automotive systems
KW - Feature interaction problem
KW - Many-objective optimization
KW - Search-based software testing
UR - http://www.scopus.com/inward/record.url?scp=85056528498&partnerID=8YFLogxK
U2 - 10.1145/3238147.3238192
DO - 10.1145/3238147.3238192
M3 - Conference contribution
AN - SCOPUS:85056528498
T3 - ASE 2018 - Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering
SP - 143
EP - 154
BT - ASE 2018 - Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering
A2 - Kastner, Christian
A2 - Huchard, Marianne
A2 - Fraser, Gordon
PB - Association for Computing Machinery, Inc
T2 - 33rd IEEE/ACM International Conference on Automated Software Engineering, ASE 2018
Y2 - 3 September 2018 through 7 September 2018
ER -