State-based testing: Industrial evaluation of the cost-effectiveness of round-trip path and sneak-path strategies

Nina Elisabeth Holt; Richard Torkar; Lionel Briand; Kai Hansen

doi:10.1109/ISSRE.2012.17

State-based testing: Industrial evaluation of the cost-effectiveness of round-trip path and sneak-path strategies

Nina Elisabeth Holt
, Richard Torkar
, Lionel Briand
, Kai Hansen

Simula Research Laboratory
Chalmers University of Technology
University of Luxembourg
ABB Group

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In the context of safety-critical software development, one important step in ensuring safe behavior is conformance testing, i.e., checking compliance between expected behavior and implementation. Round-trip path testing (RTP) is one example of conformance testing. Another essential step, however, is sneak-path testing, that is testing of how software reacts to unexpected events for a particular system state. Despite the importance of being systematic while testing, all testing activities take place, even for safety-critical software, under resource constraints. In this paper, we present an empirical evaluation of the cost-effectiveness of RTP when combined with sneak-path testing in the context of an industrial control system. Results highlight the importance of sneak-path testing since unexpected behavior is shown to be difficult to detect by other common, state-based test strategies. Results also suggest that sneak-path testing is a cost-effective supplement to RTP.

Original language	English
Title of host publication	Proceedings - 2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012
Pages	321-330
Number of pages	10
DOIs	https://doi.org/10.1109/ISSRE.2012.17
Publication status	Published - 2012
Externally published	Yes
Event	2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012 - Dallas, TX, United States Duration: 27 Nov 2012 → 30 Nov 2012

Publication series

Name	Proceedings - International Symposium on Software Reliability Engineering, ISSRE
ISSN (Print)	1071-9458

Conference

Conference	2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012
Country/Territory	United States
City	Dallas, TX
Period	27/11/12 → 30/11/12

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 9 Industry, Innovation, and Infrastructure

Keywords

Automated tool support
Cost-effectiveness
Empirical evaluation
Industrial case study
Round-trip path testing
Sneak-path testing
State-based testing
UML

Access to Document

10.1109/ISSRE.2012.17

Cite this

Holt, N. E., Torkar, R., Briand, L., & Hansen, K. (2012). State-based testing: Industrial evaluation of the cost-effectiveness of round-trip path and sneak-path strategies. In Proceedings - 2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012 (pp. 321-330). Article 6405380 (Proceedings - International Symposium on Software Reliability Engineering, ISSRE). https://doi.org/10.1109/ISSRE.2012.17

Holt, Nina Elisabeth ; Torkar, Richard ; Briand, Lionel et al. / State-based testing : Industrial evaluation of the cost-effectiveness of round-trip path and sneak-path strategies. Proceedings - 2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012. 2012. pp. 321-330 (Proceedings - International Symposium on Software Reliability Engineering, ISSRE).

@inproceedings{820ec9acd872463389798c72ee7e6dae,

title = "State-based testing: Industrial evaluation of the cost-effectiveness of round-trip path and sneak-path strategies",

abstract = "In the context of safety-critical software development, one important step in ensuring safe behavior is conformance testing, i.e., checking compliance between expected behavior and implementation. Round-trip path testing (RTP) is one example of conformance testing. Another essential step, however, is sneak-path testing, that is testing of how software reacts to unexpected events for a particular system state. Despite the importance of being systematic while testing, all testing activities take place, even for safety-critical software, under resource constraints. In this paper, we present an empirical evaluation of the cost-effectiveness of RTP when combined with sneak-path testing in the context of an industrial control system. Results highlight the importance of sneak-path testing since unexpected behavior is shown to be difficult to detect by other common, state-based test strategies. Results also suggest that sneak-path testing is a cost-effective supplement to RTP.",

keywords = "Automated tool support, Cost-effectiveness, Empirical evaluation, Industrial case study, Round-trip path testing, Sneak-path testing, State-based testing, UML",

author = "Holt, \{Nina Elisabeth\} and Richard Torkar and Lionel Briand and Kai Hansen",

year = "2012",

doi = "10.1109/ISSRE.2012.17",

language = "English",

isbn = "9780769548883",

series = "Proceedings - International Symposium on Software Reliability Engineering, ISSRE",

pages = "321--330",

booktitle = "Proceedings - 2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012",

note = "2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012 ; Conference date: 27-11-2012 Through 30-11-2012",

}

Holt, NE, Torkar, R, Briand, L & Hansen, K 2012, State-based testing: Industrial evaluation of the cost-effectiveness of round-trip path and sneak-path strategies. in Proceedings - 2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012., 6405380, Proceedings - International Symposium on Software Reliability Engineering, ISSRE, pp. 321-330, 2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012, Dallas, TX, United States, 27/11/12. https://doi.org/10.1109/ISSRE.2012.17

State-based testing: Industrial evaluation of the cost-effectiveness of round-trip path and sneak-path strategies. / Holt, Nina Elisabeth; Torkar, Richard; Briand, Lionel et al.
Proceedings - 2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012. 2012. p. 321-330 6405380 (Proceedings - International Symposium on Software Reliability Engineering, ISSRE).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - State-based testing

T2 - 2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012

AU - Holt, Nina Elisabeth

AU - Torkar, Richard

AU - Briand, Lionel

AU - Hansen, Kai

PY - 2012

Y1 - 2012

N2 - In the context of safety-critical software development, one important step in ensuring safe behavior is conformance testing, i.e., checking compliance between expected behavior and implementation. Round-trip path testing (RTP) is one example of conformance testing. Another essential step, however, is sneak-path testing, that is testing of how software reacts to unexpected events for a particular system state. Despite the importance of being systematic while testing, all testing activities take place, even for safety-critical software, under resource constraints. In this paper, we present an empirical evaluation of the cost-effectiveness of RTP when combined with sneak-path testing in the context of an industrial control system. Results highlight the importance of sneak-path testing since unexpected behavior is shown to be difficult to detect by other common, state-based test strategies. Results also suggest that sneak-path testing is a cost-effective supplement to RTP.

AB - In the context of safety-critical software development, one important step in ensuring safe behavior is conformance testing, i.e., checking compliance between expected behavior and implementation. Round-trip path testing (RTP) is one example of conformance testing. Another essential step, however, is sneak-path testing, that is testing of how software reacts to unexpected events for a particular system state. Despite the importance of being systematic while testing, all testing activities take place, even for safety-critical software, under resource constraints. In this paper, we present an empirical evaluation of the cost-effectiveness of RTP when combined with sneak-path testing in the context of an industrial control system. Results highlight the importance of sneak-path testing since unexpected behavior is shown to be difficult to detect by other common, state-based test strategies. Results also suggest that sneak-path testing is a cost-effective supplement to RTP.

KW - Automated tool support

KW - Cost-effectiveness

KW - Empirical evaluation

KW - Industrial case study

KW - Round-trip path testing

KW - Sneak-path testing

KW - State-based testing

KW - UML

UR - https://www.scopus.com/pages/publications/84876350634

U2 - 10.1109/ISSRE.2012.17

DO - 10.1109/ISSRE.2012.17

M3 - Conference contribution

AN - SCOPUS:84876350634

SN - 9780769548883

T3 - Proceedings - International Symposium on Software Reliability Engineering, ISSRE

SP - 321

EP - 330

BT - Proceedings - 2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012

Y2 - 27 November 2012 through 30 November 2012

ER -

Holt NE, Torkar R, Briand L, Hansen K. State-based testing: Industrial evaluation of the cost-effectiveness of round-trip path and sneak-path strategies. In Proceedings - 2012 IEEE 23rd International Symposium on Software Reliability Engineering, ISSRE 2012. 2012. p. 321-330. 6405380. (Proceedings - International Symposium on Software Reliability Engineering, ISSRE). doi: 10.1109/ISSRE.2012.17

State-based testing: Industrial evaluation of the cost-effectiveness of round-trip path and sneak-path strategies

Abstract

Publication series

Conference

UN SDGs

Keywords

Access to Document

Other files and links

Fingerprint

Cite this