Interrater agreement in SPICE-based assessments: Some preliminary results

K. El Emam, D. R. Goldenson, L. Briand, P. Marshall

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

The international SPICE Project intends to deliver an ISO standard on software process assessment. This project is unique in software engineering standards in that there is a set of empirical trials, the objectives of which are to evaluate the prospective standard and provide feedback before standardization. One of the enduring issues being evaluated during the trials is the reliability of assessments based on SPICE. One element of reliability is the extent to which different teams assessing the same processes produce similar ratings when presented with the same evidence. We present some preliminary results from two assessments conducted during the SPICE trials. In each of these assessments two independent teams performed the same ratings. The results indicate that in general there is at least moderate agreement between the two teams in both cases. When we take into account the severity of disagreement then the extent of agreement between the two teams is almost perfect. Also, our results indicated that interrater agreement is not the same for different SPICE processes. The findings reported in this paper provide guidance for future studies of interrater agreement in the SPICE trials and also indicate some potential issues that need to be considered within the prospective standard.

Original languageEnglish
Title of host publicationProceedings - 4th International Conference on the Software Process
Subtitle of host publicationImprovement and Practice, ICSP 1996
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages149-156
Number of pages8
ISBN (Electronic)0818677198, 9780818677199
DOIs
Publication statusPublished - 1996
Externally publishedYes
Event4th International Conference on the Software Process: Improvement and Practice, ICSP 1996 - Brighton, United Kingdom
Duration: 2 Dec 19966 Dec 1996

Publication series

NameProceedings - 4th International Conference on the Software Process: Improvement and Practice, ICSP 1996

Conference

Conference4th International Conference on the Software Process: Improvement and Practice, ICSP 1996
Country/TerritoryUnited Kingdom
CityBrighton
Period2/12/966/12/96

Fingerprint

Dive into the research topics of 'Interrater agreement in SPICE-based assessments: Some preliminary results'. Together they form a unique fingerprint.

Cite this