TY - GEN
T1 - Interrater agreement in SPICE-based assessments
T2 - 4th International Conference on the Software Process: Improvement and Practice, ICSP 1996
AU - El Emam, K.
AU - Goldenson, D. R.
AU - Briand, L.
AU - Marshall, P.
N1 - Publisher Copyright:
© 1996 IEEE.
PY - 1996
Y1 - 1996
N2 - The international SPICE Project intends to deliver an ISO standard on software process assessment. This project is unique in software engineering standards in that there is a set of empirical trials, the objectives of which are to evaluate the prospective standard and provide feedback before standardization. One of the enduring issues being evaluated during the trials is the reliability of assessments based on SPICE. One element of reliability is the extent to which different teams assessing the same processes produce similar ratings when presented with the same evidence. We present some preliminary results from two assessments conducted during the SPICE trials. In each of these assessments two independent teams performed the same ratings. The results indicate that in general there is at least moderate agreement between the two teams in both cases. When we take into account the severity of disagreement then the extent of agreement between the two teams is almost perfect. Also, our results indicated that interrater agreement is not the same for different SPICE processes. The findings reported in this paper provide guidance for future studies of interrater agreement in the SPICE trials and also indicate some potential issues that need to be considered within the prospective standard.
AB - The international SPICE Project intends to deliver an ISO standard on software process assessment. This project is unique in software engineering standards in that there is a set of empirical trials, the objectives of which are to evaluate the prospective standard and provide feedback before standardization. One of the enduring issues being evaluated during the trials is the reliability of assessments based on SPICE. One element of reliability is the extent to which different teams assessing the same processes produce similar ratings when presented with the same evidence. We present some preliminary results from two assessments conducted during the SPICE trials. In each of these assessments two independent teams performed the same ratings. The results indicate that in general there is at least moderate agreement between the two teams in both cases. When we take into account the severity of disagreement then the extent of agreement between the two teams is almost perfect. Also, our results indicated that interrater agreement is not the same for different SPICE processes. The findings reported in this paper provide guidance for future studies of interrater agreement in the SPICE trials and also indicate some potential issues that need to be considered within the prospective standard.
UR - http://www.scopus.com/inward/record.url?scp=84871239700&partnerID=8YFLogxK
U2 - 10.1109/ICSP.1996.565032
DO - 10.1109/ICSP.1996.565032
M3 - Conference contribution
AN - SCOPUS:84871239700
T3 - Proceedings - 4th International Conference on the Software Process: Improvement and Practice, ICSP 1996
SP - 149
EP - 156
BT - Proceedings - 4th International Conference on the Software Process
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 2 December 1996 through 6 December 1996
ER -