TY - JOUR
T1 - Reliability of actual and predicted judgments across time
AU - Balzer, William K.
AU - Rohrbaugh, John
AU - Murphy, Kevin R.
PY - 1983/8
Y1 - 1983/8
N2 - In order for predictions derived from judgment models to effectively represent or replace an individual's decision-making behavior, they must show adequate levels of temporal reliability. This study examined the test-retest reliability of judgment models. Thirty-three undergraduates were asked to rate the acceptability of constructing 50 hypothetical public utility plants and to describe their judgment strategy. This decision task was replicated two weeks later. Test-retest reliabilities of (a) actual judgments, (b) predictions from a statistically derived judgment model, and (c) predictions from a subjectively reported judgment model were computed. All average test-retest reliability estimates were significantly different from zero; the average reliability of judgments predicted by regression models was significantly larger than the average reliability of actual judgments and the average reliability of judgments predicted by subjective weighting strategies. Large individual differences in all three indices of reliability were noted. Further analyses showed that groups who completed the decision task before describing their subjective policies showed significantly higher reliabilities for predictions based upon their subjective policies than did most individuals who described their subjective policies before making judgments. Overall, the results support the implicit assumption that most individuals harbor temporally stable judgment policies across time, although some caution is suggested in the use of subjectively described decision strategies.
AB - In order for predictions derived from judgment models to effectively represent or replace an individual's decision-making behavior, they must show adequate levels of temporal reliability. This study examined the test-retest reliability of judgment models. Thirty-three undergraduates were asked to rate the acceptability of constructing 50 hypothetical public utility plants and to describe their judgment strategy. This decision task was replicated two weeks later. Test-retest reliabilities of (a) actual judgments, (b) predictions from a statistically derived judgment model, and (c) predictions from a subjectively reported judgment model were computed. All average test-retest reliability estimates were significantly different from zero; the average reliability of judgments predicted by regression models was significantly larger than the average reliability of actual judgments and the average reliability of judgments predicted by subjective weighting strategies. Large individual differences in all three indices of reliability were noted. Further analyses showed that groups who completed the decision task before describing their subjective policies showed significantly higher reliabilities for predictions based upon their subjective policies than did most individuals who described their subjective policies before making judgments. Overall, the results support the implicit assumption that most individuals harbor temporally stable judgment policies across time, although some caution is suggested in the use of subjectively described decision strategies.
UR - http://www.scopus.com/inward/record.url?scp=0042342384&partnerID=8YFLogxK
U2 - 10.1016/0030-5073(83)90142-3
DO - 10.1016/0030-5073(83)90142-3
M3 - Article
AN - SCOPUS:0042342384
SN - 0030-5073
VL - 32
SP - 109
EP - 123
JO - Organizational Behavior and Human Performance
JF - Organizational Behavior and Human Performance
IS - 1
ER -