The internal and external optimality of decisions based on tests

In applied measurement, test scores are usuallytransformed to decisions. Analogous to classicaltest theory, the reliability of decisions has been definedas the consistency of decisions on a test and aretest or on two parallel tests. Coefficient kappa(Cohen, 1960) is used for assessing the consistencyof decisions. This coefficient has been developed forassessing agreement between nominal scales. It isargued that the coefficient is not suited for assessingconsistency of decisions. Moreover, it is arguedthat the concept consistency of decisions is notappropriate for assessing the quality of a decisionprocedure. It is proposed that the concept consistencyof decisions be replaced by the conceptoptimality of the decision procedure. Two types ofoptimality are distinguished. The internal optimalityis the risk of the decision procedure with respectto the true score the test is measuring. The externaloptimality is the risk of the decision procedure withrespect to an external criterion. For assessing theoptimality of a decision procedure, coefficient delta(van der Linden & Mellenbergh, 1978), which canbe considered a standardization of the Bayes risk orexpected loss, can be used. Two loss functions aredealt with: the threshold and the linear loss functions.Assuming psychometric theory, coefficientdelta for internal optimality can be computed fromempirical data for both the threshold and the linearloss functions. The computation of coefficient deltafor external optimality needs no assumption of psychometrictheory. For six tests coefficient delta asan index for internal optimality is computed forboth loss functions; the results are compared withcoefficient kappa for assessing the consistency ofdecisions with the same tests.