Solving measurement problems with an answer-until-correct scoring procedure

Answer-until-correct (AUC) tests have been in usefor some time. Pressey (1950) pointed to their advantagesin enhancing learning, and Brown (1965)proposed a scoring procedure for AUC tests thatappears to increase reliability (Gilman & Ferry,1972; Hanna, 1975). This paper describes a newscoring procedure for AUC tests that (1) makes itpossible to determine whether guessing is at random,(2) gives a measure of how "far away" guessingis from being random, (3) corrects observed testscores for partial information, and (4) yields a measureof how well an item reveals whether an examineeknows or does not know the correct response.In addition, the paper derives the optimallinear estimate (under squared-error loss) of truescore that is corrected for partial information, aswell as another formula score under the assumptionthat the Dirichlet-multinomial model holds. Oncecertain parameters are estimated, the latter formulascore makes it possible to correct for partial informationusing only the examinee’s usual number-correctobserved score. The importance of this formulascore is discussed. Finally, various statisticaltechniques are described that can be used to checkthe assumptions underlying the proposed scoringprocedure.