Fitting polytomous item response theory models to multiple-choice tests

This study examined how well current softwareimplementations of four polytomous item responsetheory models fit several multiple-choice tests. Themodels were Bock’s (1972) nominal model,Samejima’s (1979) multiple-choice Model C, Thissen& Steinberg’s (1984) multiple-choice model, andLevine’s (1993) maximum-likelihood formula scoringmodel. The parameters of the first three of these modelswere estimated with Thissen’s (1986) MULTILOGcomputer program; Williams & Levine’s (1993)FORSCORE program was used for Levine’s model. Testsfrom the Armed Services Vocational Aptitude Battery, the Scholastic Aptitude Test, and the American CollegeTest Assessment were analyzed. The models were fit inestimation samples of approximately 3,000; cross-validationsamples of approximately 3,000 were used toevaluate goodness of fit. Both fit plots and X² statisticswere used to determine the adequacy of fit. Bock’smodel provided surprisingly good fit; adding parametersto the nominal model did not yield improvementsin fit. FORSCORE provided generally good fit forLevine’s nonparametric model across all tests. Indexterms: Bock’s nominal model, FORSCORE, maximumlikelihood formula scoring, MULTILOG, polytomous IRT.