Measuring the difference between two models

Two psychometric models with very differentparametric formulas and item response functionscan make virtually the same predictions in allapplications. By applying some basic results fromthe theory of hypothesis testing and from signaldetection theory, the power of the most powerfultest for distinguishing the models can be computed.Measuring model misspecification by computingthe power of the most powerful test isproposed. If the power of the most powerful testis low, then the two models will make nearly thesame prediction in every application. If the poweris high, there will be applications in which themodels will make different predictions. Thismeasure, that is, the power of the most powerfultest, places various types of model misspecification- item parameter estimation error, multidimensionality,local independence failure, learningand/or fatigue during testing-on a common scale.The theory supporting the method is presented andillustrated with a systematic study of misspecificationdue to item response function estimation error.In these studies, two joint maximum likelihoodestimation methods (LOGIST 2B and LOGIST 5) and twomarginal maximum likelihood estimation methods(BILOG and ForScore) were contrasted by measuringthe difference between a simulation model and amodel obtained by applying an estimation methodto simulation data. Marginal estimation was foundgenerally to be superior to joint estimation. Theparametric marginal method (BILOG) was superiorto the nonparametric method only for three-parameterlogistic models. The nonparametric marginalmethod (ForScore) excelled for more generalmodels. Of the two joint maximum likelihoodmethods studied, LOGIST s appeared to be moreaccurate than LOGIST 2B. Index terms: BILOG;forced-choice experiment; ForScore; ideal observermethod; item response theory, estimation, models;LOGIST; multilinear formula score theory.