Evaluation of the efficiency of item calibration

This study compared several IRT calibration proceduresto determine which procedure, if any, consistentlyproduced the most accurate item parameter estimates.A new criterion of calibration efficiency wasused for evaluating the calibration procedures; this criterionconsiders the joint effects of individual item parametererrors as they relate to the accuracy of θ estimation.Four methods of item calibration wereevaluated: (1) heuristic estimates obtained from transformationsof traditional item statistics; (2) ANCILLES,a program that first fits the c parameter and then transformstraditional item statistics to IRT a and b parameters; (3) LOGIST, a joint maximum likelihood procedure; and (4) ASCAL, a modification of LOGIST’Salgorithm which applies Bayesian priors to the abilitiesand item parameters. These were compared with eachother and with a constant item parameter baseline condition.ASCAL and LOGIST produced estimates of essentiallyequivalent accuracy, although ASCAL’s estimatesof the c parameters were slightly superior. The heuristicestimates and those from ANCILLES were generallypoor in comparison, particularly for smaller samplesizes. Index terms: Calibration efficiency, Itemcalibration, Item parameter estimation, Item response theory, Latent trait models.