An empirical study of the effects of small datasets and varying prior variances on item parameter estimation in BILOG

Long-standing difficulties in estimating itemparameters in item response theory (IRT) have beenaddressed recently with the application of Bayesianestimation models. The potential of these methodsis enhanced by their availability in the BILOG computerprogram. This study investigated the abilityof BILOG to recover known item parameters undervarying conditions. Data were simulated for a two-parameterlogistic IRT model under conditions ofsmall numbers of examinees and items, and differentvariances for the prior distributions of discriminationparameters. The results suggest that forsamples of at least 250 examinees and 15 items,BILOG accurately recovers known parameters usingthe default variance. The quality of the estimationsuffers for smaller numbers of examinees under thedefault variance, and for larger prior variances ingeneral. This raises questions about how practitionersselect a prior variance for small numbers ofexaminees and items. Index terms: BILOG, itemparameter estimation, item response theory, parameterrecovery, prior distributions, simulation.