Computerized mastery testing with nonequivalent testlets

A procedure for determining the effect of testletnonequivalence on the operating characteristics ofa testlet-based computerized mastery test (CMT) isintroduced. The procedure involves estimating theCMT decision rule twice-once with testlets treatedas equivalent and once with testlets treated asnonequivalent. In the equivalent testlet mode, thelikelihood functions (LFS) estimated for specificnumber-correct (NC) scores are assumed to be constantacross testlets and a single set of cutscores isestimated for all testlets. In the nonequivalenttestlet mode, the LFS estimated for specific NCscores are allowed to vary from one testlet to anotherand a different set of cutscores is estimated foreach permutation of testlet presentation order.Small differences between the estimated operatingcharacteristics of the equivalent testlet decision ruleand the nonequivalent testlet decision rule indicatethat the assumption of equivalent testlets was warranted.This procedure is demonstrated with datafrom the Architect Registration Examination.Index terms: Bayesian methods, computerized masterytesting, decision theory, item response theory, testequivalence, testlets.