Complex composites: Issues that arise in combining different modes of assessment

Data from the California Learning Assessment Systemare used to examine certain characteristics of testsdesigned as the composites of items of different modes.The characteristics include rater severity, test information,and definition of the latent variable. Three differentassessment modes-multiple-choice, open-ended, andinvestigation items (the latter two are referred to as performance-based modes)-were combined in a test acrossthree different test forms. Rater severity was investigatedby incorporating a rater parameter for each rater inan item response model that then was used to analyzethe data. Some rater severities were found to be quiteextreme, and the impact of this variation in rater severitieson both total scores and trait level estimates was examined.Within-rater variation in rater severity also wasexamined and was found to have significant variation.The information contribution of the three modes wascompared. Performance-based items provided more informationthan multiple-choice items and also providedgreatest precision for higher levels of the latent variable.A projection-like method was applied to investigate theeffects of assessment mode on the definition of the latentvariable. The multiple-choice items added informationto the performance-based variable. The results ofthe analysis also showed that the projection-like methoddid not practically differ from the results when the latenttrait was defined jointly by both the multiple-choice andthe performance-based items. Index terms: equating,linking, multiple assessment modes, polytomous item responsemodels, rater effects.