Theoretical considerations on scaling methodology in PISA

OECD Programme for International Student Assessment (PISA) scaling methodologies are reviewed from the mathematical perspective. In particular, the paper aims to elucidate the model structures and the model assumptions of item response theory used in the item scaling phase, the latent regression model employed in the student prior estimation phase, and the population modelling and Laplace approximation performed in the multiple imputations phase. It provides insights for maximising the impact of the innovations implemented in PISA while minimising the risk of impairing the reliability and the validity of the assessments, which is essential for maintaining technically sound international assessments. Based on the theoretical considerations, the scale robustness of PISA is confirmed against the growing number of countries/economies; nevertheless, the importance of assessing the model-data fit and investigating residual structures by sub-populations are indicated in this study. Furthermore, analysis procedures for nuisance factors such as testlet effect and locally dependent items are introduced in this paper. In summary, this study provides the theoretical considerations underpinning the stability and extensibility of the PISA scale.