The impact of growing participation in PISA on scaling outcomes : A Monte Carlo simulation study

The OECD's PISA (Programme for International Student Assessment) is expanding on new educational systems from cycle to cycle. Most of the new participants differ significantly and negatively from the core participants in the level of educational proficiency. The study has investigated a potential expansion of the new participants’ proportion from 15% of low performing countries to 50% of low-performing countries in the PISA sample. A simulation study was performed that aimed to check whether an increasing share of low performing countries among PISA participants can affect: a) key country parameters (means and within-country standard deviations), b) item parameters (difficulty and discrimination), and c) sensitivity and specificity of differential item functioning procedures. The results of the study point out that the PISA procedures are fit and robust to the increasing proportion of low-performing countries resulting in highly reliable inter country score differences and estimates of country means.