Constructing Tests that Can Measure and Compare the Maths and Physics Skills of Engineering Students in Russia and China

Although the number of engineering graduates has expanded rapidly in the last two decades, relatively little is known about the quality of engineering programs worldwide. In particular, few studies look at differences in the degree to which students are learning skills across different engineering programs within and between countries. There is particular interest in the investigation of the engineering education quality in the countries with the rapidly growing economy, such as BRICS countries. Until now, there was little research in this field and one of the main reasons for this is the difficulty in developing an assessment approach and the accompanying set of instruments, which would allow for measurement and international comparison. Our study describes a set of procedures for developing such an assessment framework of instruments, to measure and compare skill levels and gains across engineering programs. We first describe a systematic approach for constructing cross-nationally comparable instruments in maths and physics for students in the first two years of their undergraduate engineering programs. The approach includes both a priori procedures (including expert assessments to avoid construct, method, and item bias), and a posteriori procedures (including the psychometric analysis of test quality, differential item functioning, and identifying and reducing bias in the data). In addition to describing this set of procedures in theory, we also show how we systematically implemented these procedures. Drawing on data that we collected from over 24 engineering experts and 3,600 engineering students across Russia and China, we provide evidence that it is possible to create tests that are cross-culturally valid, equate-able, and free from bias