Effectiveness of collaboration in VET: Measuring skills for solving complex vocational problems with a multidimensional authentic technology-based assessment

Context: Collaboration and digitization in work and production process have been of scientific interest in the research field of vocational education and training for decades. So far, there is a lack of empirical evidence on the extent to which the performance of collaborative problem-solving can outperform individual problem-solving. An appropriate instrument for measuring collaborative performance appropriately has been lacking to this point: It is challenging to (1) collect data on interaction in collaboration in a standardized way and (2) to assign the individual performances to the group performance. In a web-based office simulation, this study examined the two-dimensional construct of collaborative problem-solving ability, which consists of cognitive and social components, and explored the effectiveness of collaboration. Methods: The test settings are systematically varied between individual and collaborative problem-solving in a domain-typical problem scenario in the economic domain. A theory-driven two-dimensional authentic instrument with a complex commercial problem in a web-based office simulation was applied to measure the performance in these two test settings. Multi-Group Confirmatory Factor Analysis was conducted to test for measurement invariance across test settings, the two-dimensional RASCH-Model incorporating between-item multidimensionality with correlated dimensions was used for ability estimation. ANOVA test was conducted to determine if there is a statistically significant difference with respect to the ability between the individual and collaborative test settings. Findings: The two-dimensional construct of collaborative problem-solving ability with correlated dimensions showed better fit than the unidimensional construct. The scalar measurement invariance was established after excluding three items. ANOVA tests confirmed that the collaborative setting enhances the problem-solving ability of learners significantly in both the cognitive component and social component, and the ability in cognitive component was more fostered. Conclusion: The findings suggested the validation of this multidimensional authentic instrument to measure the performance in a collaborative work environment in economic domains. The results showed the effectiveness of collaboration with a computer-based agent. In the practice of vocational training and education, teachers can use this instrument for the evaluation of the effectiveness of collaboration.