"Anything you can do, I can do" : examining the use of ChatGPT in situational judgement tests for professional program admission

We explored the transformative impact of ChatGPT on applicants' responses and performance in situational judgement tests (SJTs), as well as the role played by faking-prevention mechanisms, in two complementary studies. Study 1 examined how the availability of ChatGPT influenced response content and performance of real applicants (N = 107,805), who completed an SJT for admission before vs. after the release of the technology. We found only small differences in content (e.g., slightly less “authentic” words used) and performance (slight score improvements when controlling for response length, no differences otherwise). In Study 2, we used an experimental approach with (N = 138) Prolific participants completing a mock SJT, while being instructed to use ChatGPT when responding (vs. use online resources or no resources). We found only slightly higher SJT scores for the ChatGPT users, but no difference in response content. Additionally, GPTZero (i.e., a popular AI detection tool) struggled to detect ChatGPT content, and generated many false positives, in both studies. This research advances our understanding of how the release and popularization of ChatGPT can influence applicant behaviors. Given the “arms race” nature of applicant selection, they also highlight the importance of designing assessments to prevent or limit faking. Yet, the ever-evolving nature of AI calls for continuous research on the topic.