Documenting differences between humans and AI in high-stakes decisions : a labor market turing test

We developed a Labor Market Turing Test (LMTT) to measure human-AI decision alignment using data from 277 human recruiters engaged in a field experiment set in Quito, Ecuador. We augmented the pool of recruiters by creating AI teams, each of which with differing impersonation of human-like traits, and compared their choices to humans and a benchmark AI model. While AI teams were more consistent, they selected candidates with a pattern that markedly different from human choices. In fact, random decisions mir- rored human choices more closely than our most human-like AI agents. These findings reveal a fundamental tension between algorithmic consistency and human judgment. That humans were closer to a random process when com- paring candidates with equal productivity might be seen as a fairer outcome. Our LMTT framework, which involves isolating and estimating a machina la- tent trait, provides a quantitative tool for assessing human-AI alignment which can be employed across critical domains, such as healthcare, justice, and edu- cation, thereby informing the design and AI governance.