Mirror, mirror on the wall : machine predictions and self-fulfilling prophecies

We show that disclosing machine predictions to affected parties can trigger self-fulfilling prophecies. In an investment game, we experimentally vary investors' and recipients' access to a machine prediction about recipients' likelihood to pay back an investment. Recipients who privately learn about an incorrect machine prediction alter their behavior in the direction of the prediction. Furthermore, when recipients learn that an investor has disregarded a machine prediction of no-repayment, this further lowers the repayment amount. We interpret these findings as evidence that transparency regarding machine predictions can alter recipients' beliefs about what kind of person they are and what investors expect of them. Our results indicate that providing increased access to machine predictions as an isolated measure to alleviate accountability concerns may have unintended negative consequences for organizations by possibly changing customer behavior.