On Surprise-Hacking

We argue that perception, priming, and cognition research utilize the tools of stage magic in problematic ways, engaging in what we call "surprise-hacking." Surprise-hacking consists of the pre-experimental planning and staging of counterintuitive and negative results (such as blindness, bias, and irrationality). We highlight examples of surprise-hacking in the cognitive sciences, with implications for behavioral economics. For example, while Kahneman argues that priming results "are not made up, nor are they statistical flukes"—we argue that they are, like magic, experimentally staged and conjured to produce surprising outcomes. However, unlike p-hacking, surprise-hacking will not be resolved with further replications, more data, or pre-registration. Rather, we argue that better theories are needed to understand perception, rationality, and the human mind.