On Surprise-Hacking

We argue that perception, priming, and cognition research utilize the tools of stage magic in problematic ways, engaging in what we call "surprise-hacking." Surprise-hacking consists of the pre-experimental planning and staging of counterintuitive and negative results (such as blindness, bias, and irrationality). We highlight examples of surprise-hacking in the cognitive sciences, with implications for behavioral economics. For example, while Kahneman argues that priming results "are not made up, nor are they statistical flukes" — we argue that they are, like magic, experimentally staged and conjured to produce surprising outcomes. However, unlike p-hacking, surprise-hacking will not be resolved with further replications, more data, or pre-registration. Rather, we argue that better theories are needed to understand perception, rationality, and the human mind