To Be or Not to Be in the Sample? On the Consequences of Using Manipulation Checks in Experimental Accounting Research

Manipulation checks are used to improve the quality of experimental evidence by making sure that only those subjects who are tested as having successfully received the experimental treatment are included in the analysis. While standard experimental designs do not call for such a check, they are quasi-obligatory in many disciplines and also in accounting research. In our paper, we discuss the consequences of using manipulation checks resulting in the exclusion of subjects. Using an analytical model, we show that using manipulation checks to exclude cases may translate into a bias towards more significant differences between the experimental and control groups, thereby jeopardizing results to be mere artifacts (alpha error). We also provide a simulation indicating that such biases can occur even when only very few subjects have been excluded. We contribute to the literature by providing an in-depth discussion and analysis of the consequences of using manipulation checks for experimental accounting research. Based on our discussion, we develop recommendations as to how manipulation checks can be used to gain further insights into mechanisms that drove the observed results of an experimental study. In particular, we recommend that data obtained from manipulation checks should be used as an explanatory variable, not as a justification to exclude subjects from the final sample