An Empirical Method for Harmless Error

Trials are often imperfect. When inadmissible evidence is introduced or the jury is incorrectly instructed, judges must determine whether the error was prejudicial or merely harmless. In making that assessment, judges resort to speculation about the counterfactual question of whether the error changed the outcome, compared to the decision of a properly informed and instructed jury. These decisions are likely colored by confirmation and status quo biases, along with “mental contamination” of the error itself. Even when appellate judges perform these analyses accurately, their decisions appear conclusory. Scholars and judges have roundly criticized this doctrine, but no solution has emerged. We developed and piloted an unbiased and transparent method for making harmless error determinations, using randomized experiments with simulated jurors. To pilot this method on three real cases, we recruited 489 human subjects to participate as mock jurors reviewing trial vignettes that we manipulated into conditions with and without the errors. Subjects were blinded to the purpose of the study and to the first trial's outcome. By comparing verdict rates in the error and no-error conditions, we estimated whether the error was harmful.We found a high degree of correspondence between the assessments of real judges and our experimental method, which could be taken as a validation of the method and reassurance that it would not cause a radical change in the rates at which new trials are granted. Still, across the thousands of cases in which harmless error determinations are made each year, the empirical method may be more reliable since it avoids known biases. The transparency of our method may also lend greater legitimacy to harmlessness determinations. If such a method is used as a tool for litigants in real cases, courts will be called upon to establish procedures for taking such evidence and then draw lines to specify how much prejudice is too much, while also being sensitive to the limitations of statistical power. Our study is most useful as proof of concept for a new method to improve harmless error analyses