Conducting Evaluations Using Multiple Trials

This paper reviews the use of multiple trials in evaluating social programs. We define multiple trials to include both conducting evaluations in multiple sites concurrently and replications, where the intervention is replicated in one or more sites after the initial evaluation. After defining key terms, the paper discusses the rationales for conducting multiple trials, which include increasing sample size to increase statistical power; identifying the most effective program design; increasing external validity; and learning how various factors affect program impact. The paper discusses reasons why program design varies across sites, including adaptations to local environment and participant characteristics and a lack of fidelity to the program design. The issue of program fidelity, or uniformity, is complex, and the paper discusses why programs vary across sites and when it is desirable to maintain consistency and when variation should be permitted. Distinctions are drawn between evaluations of pilots and demonstrations versus ongoing programs, as well as programs where variation is permitted or encouraged versus when a fixed design is desired. The paper then discusses how multiple trials can be used to learn more about the implications of participant and place heterogeneity. The paper includes illustrations drawn from randomized controlled trial evaluations of demonstrations and ongoing programs and explains how decisions about the use of multiple trials should be built into initial evaluation designs and modified as results are obtained