Lost in the design space? : construct validity in the microfinance literature

While results from individual Randomized Controlled Trials (RCTs) often do not hold beyond their setting, the accumulation of many RCTs can be used to guide policy. But how many studies are required to confidently generalize? Our paper examines construct validity, an often neglected yet important element affecting generalizability. Construct validity deals with how the operationalization of a treatment corresponds to the broader construct it intends to speak to. The universe of potential operationalizations is referred to as the design space. As an empirical example, we review 45 microfinance RCTs to estimate the size of the design space and to underpin that variations in the treatment operationalization matter for the observed effects. We also show that most papers nevertheless generalize from the operationalized treatment to a broad construct, mostly without acknowledging underlying assumptions, and many lack a transparent reporting of construct validity-relevant features.