Comparing Performance of Methods to Deal with Differential Attrition in Lottery Based Evaluations

In randomized controlled trials, it is common for attrition rates to differ by lottery status, jeopardizing the identification of causal effects. Inverse probability weighting methods (Hirano et al, 2003; Busso et al., 2014) and estimation of informative bounds for the treatment effects (e.g. Lee, 2009; Angrist et al., 2006) have been used frequently to deal with differential attrition bias. This paper studies the performance of various methods by comparing the results using two datasets: a district-sourced dataset subject to considerable differential attrition, and an expanded state-sourced dataset with much less attrition, differential and overall. We compared the performance of different methods to correct for differential attrition in the district dataset, as well as we conducted simulation analyses to assess the sensitivity of bounding methods to their underlying assumptions. In our application, methods to correct differential attrition induced bias, whereas the unadjusted district level results were closer and more substantively similar to the estimated effects in the benchmark state dataset. Our simulation exercises showed that even small deviations from the underlying assumptions in bounding methods proposed by Angrist et al. (2006) increased bias in the estimates. In practice, researchers often do not have enough information to verify the extent to which these underlying assumptions are met, so we recommend using these methods with caution