Systematic reviews: from ‘bare bones’ reviews to policy relevance

Theory-based systematic reviews, which summarise evidence on what works, when and why, strive for greater policy relevance. Reviews that answer these questions adopt a mixed methods approach and draw on a range of study types. Answering the ‘what works’ and ‘what doesn't’ questions means drawing on effectiveness studies, conducted to standards of high-quality impact evaluation. But in formulating answers to the ‘when’ and ‘why’ questions requires a broader range of evidence from both quantitative and qualitative research. Based on a review of the methodological literature in this field and the experience of 3ie's systematic reviews program, this article provides an outline of how a theory-based approach to systematic reviews, including appropriate quantitative and qualitative evidence, can be operationalised. We propose an approach based on three principles: (1) develop a program theory for the intervention; (2) adopt a mixed methods approach and include a broader range of evidence; and (3) maintain the rigour and transparency that characterise systematic reviews. The approach translates into two broad options. Effectiveness plus reviews focus on providing a detailed causal chain analysis by drawing on a program theory and additional data collection on context and intervention implementation. And effectiveness plus with a parallel review component, which is designed to answer specific research questions related to effectiveness, adopts separate inclusion criteria, reflecting the type of studies appropriate for answering those questions.