Peer Review of Social Science Research in Global Health : A View Through Correspondence Letters to 'The Lancet'

In recent years, the interdisciplinary nature of global health has blurred the lines between medicine and social science. As medical journals publish non-experimental research articles on social policies or macro-level interventions, controversies have arisen when social scientists have criticized the rigor and quality of medical journal articles, raising general questions about the frequency and characteristics of methodological problems and the prevalence and severity of research bias and error. Published correspondence letters can be used to identify common areas of dispute within interdisciplinary global health research and seek strategies to address them. To some extent, these letters can be seen as a "crowd-sourced" (but editor-gated) approach to public peer review of published articles, from which some characteristics of bias and error can be gleaned. In December 2012, we used the online version of The Lancet to systematically identify relevant correspondence in each issue published between 2008 and 2012. We summarize and categorize common areas of dispute raised in these letters. The five concerns most frequently cited in correspondence letters are as follows: measurement error (51% of papers); omitted variables and confounding (45%); implausibility and lack of external validity (43%); missing or low-quality data (32%); and lack of transparency of methods (30%). We recommend better documentation of areas of potential bias with checklists and guidelines to facilitate more rigorous peer review, drawing on experts with econometric expertise as reviewers, and explicitly and thoroughly linking all correspondence letters to the original articles in The Lancet. Most importantly, we recommend The Lancet adopts the replication standard, whereby the data and the coding used to produce the estimates are provided at least to the journal, for reviewers to analyze and replicate the estimates reported by the authors