Analysis of science and technology oversight systems is often conducted from a single disciplinary perspective and by using a limited set of criteria for evaluation. In this article, we develop an approach that blends risk analysis, social science, public administration, legal, public policy, and ethical perspectives to develop a broad set of criteria for assessing oversight systems. Multiple methods, including historical analysis, expert elicitation, and behavioral consensus, were employed to develop multidisciplinary criteria for evaluating oversight of emerging technologies. Sixty-six initial criteria were identified from extensive literature reviews and input from our NSF-funded project Working Group. Criteria were placed in four categories reflecting the development, attributes, evolution, and outcomes of oversight systems. Expert elicitation, consensus methods, and multidisciplinary review of the literature were used to refine a condensed, operative set of criteria. Twenty-eight criteria resulted spanning four categories: seven development criteria, fifteen attribute criteria, five outcome criteria, and one evolution criterion. These criteria illuminate how oversight systems develop, operate, change, and affect society. We term our approach “integrated oversight assessment” and propose its use as a tool for analyzing relationships among features, outcomes, and trade-offs of oversight systems. Comparisons among historical case studies of oversight using a consistent set of criteria should result in defensible and evidence-supported lessons to guide the development of oversight systems for emerging technologies, such as nanotechnology
