Facing the Experts: Survey Mode and Expert Elicitation

In this paper we compare the results of two different expert elicitation methods: in-person interviews and a self-administered web-based survey. Traditional expert elicitation has been done face to face, with an elicitor meeting with an expert for a few hours to several days, depending on the complexity of the analysis. Recently, however, some groups have been using other methods to solicit expert judgments, including self-administered surveys (written, emailed, and web-based), and the use of interactive web tools to facilitate interactions during an elicitation. These elicitations require fewer resources from the assessment team than in-person interviews, and often allow participating experts to provide input on their own schedules, perhaps with additional time to think about their responses. Thus they open up the possibility of using expert elicitation to obtain inputs relevant to a broader set of decisions. To our knowledge, these newer survey-based methods have not been rigorously evaluated for efficacy. We find, much like the results in the literature on different survey modes, different results from two different modes we examined, but no clear indication of which method might be preferred. We suggest future work including some controlled, lab-based experiments and real EEs well designed to avoid sample selection biases and specifically targeted to capture survey mode effects. Such studies would help us determine whether and when the different survey modes are most effective.