Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in Their Judgments About Future Energy Technologies

Expert elicitations are frequently used to characterize future technology outcomes. However their usefulness is limited, in part because: estimates across studies are not easily comparable; choices in survey design and expert selection may bias results; and over-confidence is a persistent problem. We provide quantitative evidence of how these choices affect experts' estimates of the costs of future energy technologies. We harmonize data from 19 elicitations, involving 215 experts, on the 2030 costs of 5 energy technologies: nuclear, biofuels, bioelectricity, solar, and carbon capture. We control for expert characteristics, survey design, and public R&D investment levels on which the elicited values are conditional. We find that, on average, when experts respond to elicitations in person, they ascribe lower confidence (larger uncertainty) to their estimates than when responding via mail or online. In-person interviews also produce more optimistic assessments of best-case (10th percentile) outcomes. The impacts of expert affiliation — government, private sector, or academic — and geography — US or EU — are also significant; academics and US experts have lower confidence than other types of experts. Higher R&D investment levels have no effect on the confidence of experts' judgments. R&D reduces both the median and breakthrough (10th percentile) cost estimates, although the size of the effect varies across technologies. These results indicate the source, direction, and size of bias in energy technology elicitations. They also point to the technology specificity of some of the effects. These biases should be seriously considered, both in interpreting the results of existing elicitations and in designing new ones