The Promise of Prediction Contests

This paper examines the prediction contest as a vehicle for aggregating the opinions of a crowd of experts. After proposing a general definition distinguishing prediction contests from other mechanisms for harnessing the wisdom of crowds, we examine two common types of prediction contests. For both point forecasting contests and pick-the-winners contests, we illustrate the incentive for forecasters to submit reports that exaggerate in the direction of their private information. In contrast to conventional wisdom, this non-truthful reporting usually improves the accuracy of the resulting crowd forecast. In the context of a simple theoretical model of overlapping information and forecaster behaviors, we present closed-form expressions for the mean squared error of the crowd forecasts which help identify the situations in which prediction contests will be most useful. The paper also reports on an empirical investigation into the accuracy of a pick-the-winners contest involving hundreds of experts. In that example, the resulting crowd forecasts were found to be more accurate than forecasts from a sophisticated statistical model