Full Accuracy Scoring Accelerates the Discovery of Skilled Forecasters

Reliable detection of forecasting skill is slow and resource-intensive. Forecasters need to answer dozens of questions, which may take months or years to resolve. To accelerate this process, we propose the Full Accuracy Score (FAS), a measure that combines the strengths of ground-truth-based proper scores with the early availability of proxy scores that measure the distance between individual and consensus estimates. FAS treats the two scores as complements, using ground-truth scores on resolved questions and proxy scores on open ones. Across two forecasting tournaments, we show that a baseline FAS variant outperforms classic Brier scores in predicting final relative scores of individual forecasters. Even in cold-start settings, where the same crowd provides consensus estimates and evaluation scores, FAS outperforms Brier scores across all forecasting competitions. Access to a separate crowd of pre-qualified forecasters further boosts FAS performance. We show that FAS performs best when proxy scores are weighted approximately half as much as Brier scores. FAS accelerates skill assessment, and is especially useful in the context of long-term forecasting competitions