Constructing Quantiles via Forecast Errors : Theory and Empirical Evidence

Probabilistic forecasts such as quantiles are essential inputs to decision-making under uncertainty. The most common type of forecasts often comes in the form of point forecasts, and it is therefore necessary for the decision maker to construct uncertainty measures around the obtained point forecasts. One simple approach suggests leveraging historical forecast errors to create quantiles around the given point forecast (the E2Q method). While these E2Q estimators may take different forms, we develop a theoretical framework to understand how the bias and variance of an estimator affect its overall performance. We find that unbiasedness is not optimal, and the bias towards the center of the distribution is more costly than towards the tails. Besides, higher estimator variance leads to worse performance. These bias-variance insights allow for comparisons among a wide range of estimators. For example, with limited samples, we find that sample quantile (SQ) is inferior to normal approximation (N) for tail quantiles in both bias (in the more costly direction) and variance. In the same setting, an estimator based on a more general parametric fit, such as the generalized normal (GN), could perform worse than N. This tends to happen when the bias of N is in the less costly direction. We empirically validate our theoretical findings using the M5 accuracy competition submissions