Forecast Accuracy Measures for Exception Reporting Using Receiver Operating Characteristic Curves

This paper identifies forecasts of exceptions in product or service demand (i.e., large changes or extreme values) as a special need in forecasting, requiring new forecast accuracy measures based on the tails of sampled forecast error distributions. For this purpose, the paper introduces application of the receiver operating characteristic (ROC) framework, which has been used to assess exceptional behavior or cases in many fields. The “exception principle” of management reporting provides the corresponding forecast requirements. Seasonality estimates in univariate forecast models and leading independent variables in multivariate forecast models are among the approaches to forecasting exceptions. In a case study on serious violent crime in Pittsburgh, Pennsylvania across small sub-areas of the city, the simplest, nonnaïve univariate forecast method is best for forecasting ordinary conditions, as found in previous research using conventional forecast accuracy measures, but the most complex multivariate model is best for forecasting exceptional conditions, using ROC forecast accuracy measures.