Validating the backtests of risk measures

Financial risk model evaluation or backtesting is a key part of the internal model’sapproach to market risk management as laid out by the Basle Committee on BankingSupervision (2004). However there are a number of backtests that may be applied andthere is little guidance as to the most appropriate method. The goal of this paper is toanalyze the ability of various evaluation methodologies to gauge the accuracy of riskmodels. We compare evaluation effectiveness using the standard binomial approach,together with the interval forecast backtesting, the density forecast backtesting and theprobability forecast backtesting. Our comparison is completed for three risk measures:Value-at-Risk (VaR), Expected Shortfall (ES) and Spectral Risk measure (SRM). We payspecial attention to applications related to ES and SRM as backtesting of these modelshave not been explored in any detail thus far. Based on the Monte Carlo simulations andthe empirical study, a number of interesting results emerge. Firstly within hypothesisbasedtests, including the binomial backtesting, the interval forecast backtesting and thedensity forecasts backtesting, the overall dominance of density forecast backtesting isconfirmed. In particular, the backtesting for SRM and ES is more effective than for VaRin identifying an incorrect model from alternative models in a small sample setting.Secondly, we propose a loss function for SRM where the probability forecast backtestingis capable of identifying accurate models from alternative models. Thirdly, in all of thebacktesting methods examined, the choice of the distribution specification is a moreimportant factor in determining the evaluation performance than the choice of thevolatility specification.