Important facts on comparative backtesting of Value at Risk

VaR remains important in market risk management as Basel keeps most of the backtesting based on 1% VaR. Comparative backtesting as practiced in the current literature suffers from a major double problem. On the one hand, the score functions, although strictly consistent, may assign very good or even the best scores to clearly failing models. On the other hand, the DM test (Diebold and Mariano, 1995), as widely used to validate the score functions, fails to detect these "false best" models. We document these facts with concrete cases. To correct this issue, we propose to build the DM test using the identification functions associated with the score functions rather than on the score functions themselves. In addition, we provide a key improvement to the conditional calibration test of Nolde and Ziegel (2017). This allows for additional model validation via score functions, while providing a novel way to test the conditional coverage assumption on the standard backtesting side. The parallel and collaborative approach of the two VaR backtesting components, the standard and the comparative one, allows for increased conceptual richness and robustness of the overall backtesting. Finally, given the conceptual similarities, the points addressed regarding VaR should also concern the comparative backtesting of CVaR