Are policy counterfactuals based on structural VAR's reliable?

Based on standard New Keynesian models I show that policy counterfactuals based on the theoretical structural VAR representations of the models fail to reliably capture the impact of changes in the parameters of the Taylor rule on the (reduced-form) properties of the economy. Based on estimated models for the Great Inflation and the most recent period, I show that, as a practical matter, the problem appears to be non-negligible. These results imply that the outcomes of SVAR-based policy counterfactuals should be regarded with caution, as their informativeness for the specific issue at hand–e.g., understanding the role played by monetary policy in exacerbating the Great Depression, causing the Great Inflation, or fostering the Great Moderation–is, in principle, open to question. Finally, I argue that SVAR-based policy counterfactuals suffer from a crucial logical shortcoming: given that their reliability crucially depends on unknown structural characteristics of the underlying data generation process, such reliability cannot simply be assumed, and can instead only be ascertained with a reasonable degree of confidence by estimating structural (DSGE) models.