This paper considers tests of the effectiveness of a policy intervention, defined as a change in the parameters of a policy rule, in the context of a macroeconometric dynamic stochastic general equilibrium (DSGE) model. We consider two types of intervention, First the standard case of a parameter change that does not alter the steady state, and second one that does alter the steady state, e.g. the target rate of inflation. We consider two types of test, one a multi-horizon test, where the post-intervention policy horizon, H, is small and fixed, and a mean policy effect test where H is allowed to increase without bounds. The multi-horizon test requires Gaussian errors, but the mean policy effect test does not. It is shown that neither of these two tests are consistent, in the sense that the the power of the tests does not tend to unity as H tends to infinity, unless the intervention alters the steady state. This follows directly from the fact that DSGE variables are measured as deviations from the steady state, and the effects of policy change on target variables decay exponentially fast. We investigate the size and power of the proposed mean effect test by simulating a standard three equation New Keynesian DSGE model. The simulation results are in line with our theoretical findings and show that in all applications the tests have the correct size; but unless the intervention alters the steady state, their power does not go to unity with H.
