Out-Of-Sample Comparisons of Overfit Models

This paper uses dimension asymptotics to study why overfit linear regression models shouldbe compared out-of-sample; we let the number of predictors used by the larger model increasewith the number of observations so that their ratio remains uniformly positive. Under this limittheory, the naive Diebold-Mariano-West out-of-sample test can test hypotheses about a keyquantity for evaluating forecasting models—a time series analogue to the generalization error—as long as the out-of-sample period is small relative to the total sample size. Moreover, teststhat are designed to reject if the larger model is true, such as the usual in-sample Wald and lmtests and also Clark and McCracken’s (2001, 2005a), McCracken’s (2007) and Clark and West’s(2006, 2007) out-of-sample statistics, will choose the larger model too often when the smallermodel is more accurate.[...]