Model Selection Procedures and Their Error-Reduction Targets

This brief note compares model selection procedures in regression. On the one hand there is an observed error reduction ratio that we calculate from the data: h = SSE2/SSE1, where SSE1 and SSE2 are the sums of squared errors in Models 1 and 2, respectively. On the other hand there is a target ratio, which we call H, which is set by the model selection procedure via an expression involving the numbers of variables and observations. If h < H the procedure accepts Model 2 as superior to Model 1. The aim of this note is to derive expressions for each model selection procedure that have the form H = 1/(1x), where x varies from procedure to procedure. These common-form expressions for H allow us to compare model selection procedures transparently. The procedures we examine are: Akaike’s Information Criterion (AIC), Bayesian Information Criterion (BIC), Amemiya’s Prediction Criterion (PC), change in R2 (ΔR2), adjusted R2, and adjusted Rf2 which we have created as a simple generalization of adjusted R2. We show that it is less liberal than adjusted R2 and relates closely to AIC and PC: in so doing it provides fresh insight into their properties