Predictive Approaches for Choosing Hyperparameters in Generalised Linear Regression Models

Generalised linear regression models are powerful regression models, often designed using weight decay regularisation to achieve good generalisation. he parameters associated with this method of regularisation have an interpretation as hyperparameters in a Bayesian framework. Standard approaches to estimate the regularisation parameters are ordinary cross validation (OCV) error, generalised cross validation (GCV) error, Mallows/Akaike's Final Prediction Error (EFE) minimisation and Maximum A Posterior (MAP) approaches. In this paper, we propose and investigate predictive approaches, namely, maximisation of Geisser's surrogate Predictive Probability (GPP) and minimisation of mean square error with respect to GPP (referred as Geisser's Predictive mean square Error (GPE)) to estimate these parameters. Within the approximation used to arrive at GCV from OCV, we establish the equivalence relationships that exist among GCV , approximate GPP (AGPP) and approximate GPE (AGPE). While the asymptotic relationship between GCV and EPE is well-known, a non-asymptotic relationship between these two approaches is also established here. These approaches are tested on a number of problems and results show that the proposed approaches are strongly competitive