Relevance, Redundancy, and Regularization : Penalized Regression and the Quest for the ℓ₀ Quasi-Norm

The vector of a linear model’s coefficients represents the clearest guide to causal inference. Collinearity among variables, however, undermines the interpretation of that model. A wildly large positive coefficient on one variable may be offset by a comparably large negative coefficient on a collinear variable. Neither the size nor the sign of those coefficients can be trusted.Instability arising from collinearity and high variance can be remedied through regularized or penalized regression. These methods can select model features in graduated or categorical fashion. Enforcing ℓ2 (Ridge) and ℓ1 (Lasso) regularization can blunt or even eliminate irrelevant or redundant variables. Methods incorporating the ℓ1 norm can induce sparsity. The resulting vector of nonzero standardized coefficients delivers the ℓ0 quasi-norm as the best mathematical representation of the model’s causal inferences.In addition to Lasso, Ridge, and ElasticNet (a hybrid of Lasso and Ridge), this article describes Bayesian Ridge and automatic relevance determination. It compares the quest for sparsity through ℓ1-penalized regression with orthogonal matching pursuit, a form of backward stepwise regression. Two robust regression methods, Huber and Theil-Sen, complete the toolkit.The application of these methods to data on death rates from cancer in 3,047 counties in the United States strengthens inferences regarding cancer incidence, college education, and dependency on public health care coverage. This study also uncovers intriguing evidence on racial disparities in death by cancer. Sounder causal inference from regularization and the 95% confidence interval offset potentially misleading guidance based on conventional thresholds of statistical significance.On balance, regularized and robust regression methods achieve test prediction accuracy in the neighborhood of r2 ≈ 0.48 and RMSE ≈ 0.75, comparable to results from OLS. Affirmative improvements in prediction accuracy are a bonus: It suffices that conscious departures from the least-squares estimate remain stable and generalizable when these regression methods are extended to previously unseen data