Unified Classical and Robust Optimization for Least Squares

The solutions to robust optimization problems are sometimes too conservative because of the focus on worst-case performance. For the least-squares (LS) problem, we describe a way to overcome this by combining the classical formulation with its robust counterpart. We focus on the issue of overfitting in LS due to limited training data. We construct a sequence of problems with one end being a classical LS and the other end being a robust LS that we create for this purpose. The sequence is parameterized in terms of aspects of the data distribution that can be well-estimated even from limited samples. By choosing the right point in the sequence, we are selectively robust only to the poorly estimated aspects of the data. However, controlling estimation error does not guarantee better prediction. So we transform the problem via a process called {\em objective matching} to align estimation with prediction. Objective matching improves prediction while provably retaining the problem structure. Objective matching helps our method (called Unified Least Squares or ULS) consistently match or outperform other state-of-the-art techniques, including ridge and LASSO regression, on simulations and real-world data sets