Survey of Model Selection and Model Combination

A general statistical modeling problem is that given a class of competing models and new data, how one can improve the overall model performance. In general, there exist two solutions for this problem, namely model selection and model combination. Model selection is to select a single best model while model combination builds a composite model by aggregating all available information. However, except for this difference model selection and model combination share the most important key elements such as model performance evaluation and are closely related to each other. A generalized flexible framework for designing predictive model performance evaluation method is put forward and possible choices for its two components, that is, model distance measure and generability estimation, are categorized and reviewed. After than, a unified framework is proposed to accommodate both model selection and model combination, in which model selection works as an extreme case of model combination. Finally, many model selection and combination methods in the literature, all of which can be fit into the unified framework, are reviewed