In least squares, least absolute deviations, and even generalized M-estimation, outlying observations sometimes strongly influence the estimation result, masking an important and interesting relationship existing in the majority of observations. The S-estimators are a class of estimators that overcome this difficulty by smoothly downweighting outliers in fitting regression functions to data. In this paper, we propose a method of model selection suitable in S-estimation. The proposed method chooses a model to minimize a criterion named the penalized S-scale criterion (PSC), which is decreasing in the sample S-scale of fitted residuals and increasing in the number of parameters. We study the large sample behavior of the PSC in nonlinear regression with dependent, heterogeneous data, to establish sets of conditions sufficient for the PSC to consistently select the model with the best fitting performance in terms of the population S-scale, and the one with the minimum number of parameters if there are multiple best performers. Our analysis allows for partial unidentifiability, which is often a practically important possibility when selecting one among nonlinear regression models. We offer two examples to demonstrate how our large sample results could be applied in practice. We also conduct Monte Carlo simulations to verify that the PSC performs as our large sample theory indicates, and assess the reliability of the PSC method in comparison with the familiar Akaike and Schwarz information criteria.
