Effects of Model Selection on the Coverage Probability of Confidence Intervals in Binary-Response Logistic Regression

While model selection is viewed as a fundamental task in data analysis, it imposesconsiderable effects on the subsequent inference. In applied statistics, it is common tocarry out a data-driven approach in model selection and draw inference conditional on theselected model, as if it is given a priori. Parameter estimates following this procedure,however, generally do not reflect uncertainty about the model structure. As far asconfidence intervals are concerned, it is often misleading to report estimates based uponconventional 1−α without considering possible post-model-selection impact. This paperaddresses the coverage probability of confidence intervals of logit coefficients in binary-responselogistic regression. We conduct simulation studies to examine the performanceof automatic model selectors AIC and BIC, and their subsequent effects on actualcoverage probability of interval estimates. Important considerations (e.g. model structure,covariate correlation, etc.) that may have key influence are investigated. This studycontributes in terms of understanding quantitatively how the post-model-selectionconfidence intervals perform in terms of coverage in binary-response logistic regressionmodels.A major conclusion was that while it is usually below the nominal level, there is nosimple predictable pattern with regard to how and how far the actual coverage probabilityof confidence intervals may fall. The coverage probability varies given the effects ofmultiple factors:(1) While the model structure always plays a role of paramount importance, thecovariate correlation significantly affects the interval's coverage, with the tendency that ahigher correlation indicates a lower coverage probability.(2) No evidence shows that AIC inevitably outperforms BIC in terms of achievinghigher coverage probability, or vice versa. The model selector's performance isdependent upon the uncertain model structure and/or the unknown parameter vector θ .(3) While the effect of sample size is intriguing, a larger sample size does notnecessarily achieve asymptotically more accurate inference on interval estimates.(4) Although the binary threshold of the logistic model may affect the coverageprobability, such effect is less important. It is more likely to become substantial with anunrestricted model when extreme values along the dimensions of other factors (e.g. smallsample size, high covariate correlation) are observed.