A study of discontinuity effects in regression inference based on web-augmented mixed mode surveys

We consider a situation where the sample design of a survey is modified over time in order to save resources. The former design is a classical large-scale survey. The new design is a mixed mode survey where a smaller classical sample is augmented by records of an online survey. For the online survey no inclusion probabilities are available. We study how this change of data collection affects regression coefficient estimation when the model remains constant in the population over time. Special emphasis is placed on situations where the online records are selective with respect to the model. We develop a statistical framework to quantify so-called survey discontinuities in regression analysis. The term refers to differences between coefficient estimates that solely stem from the survey redesign. For this purpose, we apply hypothesis tests to identify whether observed differences in estimates are significant. Further, we discuss propensity estimation and calibration as potential methods to reduce selection biases stemming from the web survey. A Monte Carlo simulation study is conducted to test the methods under different degrees of selectivity. We find that even mild informativeness significantly impairs regression inference relative to the former survey despite bias correction.