Use of Area Under the Curve (AUC) from Propensity Model to Estimate Accuracy of the Estimated Effect of Exposure

Objective: To investigate the relationship between the area under the Receiver Operating Characteristic curve (AUC) of the propensity model for exposure and the accuracy of the estimated effect of the exposure on the outcome of interest.Methods: A Monte Carlo simulation study was performed where multiple realizations of three binary variables: outcome, exposure of interest and a covariate were repeatedly generated from the distribution determined by the parameters of the propensity and main models and the prevalence of the exposure. Propensity model was a logistic regression with the exposure of interest as a dependent variable and a single covariate as an independent variable. Main model was a logistic regression with outcome as a dependent variable, exposure of interest and covariate as independent variables. A total of 500 simulations were performed for each considered combination of the model parameters and the prevalence of the exposure. AUC was estimated from the probabilities predicted by the propensity score model. The accuracy of the estimated effect of exposure was primarily assessed with the square root of Mean Square Error (RMSE); the fifth and ninety-fifth percentile of the empirical distribution of the estimator were used to illustrate a range of not unlikely deviations from the true value.Results: The square root of Mean Square Error of the estimated effect of exposure increases as AUC increases from 0.6 to 0.9. Varying values for parameters of the propensity score model or the main effect model does not change the direction of this trend. As the proportion of exposed subjects changes away from 0.5 the RMSE increases, but the effect of AUC on RMSE remains approximately the same. Similarly, as sample size changes from 50 to 100 or 200, the RMSE of effect estimate decreases on average, but the effect of AUC on RMSE remains approximately the same. Also, the rate of change in RMSE increases with increasing AUC; the rate is the lowest when AUC changes from 0.6 to 0.7 and is highest when AUC changes from 0.8 to 0.9.Conclusions: The AUC of the propensity score model for exposure provides a single, relatively easy to compute, and suitable for various kind of data statistic, which can be used as an important indicator of the accuracy of the estimated effect of exposure on the outcome of interest. The public health importance is that it can be considered as an alternative to the previously suggested (Rubin, 2001) simultaneous consideration of the conditions of closeness of means and variances of the propensity scores in the different exposure groups. Our simulations indicate that the estimated effect of exposure is highly unreliable if AUC of the propensity model is larger than 0.8; at the same time AUCs of less than 0.7 are not associated with any substantial increase of inaccuracy of the estimated effect of exposure.