Observational vs Experimental Data When Making Automated Decisions Using Machine Learning

Decisions supported by machine learning algorithms often have the goal of improving (and thus affecting) an outcome by means of an intervention. Common examples include influencing someone's purchasing behavior with an advertisement or increasing customer retention with a special offer. Unfortunately, if these algorithms use observational data to estimate the effect of the interventions, the resulting estimates will likely suffer from confounding bias. Investing in experimental data offers a way to deal with confounding, but such data are often costly to acquire and may be prone to other sources of bias. This paper addresses the question of whether using readily-available (but confounded) data can lead to decisions that are at least as good as when investing in costly unconfounded data. We present a theoretical comparison between the use of confounded and unconfounded data when the goal is to build models to make automated intervention decisions. The key insight of the work is that optimizing to make the correct decision generally involves understanding whether a causal effect is above or below a given threshold, which is different from optimizing to reduce the magnitude of the bias in a causal-effect estimate. As a result, models trained with confounded data may lead to decisions that are just as good (or better) in certain scenarios, such as when larger causal effects are more likely to be overestimated or when the benefits of larger and cheaper data outweigh the detrimental effects of confounding. The theoretical results are tested by comparing the two approaches using the wide variety of benchmark data sets (77 scenarios and 7,700 data sets in total) from the 2016 ACIC causal modeling competition. Finally, we provide several straightforward tools that may be used in practice to determine whether a confounded model should be deployed to make automatic intervention decisions. It is our hope that a better understanding of this perspective---that confounding does not affect causal effect estimation and causal decision making in the same way---will help improve the chance of success of future methodological developments for automated decision making