Decision-centric Active Learning of Binary-Outcome Models

It can be expensive to acquire the data required for businesses toemploy data-driven predictive modeling, for example to model consumerpreferences to optimize targeting. Prior research has introduced“active learning” policies for identifying data that areparticularly useful for model induction, with the goal of decreasing thestatistical error for a given acquisition cost (error-centricapproaches). However, predictive models are used as part of adecision-making process, and costly improvements in model accuracy donot always result in better decisions. This paper introduces a newapproach for active data acquisition that targets decision-makingspecifically. The new decision-centric approach departs from traditionalactive learning by placing emphasis on acquisitions that are more likelyto affect decision-making. We describe two different types ofdecision-centric techniques. Next, using direct-marketing data, wecompare various data-acquisition techniques. We demonstrate thatstrategies for reducing statistical error can be wasteful in adecision-making context, and show that one decision-centric technique inparticular can improve targeting decisions significantly. We also showthat this method is robust in the face of decreasing quality of utilityestimations, eventually converging to uniform random sampling, and thatit can be extended to situations where different data acquisitions havedifferent costs. The results suggest that businesses should considermodifying their strategies for acquiring information through normalbusiness transactions. For example, a firm such as Amazon.com thatmodels consumer preferences for customized marketing may acceleratelearning by proactively offering recommendations—not merely toinduce immediate sales, but for improving recommendations in the future.