Targeted Automation of Order Decisions Using Machine Learning

In many practical settings, human decision-makers can review the recommendations that a decision support model generates and either approve or override the recommendation. Overrides can be rare when the decision support model is well-tuned to its task. However, the human decision-maker is still burdened with reviewing a potentially large number of recommendations. In this paper, we examine whether a machine learning (ML) prediction model can address this inefficiency.We develop a set of ML models in conjunction with our research partner, a large materials handling equipment manufacturer, and employ the models using our partner's procurement ordering process. This process covers 535,168 orders at suppliers over four years for 36 decision-makers (planners). Using only features that are available at the time a system makes a recommendation, our proposed set of ML models predict (1) whether or not the decision-maker will modify the recommendation and (2) whether such a modification will improve or impair the performance of the system. Our analysis is complicated by the fact that some of the planners have few observations in our training data, leading to poor predictions for those planner-level models. We address this by augmenting our planner-level models with cluster-level models. The latter draws inferences from the predicted behavior of clusters of planners with similar behavior.Using our ML models, we identify a material portion of the order recommendations that can be automated with little, or even a positive, impact on performance.Using our results to automate orders, the firm can (1) free the decision-makers' time for other value-added activities and (2) improve the system's performance