Calibrate to Operate : Aligning Patient-Level Predictions with System-Wide Forecasts

We explore the challenges of re-using machine learning models originally developed for patient-level predictions to forecast system-wide quantities in hospitals. As a running example, we examine the use of patient-level predictions of length of stay to predict short-term occupancy in an emergency department. We first provide analytical evidence for the potential consequences of the misalignment between the patient-level and system-level prediction tasks: the emergence of bias and the failure to detect non-stationarity. Using patient-level data from an emergency department, we demonstrate the existence of such challenges in our data. We show that the simple calibration procedures we propose produce unbiased and more accurate system-wide predictions. We also show that dynamic re-training of the calibration over time further improves the accuracy in our non-stationary setting. Overall, we provide a practical roadmap for hospitals considering making sophisticated patient-level predictions to improve system operations