Contextual Learning with Online Convex Optimization with Applications to Medical Decision-Making

Optimizing the treatment regimen is a crucial medical decision-making problem. This can be thought of as a two-dimensional decision-making problem with a nested structure, because it involves determining the optimal medication along with its optimal dose. In many cases, it is a challenge to determine the most effective medication for an individual. Even when a suitable medication is identified, dosing it optimally remains a siginificant challenge. Making these two nested decisions involves adaptively learning a personalized disease progression control model. We formulate this as a new contextual multi-armed bandit model under a two-dimensional control with a nested structure. We develop a joint contextual learning and optimization algorithm for this model, which sequentially selects for a patient (i) the best medication based on their contextual information and (ii) the corresponding dose that is optimized over the prior history of all patients who received the same medication. This algorithm synergizes the strength of the contextual bandit with online convex optimization techniques in a seamless fashion. We prove that it admits a sub-linear regret, which is tight up to a logarithmic factor. We demonstrate the practicality of our learning and optimization methodology using clinical trial data on type 2 diabetes patients with high blood pressure and at high risk of cardiovascular disease. Our analysis suggests that our method has the potential to outperform the current practice. We benchmark several policies to show the advantage of our method and provide several insights. Our framework could be widely used in many applications with two-dimensional nested decision-making