Bandits Atop Reinforcement Learning : Tackling Online Inventory Models With Cyclic Demands

Motivated by a long-standing gap between inventory theory and practice, we study online inventory models with unknown cyclic demand distributions. We design efficient bandits-atop-reinforcement-learning algorithms that cater to the structure of inventory problems. We apply the standard performance measure in online learning literature, regret, defined as the difference between the expected total cost of a policy and that of the optimal policy that has full knowledge of the demand distributions.We first introduce the episodic models where inventory is discarded at the end of every cycle, and then build upon these models to analyze the non-discarding models. For episodic models with cyclic demands, our reinforcement learning (RL) policies HQL and FQL achieve ~O(\sqrt{T}) regret for the lost-sales model with zero lead time, and the multi-product backlogging model with positive lead times, fixed joint-ordering costs and order limits, matching the regret lower bound that we prove in this paper. For the non-discarding models, we construct a bandit learning algorithm on top of the previous RL algorithms, named Meta-HQL. Meta-HQL achieves ~O(\sqrt{T}) regret for the non-discarding lost-sales model with zero lead time, again matching the regret lower bound. For the non-discarding multi-product backlogging model with positive lead times, fixed joint-ordering costs and order limits, our policy Mimic-QL achieves ~O(T^{5/6}) regret bound. Our policies remove the regret dependence on the state-action space size for inventory problems, which is a significant improvement over existing RL algorithms. We experimented with a real supermarket dataset and also with a synthetic dataset. For both sets of experiments, our policy converges to the optimal policy rapidly and dramatically outperforms the best policy that models the demands as i.i.d. instead of cyclic and stochastic