A Gradient-Based Reinforcement Learning Model of Market Equilibration

This paper formulates a gradient-based reinforcement learning (GRL) model within a game-theoretic machine learning framework where players start from their initial circumstances with dispersed information, using the expected gradient to update choice propensities, and converge to the predicted equilibrium of belief-based models. GRL entails a model-free simulation method to estimate the gradient of expected payoff with respect to choice propensities in repeated games. As the gradient points to the steepest direction towards discovering steady-state equilibrium, GRL provides a theoretical refinement of the updating rule that optimally balances the trade-off between reinforcing the past successful strategies (‘exploitation’) and exploring other strategies (‘exploration’) in choosing actions. A simulated call market demonstrates the effectiveness and stability of GRL in the course of market equilibration, whereas the failure of simple reinforcement rules is attributed to under-exploration