Reinforcement Learning with Foregone Payoff Information in Normal Form Games

This paper studies the reinforcement learning of Erev and Roth with foregone payoff information in normal form games: players observe not only the realised payoffs but also foregone payoffs, the ones which they could have obtained if they had chosen the other actions. We provide conditions under which the reinforcement learning process almost surely converges to a mixed action profile at which each action is chosen with a probability proportional to its expected payoff. In particular, in symmetric 2×2 partnership games without a Pareto-dominant Nash equilibrium and matching-pennies games, the action profile corresponds to the mixed Nash equilibrium. However, in general, the action profile does not correspond to a Nash equilibrium: it corresponds to a Nash equilibrium if and only if for each player, all the actions are chosen with equal probability at the equilibrium. Instead, we show that the action profile corresponds to a perturbed equilibrium, regular quantal response equilibrium (Goeree et al., 2005), when there is no distortion on the foregone payoffs. Therefore, for any normal form games, if the conditions hold, the process almost surely converges to the equilibrium