Collusion by Mistake : Does Algorithmic Sophistication Drive Supra-Competitive Profits?

A burgeoning literature shows that self-learning algorithms may, under some conditions, reach seemingly-collusive outcomes: after repeated interaction, competing algorithms earn supra-competitive profits, at the expense of efficiency and consumer welfare. However, these simulations results, generally based on Q-learning, are in stark contrast with several foundational theoretical works that show that ideally-designed reinforcement learning agents necessarily converge to the competitive outcome. In this paper, we show theoretically and empirically that algorithmic seeming collusion can be caused by imperfect exploration: allowing for more robust exploration does lead otherwise seemingly-collusive Q-learning algorithms to play competitively. Further, more sophisticated algorithms are shown to consistently play their best response and exploit the seemingly-collusive algorithms. In contrast to the existing literature, our research suggests that seemingly-collusive outcomes may arise not because algorithms are able to learn sophisticated collusive strategies, but because they fail to learn the best-response strategy. In light of these findings, we recommend that algorithmic collusion should not be analyzed solely via Q-learning as the main results may not necessarily convey to other commonly used and more sophisticated algorithms