On the Measurement of the Predictive Success of Learning Theories in Repeated Games

The growing literature on learning in games has produced various results on the predictive success of learning theories. These results, however, were based on various methods of comparison. The present paper uses experimental data on a set of four games in order to check on the robustness of rankings among learning rules across measures. We characterise measures along three dimensions: (i) the scoring rule, (ii) the method of comparison, and (iii) the definition of observations and apply all thus defined measures to 12 learning rules. The results show that rankings are indeed sensitive to the measure used. Furthermore, we point at deficiencies of certain measures that have been applied in the past and suggest the use of simulated data when learning rules are supposed to predict realisations of random variables.