Outlier Aversion in Evaluating Performance : Evidence from Figure Skating

The quality of subjective performance evaluation is dependent on the incentive structures faced by evaluators, in particular on how they are monitored and themselves evaluated. Figure skating competitions provide a unique opportunity to study subjective evaluation. This paper develops and tests a simple model of what I call "outlier aversion bias" in which subjective evaluators avoid submitting outlying judgments. We find significant evidence for the existence of outlier aversion. Individual judges within a game manipulate scores to achieve a targeted level of agreement with the other judges. Furthermore, a natural experiment shows that the dispersion of scores across judges depends upon the type of judge-assessment system and its implication for outlier aversion. Agreement may not be a good criterion for the validity of an evaluation system, contradicting the industrial psychology and personnel management literature.