Robustness of personal rankings: The Handelsblatt example

In the last years, Handelsblatt has published several rankings of business economists from German, Swiss and Austrian research institutions based on their journal publication output. These rankings have a strong influence on the academic profession. We scrutinize the Handelsblatt methodology by examining the effect the rankings' underlying algorithms and assumptions have on the scores and ranks of individual researchers. In doing so, we clarify how robust the result is with respect to these internal parameters. Since the parameters used by Handelsblatt are not scientifically substantiated but defined ad hoc, this question is of great importance. For each parameter variation, we provide several robustness measures for both the Handelsblatt life's work ranking and the Handelsblatt recent research performance ranking. E.g., if one applies a weighting scheme that lays more emphasis on first tier journal publications such that the weight of a particular category is always double of the weight of the next lower category, rank correlations based on all researchers in both personal rankings exceed 80 %. However, if one solely considers the top 25 performing researchers rank correlations fall below 50 and 20 % of researchers even drop out of this top group. Further research as well as the discussion in the academic community should clarify whether these correlations verify the robustness of the ranking or manifest the opposite.