The effects of choice of database and data retrieval methods on research performance evaluations of Asian universities

Purpose – The purpose of this paper is to study the effects of the choice of database and data retrieval methods on the research performance of a number of selected Asian universities from 33 countries using two different indicators (publication volume and citation count) and three subject fields (energy, environment and materials) during the period 2005‐2009. Design/methodology/approach – To determine the effect of the choice of database, Scopus and Web of Science databases were queried to retrieve the publications and citations of the top ten Asian universities in three subject fields. In ascertaining the effect of data retrieval methods, the authors proposed a new data retrieval method called Keyword‐based Data Retrieval (KDR), which uses relevant keywords identified by independent experts to retrieve publications and their citations of the top 30 Asian universities in the Environment field from the entire Scopus database. The results were then compared with those retrieved using the Conventional Data Retrieval (CDR) method. Findings – The Asian university ranking order is strongly affected by the choice of database, indicator, and the data retrieval method used. The KDR method yields many more publications and citation counts than the CDR method, shows better understanding of the university ranking results, and retrieves publications and citations in source titles outside those classified by the database. Moreover the publications found by the KDR method have a multidisciplinary research focus. Originality/value – The paper concludes that KDR is a more suitable methodology to retrieve data for measuring university research performance, particularly in an environment where universities are increasingly engaging in multidisciplinary research.