Comparative evaluation of web search engines in health information retrieval

Purpose – The intent of this work is to evaluate several generalist and health‐specific search engines for retrieval of health information by consumers: to compare the retrieval effectiveness of these engines for different types of clinical queries, medical specialties and condition severity; and to compare the use of evaluation metrics for binary relevance scales and for graded ones. Design/methodology/approach – The authors conducted a study in which users evaluated the relevance of documents retrieved by four search engines for two different health information needs. Users could choose between generalist (Bing, Google, Sapo and Yahoo!) and health‐specific (MedlinePlus, SapoSaúde and WebMD) search engines. The authors then analysed the differences between search engines and groups of information needs with six different measures: graded average precision (gap), average precision (ap), gap@5, gap@10, ap@5 and ap@10. Findings – The results show that generalist web search engines surpass the precision of health‐specific engines. Google has the best performance, mainly in the top ten results. It was found that information needs associated with severe conditions are associated with higher precision, as are overview and psychiatry questions. Originality/value – The study is one of the first to use a recently proposed measure to evaluate the effectiveness of retrieval systems with graded relevance scales. It includes tasks from several medical specialties, types of clinical questions and different levels of severity which, to the best of the authors' knowledge, has not been done before. Moreover, users have considerable involvement in the experiment. The results help in understanding how search engines differ in their responses to health information needs, what types of online health information are more common on the web and how to improve this type of search.