Automatic performance evaluation of web search engines using judgments of metasearch engines

Purpose – The purpose of this paper is to introduce two new automatic methods for evaluating the performance of search engines. The reported study uses the methods to experimentally investigate which search engine among three popular search engines (Ask.com, Bing and Google) gives the best performance. Design/methodology/approach – The study assesses the performance of three search engines. For each one the weighted average of similarity degrees between its ranked result list and those of its metasearch engines is measured. Next these measures are compared to establish which search engine gives the best performance. To compute the similarity degree between the lists two measures called the “tendency degree” and “coverage degree” are introduced; the former assesses a search engine in terms of results presentation and the latter evaluates it in terms of retrieval effectiveness. The performance of the search engines is experimentally assessed based on the 50 topics of the 2002 TREC web track. The effectiveness of the methods is also compared with human‐based ones. Findings – Google outperformed the others, followed by Bing and Ask.com. Moreover significant degrees of consistency – 92.87 percent and 91.93 percent – were found between automatic and human‐based approaches. Practical implications – The findings of this work could help users to select a truly effective search engine. The results also provide motivation for the vendors of web search engines to improve their technology. Originality/value – The paper focuses on two novel automatic methods to evaluate the performance of search engines and provides valuable experimental results on three popular ones.