On the need for open-source ground truths for medical information retrieval systems

Smart information retrieval systems are becoming increasingly prevalent due to the rate at which the amount of digitized raw data has increased, and continues to increase. This is especially true in the medical domain, as there is much data stored in unstructured formats which contain "hidden" information within them. By hidden, this means information that cannot ordinarily be found by performing a simple text search. To test the information retrieval systems that handle such data, a ground truth, or gold standard, is normally required in order to gain performance values according to an information need. In this paper we emphasize the lack of freely available, annotated medical data and wish to encourage the community of developers working in this area to make available whatever data they can. Also, the importance of such annotated medical data is raised, especially its importance and potential impact on teaching and training in medicine. As well as this, this paper will point out some of the advantages that access to a freely available pool of annotated medical objects would provide to several areas of medicine and informatics. The paper then discusses some of the considerations that would have to be made for any future systems developed that would provide a service to make the creating, sharing, and annotating of such data easy to perform (by using an online, web-based interface, for example). Finally, the paper discusses in detail the benefits of such a system to teaching and examining medical students. -- Evaluation ; text mining ; information retrieval ; medicine ; health care