Automatic evaluation of digital libraries with 5SQual

Digital libraries (DLs) are complex information systems which can present changes in their structure, content, and services. These complexities and dynamics make system maintenance a non-trivial task, since it requires periodical evaluation of the different DL components. Generally, these evaluations are customized per system and are performed only when problems occur and administrator intervention is required. This work aims to change the situation. We present 5SQual, a tool which provides ways to perform automatic and configurable evaluations of some of the most important DL components, among them, digital objects, metadata, and services. The tool implements diverse numeric indicators that are associated with eight quality dimensions described in the 5S quality model. Its generic architecture was developed to be applicable to various DLs and scenarios. In sum, the main contributions of this work include: (i) the design and implementation of 5SQual, a tool that validates a theoretical DL quality model; (ii) the demonstration of the applicability of the tool in several usage scenarios; and (iii) the evaluation (with usability specialists) of its graphical interface specially designed to guide the configuration of 5SQual evaluations. We also present the results of interviews conducted with administrators of real DLs regarding their expectations and opinions about 5SQual.