Understanding Recommendation Quality Using Embeddings

Current methods of evaluating the quality of recommender systems are based on averages of metrics such as the average normalized discounted cumulative gain, average diversity and average reciprocity. Averages of metrics give a good sense of the overall quality of the recommendations, but not of how their quality is distributed with respect to the recommendation system's users or items. This paper presents a visual method, based on embedding a high dimensional content feature-space into a 2D image, that is capable of providing insights in which users are receiving high quality recommendations and how biased recommendation quality is with respect to different types of users. Through a proof of concept in the domain of job recommendation we show that our method allows business people to come to relevant answers to the question “For which of my users does my recommender system work well/poorly?”