Incorporating Editorial Feedback in the Evaluation of News Recommender Systems

Research in the recommender systems field typically applies a rather traditional evaluation methodology when assessing the quality of recommendations. This methodology heavily relies on incorporating different forms of user feedback (e.g., clicks) representing the specific needs and interests of the users. While this methodology may offer various benefits, it may fail to comprehensively project the complexities of certain application domains, such as the news domain. This domain is distinct from other domains primarily due to the strong influence of editorial control in the news delivery process. Incorporation of this role can profoundly impact how the relevance of news articles is measured when recommended to the users. Despite its critical importance, there appears to be a research gap in investigating the dynamics between the roles of editorial control and personalization in the community of recommender systems. In this paper, we address this gap by conducting experiments where the relevance of recommendations is assessed from an editorial perspective. We received a real-world dataset from TV 2, one of the largest editor-managed commercial media houses in Norway, which includes editors’ feedback on how news articles are being related. In our experiment, we considered a scenario where algorithm-generated recommendations, using the K-Nearest Neighbor (KNN) model, employing various text embedding models to encode different sections of the news articles (e.g., title, lead title, body text, and full text), are compared against the editorial feedback. The results are promising, demonstrating the effectiveness of the recommendation in fulfilling the editorial prospects.