Evaluating Group Fairness in News Recommendations: A Comparative Study of Algorithms and Metrics

Beyond accuracy metrics, such as fairness and diversity, have become widely studied topics in recommender systems. Improving these metrics is important not only from an ethical and legal perspective, but can also improve overall user satisfaction. Although these metrics are widely discussed, very little empirical research has been done, especially comparing multiple algorithms across different metrics. This work explores the role of fairness and diversity in news recommender systems, specifically in the context of the Austrian media landscape. This study aims to identify the most effective approaches for generating fair and diverse news recommendations, while addressing the potential negative consequences of biased recommendations and filter bubbles, such as societal polarization and the suppression of information. This includes an extensive literature review of relevant group unfairness metrics and state-of-the-art fairness-aware algorithms. A dataset of articles from an Austrian newspaper was used for empirical research, with analysis performed on fairness, and diversity of recommendations. The key message of the study is that accuracy and fairness can be achieved simultaneously with the right modeling approach, while diversity can be held constant using these modeling techniques. The study recommends the use of Personalized Fairness based on Causal Notion models for accuracy and reducing certain unfairness metrics, and finds Fairness Objectives for Collaborative Filtering models more effective at reducing other types of unfairness. The findings contribute to the field by demonstrating the importance of incorporating these metrics into the design and evaluation of recommender systems.