short-paper

Measuring anti-relevance: a study on when recommendation algorithms produce bad suggestions

Authors:

Pablo Sánchez,

Alejandro BellogínAuthors Info & Claims

RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems

Pages 367 - 371

https://doi.org/10.1145/3240323.3240382

Published: 27 September 2018 Publication History

Abstract

Typically, performance of recommender systems has been measured focusing on the amount of relevant items recommended to the users. However, this perspective provides an incomplete view of an algorithm's quality, since it neglects the amount of negative recommendations by equating the unknown and negatively interacted items when computing ranking-based evaluation metrics. In this paper, we propose an evaluation framework where anti-relevance is seamlessly introduced in several ranking-based metrics; in this way, we obtain a different perspective on how recommenders behave and the type of suggestions they make. Based on our results, we observe that non-personalized approaches tend to return less bad recommendations than personalized ones, however the amount of unknown recommendations is also larger, which explains why the latter tend to suggest more relevant items. Our metrics based on anti-relevance also show the potential to discriminate between algorithms whose performance is very similar in terms of relevance.

References

[1]

Ricardo A. Baeza-Yates and Berthier A. Ribeiro-Neto. 2011. Modern Information Retrieval - the concepts and technology behind search, Second edition. Pearson Education Ltd., Harlow, England. http://www.mir2ed.org/

Digital Library

[2]

Nicholas J Belkin, Michael Cole, and Ralf Bierig. 2008. Is relevance the right criterion for evaluating interactive information retrieval. In In Proceedings of the SIGIR 2008 Workshop on Beyond Binary Relevance: Preferences, Diversity, and SetLevel judgments.

[3]

Rocío Cañamares and Pablo Castells. 2018. From the PRP to the Low Prior Discovery Recall Principle for Recommender Systems. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08--12, 2018, Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (Eds.). ACM, 1081--1084.

Digital Library

[4]

Iván Cantador, Peter Brusilovsky, and Tsvi Kuflik. 2011. Second workshop on information heterogeneity and fusion in recommender systems (HetRec2011). In Proceedings of the 2011 ACM Conference on Recommender Systems, RecSys 2011, Chicago, IL, USA, October 23--27, 2011, Bamshad Mobasher, Robin D. Burke, Dietmar Jannach, and Gediminas Adomavicius (Eds.). ACM, 387--388.

Digital Library

[5]

Pablo Castells, Neil J. Hurley, and Saul Vargas. 2015. Novelty and Diversity in Recommender Systems. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 881--918.

[6]

Charles L. A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan Büttcher, and Ian MacKinnon. 2008. Novelty and diversity in information retrieval evaluation. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2008, Singapore, July 20--24, 2008, Sung-Hyon Myaeng, Douglas W. Oard, Fabrizio Sebastiani, Tat-Seng Chua, and Mun-Kew Leong (Eds.). ACM, 659--666.

Digital Library

[7]

Michael D. Ekstrand and John Riedl. 2012. When recommenders fail: predicting recommender failure for algorithm selection and combination. In Sixth ACM Conference on Recommender Systems, RecSys '12, Dublin, Ireland, September 9--13, 2012, Padraig Cunningham, Neil J. Hurley, Ido Guy, and Sarabjot Singh Anand (Eds.). ACM, 233--236.

Digital Library

[8]

Zeno Gantner, Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2011. MyMediaLite: a free recommender system library. In Proceedings of the 2011 ACM Conference on Recommender Systems, RecSys 2011, Chicago, IL, USA, October 23--27, 2011, Bamshad Mobasher, Robin D. Burke, Dietmar Jannach, and Gediminas Adomavicius (Eds.). ACM, 305--308.

Digital Library

[9]

Huan Gui, Haishan Liu, Xiangrui Meng, Anmol Bhasin, and Jiawei Han. 2016. Downside management in recommender systems. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2016, San Francisco, CA, USA, August 18--21, 2016, Ravi Kumar, James Caverlee, and Hanghang Tong (Eds.). IEEE Computer Society, 394--401.

Digital Library

[10]

Asela Gunawardana and Guy Shani. 2015. Evaluating Recommender Systems. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 265--308.

[11]

F. Maxwell Harper and Joseph A. Konstan. 2016. The MovieLens Datasets: History and Context. TiiS 5, 4 (2016), 19:1--19:19.

Digital Library

[12]

Jonathan L. Herlocker, Joseph A. Konstan, Loren G. Terveen, and John Riedl. 2004. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22, 1 (2004), 5--53.

Digital Library

[13]

Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative Filtering for Implicit Feedback Datasets. In Proceedings of the 8th IEEE International Conference on Data Mining (ICDM 2008), December 15--19, 2008, Pisa, Italy. IEEE Computer Society, 263--272.

Digital Library

[14]

Joseph A. Konstan and John Riedl. 2012. Recommender systems: from algorithms to user experience. User Model. User-Adapt. Interact. 22, 1--2 (2012), 101--123.

Digital Library

[15]

Benjamin M. Marlin, Richard S. Zemel, Sam T. Roweis, and Malcolm Slaney. 2007. Collaborative Filtering and the Missing at Random Assumption. In UAI 2007, Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence, Vancouver, BC, Canada, July 19--22, 2007, Ronald Parr and Linda C. van der Gaag (Eds.). AUAI Press, 267--275. https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2&article_id=1699&proceeding_id=23

Digital Library

[16]

Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being accurate is not enough: how accuracy metrics have hurt recommender systems. In Extended Abstracts Proceedings of the 2006 Conference on Human Factors in Computing Systems, CHI 2006, Montréal, Québec, Canada, April 22--27, 2006, Gary M. Olson and Robin Jeffries (Eds.). ACM, 1097--1101.

Digital Library

[17]

Sung-Hyon Myaeng and Robert R. Korfhage. 1990. Integration of user profiles: models and experiments in information retrieval. Inf. Process. Manage. 26, 6 (1990), 719--738.

Digital Library

[18]

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18--21, 2009, Jeff A. Bilmes and Andrew Y. Ng (Eds.). AUAI Press, 452--461. https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2&article_id=1630&proceeding_id=25

Digital Library

[19]

S. E. Robertson. 1997. Readings in Information Retrieval. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, Chapter The Probability Ranking Principle in IR, 281--286. http://dl.acm.org/citation.cfm?id=275537.275701

Digital Library

[20]

Alan Said and Alejandro Bellogín. 2014. Comparative recommender system evaluation: benchmarking recommendation frameworks. In Eighth ACM Conference on Recommender Systems, RecSys '14, Foster City, Silicon Valley, CA, USA - October 06 - 10, 2014, Alfred Kobsa, Michelle X. Zhou, Martin Ester, and Yehuda Koren (Eds.). ACM, 129--136.

Digital Library

[21]

Pablo Sánchez and Alejandro Bellogín. 2018. Time-Aware Novelty Metrics for Recommender Systems. In Advances in Information Retrieval - 40th European Conference on IR Research, ECIR 2018, Grenoble, France, March 26--29, 2018, Proceedings (Lecture Notes in Computer Science), Gabriella Pasi, Benjamin Piwowarski, Leif Azzopardi, and Allan Hanbury (Eds.), Vol. 10772. Springer, 357--370.

[22]

Harald Steck. 2013. Evaluation of recommendations: rating-prediction and ranking. In Seventh ACM Conference on Recommender Systems, RecSys '13, Hong Kong, China, October 12--16, 2013, Qiang Yang, Irwin King, Qing Li, Pearl Pu, and George Karypis (Eds.). ACM, 213--220.

Digital Library

[23]

Saul Vargas and Pablo Castells. 2011. Rank and relevance in novelty and diversity metrics for recommender systems. In Proceedings of the 2011 ACM Conference on Recommender Systems, RecSys 2011, Chicago, IL, USA, October 23--27, 2011, Bamshad Mobasher, Robin D. Burke, Dietmar Jannach, and Gediminas Adomavicius (Eds.). ACM, 109--116.

Digital Library

[24]

Ellen M. Voorhees. 2004. Measuring ineffectiveness. In SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, July 25--29, 2004, Mark Sanderson, Kalervo Järvelin, James Allan, and Peter Bruza (Eds.). ACM, 562--563.

Digital Library

Cited By

Vadhil FNanne MSalihi M(2024)A Novel Artificial Intelligence-Based Intrusion Detection System—NAI2DSArtificial Intelligence and Its Practical Applications in the Digital Economy10.1007/978-3-031-71426-9_14(168-181)Online publication date: 25-Nov-2024
https://doi.org/10.1007/978-3-031-71426-9_14
Park MLee K(2022)Exploiting Negative Preference in Content-based Music Recommendation with Contrastive LearningProceedings of the 16th ACM Conference on Recommender Systems10.1145/3523227.3546768(229-236)Online publication date: 12-Sep-2022
https://dl.acm.org/doi/10.1145/3523227.3546768
Mena-Maldonado ECañamares RCastells PRen YSanderson MHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Agreement and Disagreement between True and False-Positive Metrics in Recommender Systems EvaluationProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401096(841-850)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401096
Show More Cited By

Measuring anti-relevance: a study on when recommendation algorithms produce bad suggestions
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals

Recommendations

Relevance Meets Coverage: A Unified Framework to Generate Diversified Recommendations
Regular Papers, Survey Papers and Special Issue on Recommender System Benchmarks

Collaborative filtering (CF) models offer users personalized recommendations by measuring the relevance between the active user and each individual candidate item. Following this idea, user-based collaborative filtering (UCF) usually selects the local ...
Deriving Item Features Relevance from Past User Interactions
UMAP '17: Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization

Item-based recommender systems suggest products based on the similarities between items computed either from past user preferences (collaborative filtering) or from item content features (content-based filtering). Collaborative filtering has been proven ...
Unified relevance models for rating prediction in collaborative filtering

Collaborative filtering aims at predicting a user's interest for a given item based on a collection of user profiles. This article views collaborative filtering as a problem highly related to information retrieval, drawing an analogy between the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems

September 2018

600 pages

ISBN:9781450359016

DOI:10.1145/3240323

General Chairs:
Sole Pera
Boise State University
,
Michael Ekstrand
Boise State University
,
Program Chairs:
Xavier Amatriain
Curai, USA
,
John O'Donovan
University of California, Santa Barbara

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper

Funding Sources

MINECO

Conference

RecSys '18

Sponsor:

SIGCHI

RecSys '18: Twelfth ACM Conference on Recommender Systems

October 2, 2018

British Columbia, Vancouver, Canada

Acceptance Rates

RecSys '18 Paper Acceptance Rate 32 of 181 submissions, 18%;

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
285
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)2

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Vadhil FNanne MSalihi M(2024)A Novel Artificial Intelligence-Based Intrusion Detection System—NAI2DSArtificial Intelligence and Its Practical Applications in the Digital Economy10.1007/978-3-031-71426-9_14(168-181)Online publication date: 25-Nov-2024
https://doi.org/10.1007/978-3-031-71426-9_14
Park MLee K(2022)Exploiting Negative Preference in Content-based Music Recommendation with Contrastive LearningProceedings of the 16th ACM Conference on Recommender Systems10.1145/3523227.3546768(229-236)Online publication date: 12-Sep-2022
https://dl.acm.org/doi/10.1145/3523227.3546768
Mena-Maldonado ECañamares RCastells PRen YSanderson MHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Agreement and Disagreement between True and False-Positive Metrics in Recommender Systems EvaluationProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401096(841-850)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401096
Sánchez PBogers TSaid ABrusilovsky PTikk D(2019)Exploiting contextual information for recommender systems oriented to tourismProceedings of the 13th ACM Conference on Recommender Systems10.1145/3298689.3347062(601-605)Online publication date: 10-Sep-2019
https://dl.acm.org/doi/10.1145/3298689.3347062

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents