extended-abstract

Exploring Data Splitting Strategies for the Evaluation of Recommendation Models

Authors:

Richard McCreadie,

Craig Macdonald,

Iadh OunisAuthors Info & Claims

RecSys '20: Proceedings of the 14th ACM Conference on Recommender Systems

Pages 681 - 686

https://doi.org/10.1145/3383313.3418479

Published: 22 September 2020 Publication History

Abstract

Effective methodologies for evaluating recommender systems are critical, so that different systems can be compared in a sound manner. A commonly overlooked aspect of evaluating recommender systems is the selection of the data splitting strategy. In this paper, we both show that there is no standard splitting strategy and that the selection of splitting strategy can have a strong impact on the ranking of recommender systems during evaluation. In particular, we perform experiments comparing three common data splitting strategies, examining their impact over seven state-of-the-art recommendation models on two datasets. Our results demonstrate that the splitting strategy employed is an important confounding variable that can markedly alter the ranking of recommender systems, making much of the currently published literature non-comparable, even when the same datasets and metrics are used.

References

[1]

Ting Bai, Lixin Zou, Wayne Xin Zhao, Pan Du, Weidong Liu, Jian-Yun Nie, and Ji-Rong Wen. 2019. CTRec: A Long-Short Demands Evolution Model for Continuous-Time Recommendation. In SIGIR. 675–684.

[2]

Pedro G Campos, Fernando Díez, and Manuel Sánchez-Montañés. 2011. Towards a more realistic evaluation: testing the ability to predict future tastes of matrix factorization-based recommenders. In RecSys. 309–312.

[3]

Rocío Cañamares, Pablo Castells, and Alistair Moffat. 2020. Offline evaluation options for recommender systems. Information Retrieval Journal(2020), 1–24.

[4]

Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In RecSys. 101–109.

[5]

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In WWW. 173–182.

[6]

Haoji Hu and Xiangnan He. 2019. Sets2Sets: Learning from Sequential Sets with Neural Networks. In SIGKDD. 1491–1499.

[7]

Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In ICDM. 197–206.

[8]

Daniel D Lee and H Sebastian Seung. 2001. Algorithms for non-negative matrix factorization. In NeurIPS. 556–562.

[9]

Jiacheng Li, Yujie Wang, and Julian McAuley. 2020. Time Interval Aware Self-Attention for Sequential Recommendation. In WSDM. 322–330.

[10]

Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018. Variational autoencoders for collaborative filtering. In WWW. 689–698.

[11]

Jarana Manotumruksa, Craig Macdonald, and Iadh Ounis. 2018. A Contextual Attention Recurrent Architecture for Context-aware Venue recommendation. In SIGIR. 555–564.

[12]

Zaiqiao Meng, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2019. Variational Bayesian Context-aware Representation for Grocery Recommendation. In CARS2.0@RecSys.

[13]

Zaiqiao Meng, Richard McCreadie, Craig Macdonald, Iadh Ounis, Shangsong Liang, Siwei Liu, Guangtao Zeng, Liang Junha, Yucheng Liang, Qiang Zhang, Xi Wang, and Wu Yaxiong. 2020. BETA-Rec: Build, Evaluate and Tune Automated Recommender Systems. (2020).

[14]

Massimo Quadrana, Paolo Cremonesi, and Dietmar Jannach. 2018. Sequence-aware recommender systems. Comput. Surveys 51, 4 (2018), 66.

Digital Library

[15]

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In UAI. 452–461.

Digital Library

[16]

Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factorizing personalized markov chains for next-basket recommendation. In WWW. 811–820.

[17]

Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. 2020. Neural Collaborative Filtering vs. Matrix Factorization Revisited. arXiv:2005.09683 (2020).

[18]

Steffen Rendle, Li Zhang, and Yehuda Koren. 2019. On the difficulty of evaluating baselines: A study on recommender systems. arXiv preprint arXiv:1905.01395(2019).

[19]

Noveen Sachdeva, Giuseppe Manco, Ettore Ritacco, and Vikram Pudi. 2019. Sequential Variational Autoencoders for Collaborative Filtering. In WSDM. 600–608.

[20]

Alan Said and Alejandro Bellogín. 2014. Comparative Recommender System Evaluation: Benchmarking Recommendation Frameworks. In RecSys. 129–136.

[21]

Guy Shani and Asela Gunawardana. 2011. Evaluating recommendation systems. In Recommender systems handbook. Springer, 257–297.

[22]

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In CIKM. 1441–1450.

[23]

Qiaoyu Tan, Ninghao Liu, Xing Zhao, Hongxia Yang, Jingren Zhou, and Xia Hu. 2020. Learning to Hash with Graph Neural Networks for Recommender Systems. In WWW. 1988–1998.

[24]

Mengting Wan, Di Wang, Jie Liu, Paul Bennett, and Julian McAuley. 2018. Representing and Recommending Shopping Baskets with Complementarity, Compatibility and Loyalty. In CIKM. 1133–1142.

[25]

Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019. Kgat: Knowledge graph attention network for recommendation. In SIGKDD. 950–958.

Digital Library

[26]

Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural graph collaborative filtering. In SIGIR. 165–174.

[27]

Teng Xiao, Shangsong Liang, and Zaiqiao Meng. 2019. Dynamic Collaborative Recurrent Learning. In CIKM. 1151–1160.

[28]

Hamed Zamani and W Bruce Croft. 2020. Learning a Joint Search and Recommendation Model from User-Item Interactions. In WSDM. 717–725.

[29]

Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. Comput. Surveys 52, 1 (2019), 1–38.

Digital Library

Cited By

Kim YChang T(2024)Deep Learning-Based Freight Recommendation System for Freight Brokerage PlatformSystems10.3390/systems1211047712:11(477)Online publication date: 7-Nov-2024
https://doi.org/10.3390/systems12110477
Slade ERennick-Egglestone SNg FKotera YLlewellyn-Beardsley JNewby CGlover TKeppens JSlade M(2024)The Implementation of Recommender Systems for Mental Health Recovery Narratives: Evaluation of Use and PerformanceJMIR Mental Health10.2196/4575411(e45754)Online publication date: 29-Mar-2024
https://doi.org/10.2196/45754
Rahman AMollah M(2024)Reverse engineering and analysis of microstructure polymer fiber via artificial neural networks: simplifying the design approachJournal of Optical Communications10.1515/joc-2023-0361Online publication date: 6-Mar-2024
https://doi.org/10.1515/joc-2023-0361
Show More Cited By

Index Terms

Exploring Data Splitting Strategies for the Evaluation of Recommendation Models
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
  2. Information systems applications
    1. Data mining

Index terms have been assigned to the content through auto-classification.

Recommendations

Experimental evaluation of context-dependent collaborative filtering using item splitting

Collaborative Filtering (CF) computes recommendations by leveraging a historical data set of users' ratings for items. CF assumes that the users' recorded ratings can help in predicting their future ratings. This has been validated extensively, but in ...
User-Specific Feature-Based Similarity Models for Top-n Recommendation of New Items
Survey Paper, Regular Papers and Special Section on Participatory Sensing and Crowd Intelligence

Recommending new items for suitable users is an important yet challenging problem due to the lack of preference history for the new items. Noncollaborative user modeling techniques that rely on the item features can be used to recommend new items. ...
Context-based splitting of item ratings in collaborative filtering
RecSys '09: Proceedings of the third ACM conference on Recommender systems

Collaborative Filtering (CF) recommendations are computed by leveraging a historical data set of users' ratings for items. It assumes that the users' previously recorded ratings can help in predicting future ratings. This has been validated extensively, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

RecSys '20: Proceedings of the 14th ACM Conference on Recommender Systems

September 2020

796 pages

ISBN:9781450375832

DOI:10.1145/3383313

Copyright © 2020 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 September 2020

Check for updates

Author Tags

Qualifiers

Extended-abstract
Research
Refereed limited

Funding Sources

European Community's Horizon 2020 research and innovation programme

Conference

RecSys '20

Sponsor:

RecSys '20: Fourteenth ACM Conference on Recommender Systems

September 22 - 26, 2020

Virtual Event, Brazil

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

46
Total Citations
View Citations
946
Total Downloads

Downloads (Last 12 months)218
Downloads (Last 6 weeks)26

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kim YChang T(2024)Deep Learning-Based Freight Recommendation System for Freight Brokerage PlatformSystems10.3390/systems1211047712:11(477)Online publication date: 7-Nov-2024
https://doi.org/10.3390/systems12110477
Slade ERennick-Egglestone SNg FKotera YLlewellyn-Beardsley JNewby CGlover TKeppens JSlade M(2024)The Implementation of Recommender Systems for Mental Health Recovery Narratives: Evaluation of Use and PerformanceJMIR Mental Health10.2196/4575411(e45754)Online publication date: 29-Mar-2024
https://doi.org/10.2196/45754
Rahman AMollah M(2024)Reverse engineering and analysis of microstructure polymer fiber via artificial neural networks: simplifying the design approachJournal of Optical Communications10.1515/joc-2023-0361Online publication date: 6-Mar-2024
https://doi.org/10.1515/joc-2023-0361
Fan YJi YZhang JSun A(2024)Our Model Achieves Excellent Performance on MovieLens: What Does It Mean?ACM Transactions on Information Systems10.1145/367516342:6(1-25)Online publication date: 18-Oct-2024
https://dl.acm.org/doi/10.1145/3675163
Belal YBen Mokhtar SHaddadi HWang JMashhadi A(2024)Survey of Federated Learning Models for Spatial-Temporal Mobility ApplicationsACM Transactions on Spatial Algorithms and Systems10.1145/366608910:3(1-39)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1145/3666089
Klenitskiy AVolodkevich APembek AVasilev A(2024)Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential RecommendationsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688195(1067-1072)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688195
Frolov EMatveeva TMirvakhabova LOseledets I(2024)Self-Attentive Sequential Recommendations with Hyperbolic RepresentationsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688180(981-986)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688180
Milogradskii ALashinin OP AAnanyeva MKolesnikov S(2024)Revisiting BPR: A Replicability Study of a Common Recommender System BaselineProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688073(267-277)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688073
Boratto LFabbri FFenu GMarras MMedda G(2024)Fair Augmentation for Graph Collaborative FilteringProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688064(158-168)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688064
Rampisela TRuotsalo TMaistro MLioma CHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and RelevanceProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657832(271-281)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657832
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents