survey

A Systematic Study on Reproducibility of Reinforcement Learning in Recommendation Systems

Authors:

Emanuele Cavenaghi,

Gabriele Sottocornola,

Markus ZankerAuthors Info & Claims

ACM Transactions on Recommender Systems, Volume 1, Issue 3

Article No.: 11, Pages 1 - 23

https://doi.org/10.1145/3596519

Published: 14 July 2023 Publication History

Abstract

Reproducibility is a main principle in science and fundamental to ensure scientific progress. However, many recent works point out that there are widespread deficiencies for this aspect in the AI field, making the reproducibility of results impractical or even impossible. We therefore studied the state of reproducibility support on the topic of Reinforcement Learning & Recommender Systems to analyse the situation in this context. We collected a total of 60 papers and analysed them by defining a set of variables to inspect the most important aspects that enable reproducibility, such as dataset, pre-processing code, hardware specifications, software dependencies, algorithm implementation, algorithm hyperparameters, and experiment code. Furthermore, we used the ACM Badges definitions assigning them to the selected papers. We discovered that, like in many other AI domains, the Reinforcement Learning & Recommender Systems field is grappling with a reproducibility crisis, as none of the selected papers were reproducible when strictly applying the ACM Badges definitions according to our analysis.

References

[1]

Xiangyu Zhao, Liang Zhang, Long Xia, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2019. Deep Reinforcement Learning for List-wise Recommendations. arXiv:1801.00209 [cs.LG].

[2]

M. Mehdi Afsar, Trafford Crump, and Behrouz Far. 2021. Reinforcement learning based recommender systems: A survey. arXiv preprint arXiv:2101.06286 1 (2021).

[3]

M. Mehdi Afsar, Trafford Crump, and Behrouz Far. 2022. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys 55, 7, Article 145 (Dec.2022), 38 pages.

[4]

Xueying Bai, Jian Guan, and Hongning Wang. 2019. A model-based reinforcement learning with adversarial training for online recommendation. In Advances in Neural Information Processing Systems. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, Curran Associates, Inc., Vancouver.

[5]

Richard Bellman. 1957. A Markovian decision process. Journal of Mathematics and Mechanics 6, 5 (1957), 679–684.

[6]

Robin Burke. 1999. Integrating knowledge-based and collaborative-filtering recommender systems. In Proceedings of the Workshop on AI and Electronic Commerce. AAAI, 69–72.

[7]

Walter Carrer-Neto, María Luisa Hernández-Alcaraz, Rafael Valencia-García, and Francisco García-Sánchez. 2012. Social knowledge-based recommender system. Application to the movies domain. Expert Systems with Applications 39, 12 (2012), 10990–11000.

Digital Library

[8]

Jia-Wei Chang, Ching-Yi Chiou, Jia-Yi Liao, Ying-Kai Hung, Chien-Che Huang, Kuan-Cheng Lin, and Ying-Hung Pu. 2019. Music recommender using deep embedding-based features and behavior-based reinforcement learning. Multimedia Tools and Applications 80, 26 (2019), 1–28.

[9]

Haokun Chen, Xinyi Dai, Han Cai, Weinan Zhang, Xuejian Wang, Ruiming Tang, Yuzhou Zhang, and Yong Yu. 2019. Large-scale interactive recommendation with tree-structured policy gradient. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (2019), 3312–3320.

Digital Library

[10]

Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H. Chi. 2019. Top-k off-policy correction for a REINFORCE recommender system. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining. ACM, New York, NY, 456–464.

Digital Library

[11]

Minmin Chen, Bo Chang, Can Xu, and Ed H. Chi. 2021. User response models to improve a REINFORCE recommender system. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. ACM, New York, NY, 121–129.

Digital Library

[12]

Shi-Yong Chen, Yang Yu, Qing Da, Jun Tan, Hai-Kuan Huang, and Hai-Hong Tang. 2018. Stabilizing reinforcement learning in dynamic environment with application to online recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, 1187–1196.

Digital Library

[13]

Xiaocong Chen, Chaoran Huang, Lina Yao, Xianzhi Wang, Wei Liu, and Wenjie Zhang. 2020. Knowledge-guided deep reinforcement learning for interactive recommendation. In Proceedings of the 2020 International Joint Conference on Neural Networks. Vol. 1, IEEE, 1–8.

[14]

Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song. 2019. Generative adversarial user model for reinforcement learning based recommendation system. In Proceedings of the 36th International Conference on Machine Learning. Vol. 97, PMLR, 1052–1061.

[15]

Xiaocong Chen, Lina Yao, Julian McAuley, Guanglin Zhou, and Xianzhi Wang. 2021. A survey of deep reinforcement learning in recommender systems: A systematic review and future directions. arXiv preprint arXiv:2109.03540 1 (2021).

[16]

Xiaocong Chen, Lina Yao, Julian McAuley, Guanglin Zhou, and Xianzhi Wang. 2023. Deep reinforcement learning in recommender systems: A survey and new perspectives. Knowledge-Based Systems 264, 1 (2023), 110335.

[17]

Xiaocong Chen, Lina Yao, Aixin Sun, Xianzhi Wang, Xiwei Xu, and Liming Zhu. 2021. Generative inverse deep reinforcement learning for online recommendation. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management. ACM, New York, NY, 201–210.

Digital Library

[18]

Sungwoon Choi, Heonseok Ha, Uiwon Hwang, Chanju Kim, Jung-Woo Ha, and Sungroh Yoon. 2018. Reinforcement learning based recommender system using biclustering technique. (2018). arXiv:1801.05532 [cs.IR].

[19]

Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. 2016. Deep reinforcement learning in large discrete action spaces. (2016). arXiv:1512.07679 [cs.AI].

[20]

Jun Feng, Heng Li, Minlie Huang, Shichen Liu, Wenwu Ou, Zhirong Wang, and Xiaoyan Zhu. 2018. Learning to collaborate: Multi-scenario ranking via multi-agent reinforcement learning. In Proceedings of the 2018 World Wide Web Conference. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1939–1948.

Digital Library

[21]

Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys’19 Copenhagen, Denmark), Association for Computing Machinery, New York, NY, 101–109.

[22]

Rong Gao, Haifeng Xia, Jing Li, Donghua Liu, Shuai Chen, and Gang Chun. 2019. DRCGR: Deep reinforcement learning framework incorporating CNN and GAN-based for interactive recommendation. In Proceedings of the 2019 IEEE International Conference on Data Mining (ICDM’19), Vol. 1. IEEE, 1048–1053.

[23]

David Goldberg. 1991. What every computer scientist should know about floating-point arithmetic. ACM Computing Surveys 23, 1 (1991), 5–48.

Digital Library

[24]

Yu Gong, Yu Zhu, Lu Duan, Qingwen Liu, Ziyu Guan, Fei Sun, Wenwu Ou, and Kenny Q. Zhu. 2019. Exact-k recommendation via maximal clique optimization. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’19 Anchorage, AK), ACM, New York, NY, 617–626.

Digital Library

[25]

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press.

Digital Library

[26]

Claudio Greco, Alessandro Suglia, Pierpaolo Basile, and Giovanni Semeraro. 2017. Converse-et-impera: Exploiting deep learning and hierarchical reinforcement learning for conversational recommender systems. In AI*IA 2017 Advances in Artificial Intelligence. Springer International Publishing, Cham, 372–386.

[27]

Odd Erik Gundersen and Sigbjørn Kjensmo. 2018. State of the art: Reproducibility in artificial intelligence. Proceedings of the AAAI Conference on Artificial Intelligence 32, 1 (2018), 1644–1651.

[28]

Jianhua Han, Yong Yu, Feng Liu, Ruiming Tang, and Yuzhou Zhang. 2019. Optimizing ranking algorithm in recommender system via deep reinforcement learning. In Proceedings of the 2019 International Conference on Artificial Intelligence and Advanced Manufacturing (AIAM’19), Vol. 1, IEEE, 22–26.

[29]

Naieme Hazrati and Francesco Ricci. 2022. Recommender systems effect on the evolution of users’ choices distribution. Information Processing & Management 59, 1 (2022), 102766.

Digital Library

[30]

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web (WWW’17 Perth, Australia). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 173–182.

Digital Library

[31]

Benjamin J. Heil, Michael M. Hoffman, Florian Markowetz, Su-In Lee, Casey S. Greene, and Stephanie C. Hicks. 2021. Reproducibility standards for machine learning in the life sciences. Nature Methods 18, 10 (2021), 1132–1135.

[32]

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016. Session-based recommendations with recurrent neural networks. (2016). arXiv:1511.06939 [cs.LG].

[33]

Daocheng Hong, Yang Li, and Qiwen Dong. 2020. Nonintrusive-sensing and reinforcement-learning based adaptive personalized music recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’20 Virtual Event, China). ACM, New York, NY, 1721–1724.

Digital Library

[34]

Binbin Hu, Chuan Shi, and Jian Liu. 2017. Playlist recommendation based on reinforcement learning. In Intelligence Science I. Springer International Publishing, Cham, 172–182.

[35]

Matthew Hutsonm. 2018. Artificial intelligence faces reproducibility crisis. Science 359, 6377 (2018), 725–726.

[36]

Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Morgane Lustman, Vince Gatto, Paul Covington, Jim McFadden, Tushar Chandra, and Craig Boutilier. 2019. Reinforcement learning for slate-based recommender systems: A tractable decomposition and practical methodology. (2019). arXiv:1905.12767 [cs.LG].

[37]

Wacharawan Intayoad, Chayapol Kamyod, and Punnarumol Temdee. 2018. Reinforcement learning for online learning recommendation system. In Proceedings of the 2018 Global Wireless Summit (GWS), Vol. 1, IEEE, 167–170.

[38]

Sarika Jain, Anjali Grover, Praveen Singh Thakur, and Sourabh Kumar Choudhary. 2015. Trends, problems and solutions of recommender system. In Proceedings of the International Conference on Computing, Communication & Automation, Vol. 1, IEEE, 955–958.

[39]

Dietmar Jannach, Markus Zanker, Alexander Felfernig, and Gerhard Friedrich. 2010. Recommender Systems: An Introduction. Cambridge University Press.

[40]

Yu Lei and Wenjie Li. 2019. Interactive recommendation with user-specific deep reinforcement learning. ACM Transactions on Knowledge Discovery from Data (TKDD) 13, 6 (2019), 1–15.

Digital Library

[41]

Yu Lei, Hongbin Pei, Hanqi Yan, and Wenjie Li. 2020. Reinforcement learning based recommendation with graph convolutional q-network. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’20 Virtual Event, China). ACM, New York, NY, 1757–1760.

Digital Library

[42]

Huizhi Liang. 2020. DRprofiling: Deep reinforcement user profiling for recommendations in heterogenous information networks. IEEE Transactions on Knowledge and Data Engineering 34, 4 (2020), 1723–1734.

[43]

Elad Liebman, Maytal Saar-Tsechansky, and Peter Stone. 2015. DJ-MC: A reinforcement-learning agent for music playlist recommendation. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15 Istanbul, Turkey). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 591–599.

[44]

Feng Liu, Huifeng Guo, Xutao Li, Ruiming Tang, Yunming Ye, and Xiuqiang He. 2020. End-to-end deep reinforcement learning based recommendation with supervised embedding. In Proceedings of the 13th International Conference on Web Search and Data Mining (WSDM’20 Houston, TX). ACM, New York, NY, 384–392.

Digital Library

[45]

Feng Liu, Ruiming Tang, Huifeng Guo, Xutao Li, Yunming Ye, and Xiuqiang He. 2020. Top-aware reinforcement learning based recommendation. Neurocomputing 417, 1 (2020), 255–269.

[46]

Feng Liu, Ruiming Tang, Xutao Li, Weinan Zhang, Yunming Ye, Haokun Chen, Huifeng Guo, and Yuzhou Zhang. 2018. Deep reinforcement learning based recommendation with explicit user-item interactions modeling. arXiv preprint arXiv:1810.12027 (2018).

[47]

Feng Liu, Ruiming Tang, Xutao Li, Weinan Zhang, Yunming Ye, Haokun Chen, Huifeng Guo, Yuzhou Zhang, and Xiuqiang He. 2020. State representation modeling for deep reinforcement learning based recommendation. Knowledge-Based Systems 205, 1 (2020), 106170.

[48]

Su Liu, Ye Chen, Hui Huang, Liang Xiao, and Xiaojun Hei. 2018. Towards smart educational recommendations with reinforcement learning in classroom. In Proceedings of the 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), Vol. 1, IEEE, 1079–1084.

[49]

Zhongqi Lu and Qiang Yang. 2016. Partially observable Markov decision process for recommender systems. arXiv preprint arXiv:1608.07793 (2016).

[50]

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Ji Yang, Minmin Chen, Jiaxi Tang, Lichan Hong, and Ed H. Chi. 2020. Off-policy learning in two-stage recommender systems. In Proceedings of the Web Conference 2020 (WWW’20 Taipei, Taiwan). ACM, New York, NY, 463–473.

Digital Library

[51]

Isshu Munemasa, Yuta Tomomatsu, Kunioki Hayashi, and Tomohiro Takagi. 2018. Deep reinforcement learning for recommender systems. In Proceedings of the 2018 International Conference on Information and Communications Technology (ICOIACT), Vol. 1, IEEE, 226–233.

[52]

Michael J. Pazzani and Daniel Billsus. 2007. Content-based recommendation systems. In The Adaptive Web. Springer, 325–341.

[53]

Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. 2021. Problems and opportunities in training deep learning software systems: An analysis of variance. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE’20 Virtual Event, Australia). ACM, New York, NY, 771–783.

[54]

Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research 22, 1 (2021), 7459–7478.

[55]

Faxin Qi, Xiangrong Tong, Lei Yu, and Yingjie Wang. 2019. Personalized project recommendations: Using reinforcement learning. EURASIP Journal on Wireless Communications and Networking 2019, 1 (2019), 1–17.

Digital Library

[56]

Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. 2020. Neural collaborative filtering vs. matrix factorization revisited. In Proceedings of the 14th ACM Conference on Recommender Systems (RecSys’20 Virtual Event, Brazil). ACM, New York, NY, 240–248.

Digital Library

[57]

Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. 1994. GroupLens: An open architecture for collaborative filtering of netnews. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work (CSCW’94 Chapel Hill, North Carolina). ACM, New York, NY, 175–186.

Digital Library

[58]

Francesco Ricci, Lior Rokach, and Bracha Shapira. 2011. Introduction to recommender systems handbook. In Recommender Systems Handbook. Springer, 1–35.

[59]

Herbert Robbins. 1952. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society 58, 5 (1952), 527–535.

[60]

Aleksandrs Slivkins. 2019. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning 12, 1–2 (2019), 1–286.

Digital Library

[61]

Dusan Stamenkovic, Alexandros Karatzoglou, Ioannis Arapakis, Xin Xin, and Kleomenis Katevas. 2022. Choosing the best of both worlds: Diverse and novel recommendations through multi-objective reinforcement learning. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM’22 Virtual Event, AZ). ACM, New York, NY, 957–965.

Digital Library

[62]

Yueming Sun and Yi Zhang. 2018. Conversational recommender system. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’18 Ann Arbor, MI, USA). ACM, New York, NY, 235–244.

Digital Library

[63]

Peter Sunehag, Richard Evans, Gabriel Dulac-Arnold, Yori Zwols, Daniel Visentin, and Ben Coppin. 2015. Deep reinforcement learning with attention for slate Markov decision processes with high-dimensional states and actions. arXiv preprint arXiv:1512.01124 (2015).

[64]

Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press.

Digital Library

[65]

Georgios Theocharous, Philip S. Thomas, and Mohammad Ghavamzadeh. 2015. Personalized ad recommendation systems for life-time value optimization with guarantees. In Proceedings of the 24th International Joint Conference on Artificial Intelligence. AAAI, Buenos Aires, Argentina, 20.

[66]

Emmanouil Vozalis and Konstantinos G. Margaritis. 2003. Analysis of recommender systems algorithms. In Proceedings of the 6th Hellenic European Conference on Computer Mathematics & Its Applications. 732–745.

[67]

Kai Wang, Zhene Zou, Qilin Deng, Jianrong Tao, Runze Wu, Changjie Fan, Liang Chen, and Peng Cui. 2021. Reinforcement learning with a disentangled universal value function for item recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence 35, 5, 4427–4435.

[68]

Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha. 2018. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’18 London, United Kingdom). ACM, New York, NY, 2447–2456.

Digital Library

[69]

Pengfei Wang, Yu Fan, Long Xia, Wayne Xin Zhao, Shaozhang Niu, and Jimmy Huang. 2020. KERL: A knowledge-guided reinforcement learning model for sequential recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’20 Virtual Event, China). ACM, New York, NY, 209–218.

Digital Library

[70]

Xiting Wang, Yiru Chen, Jie Yang, Le Wu, Zhengtao Wu, and Xing Xie. 2018. A reinforcement learning framework for explainable recommendation. In Proceedings of the 2018 IEEE International Conference on Data Mining (ICDM), Vol. 1, IEEE, 587–596.

[71]

Xiang Wang, Yaokun Xu, Xiangnan He, Yixin Cao, Meng Wang, and Tat-Seng Chua. 2020. Reinforced negative sampling over knowledge graph for recommendation. In Proceedings of the Web Conference 2020 (WWW’20 Taipei, Taiwan). ACM, New York, NY, 99–109.

Digital Library

[72]

Yu Wang. 2020. A hybrid recommendation for music based on reinforcement learning. Advances in Knowledge Discovery and Data Mining 12084, 1 (2020), 91–103.

Digital Library

[73]

Peter Wei, Stephen Xia, Runfeng Chen, Jingyi Qian, Chong Li, and Xiaofan Jiang. 2020. A deep-reinforcement-learning-based recommender system for occupant-driven energy optimization in commercial buildings. IEEE Internet of Things Journal 7, 7 (2020), 6402–6413.

[74]

Yikun Xian, Zuohui Fu, S. Muthukrishnan, Gerard de Melo, and Yongfeng Zhang. 2019. Reinforcement knowledge graph reasoning for explainable recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’19 Paris, France). ACM, New York, NY, 285–294.

Digital Library

[75]

Yilin Xiao, Liang Xiao, Xiaozhen Lu, Hailu Zhang, Shui Yu, and H. Vincent Poor. 2020. Deep-reinforcement-learning-based user profile perturbation for privacy-aware recommendation. IEEE Internet of Things Journal 8, 6 (2020), 4560–4568.

[76]

Ruobing Xie, Shaoliang Zhang, Rui Wang, Feng Xia, and Leyu Lin. 2021. Hierarchical reinforcement learning for integrated recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence 35, 5 (2021), 4521–4528. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16580.

[77]

Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M. Jose. 2020. Self-supervised reinforcement learning for recommender systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’20 Virtual Event, China). ACM, New York, NY, 931–940.

Digital Library

[78]

Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M. Jose. 2022. Supervised advantage actor-critic for recommender systems. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM’22 Virtual Event, AZ, USA). ACM, New York, NY, 1186–1196.

Digital Library

[79]

Tong Yu, Yilin Shen, Ruiyi Zhang, Xiangyu Zeng, and Hongxia Jin. 2019. Vision-language recommendation via attribute augmented multimodal reinforcement learning. In Proceedings of the 27th ACM International Conference on Multimedia (MM’19 Nice, France). ACM, New York, NY, 39–47.

Digital Library

[80]

Zhang Yuyan, Su Xiayao, and Liu Yong. 2019. A novel movie recommendation system based on deep reinforcement learning with prioritized experience replay. In Proceedings of the 2019 IEEE 19th International Conference on Communication Technology. Vol. 1, IEEE, 1496–1500.

[81]

Jing Zhang, Bowen Hao, Bo Chen, Cuiping Li, Hong Chen, and Jimeng Sun. 2019. Hierarchical reinforcement learning for course recommendation in MOOCs. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (2019), 435–442.

Digital Library

[82]

Chenfei Zhao and Lan Hu. 2019. CapDRL: A deep capsule reinforcement learning for movie recommendation. In Proceedings of the Pacific Rim International Conference on Artificial Intelligence. Springer, Springer International Publishing, Cham, 734–739.

Digital Library

[83]

Dongyang Zhao, Liang Zhang, Bo Zhang, Lizhou Zheng, Yongjun Bao, and Weipeng Yan. 2020. MaHRL: Multi-goals abstraction based deep hierarchical reinforcement learning for recommendations. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’20 Virtual Event, China). ACM, New York, NY, 871–880.

Digital Library

[84]

Xiangyu Zhao, Long Xia, Jiliang Tang, and Dawei Yin. 2019. Deep reinforcement learning for search, recommendation, and online advertising: A survey. ACM SIGWEB Newsletter 2019, Spring (2019), 1–15.

Digital Library

[85]

Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018. Deep reinforcement learning for page-wise recommendations. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys’18 Vancouver, British Columbia, Canada). ACM, New York, NY, 95–103.

Digital Library

[86]

Xiangyu Zhao, Long Xia, Lixin Zou, Hui Liu, Dawei Yin, and Jiliang Tang. 2020. Whole-chain recommendations. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM’20 Virtual Event, Ireland). ACM, New York, NY, 1883–1891.

Digital Library

[87]

Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. 2018. Recommendations with negative feedback via pairwise deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’18 London, United Kingdom). ACM, New York, NY, 1040–1048.

Digital Library

[88]

Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018. DRN: A deep reinforcement learning framework for news recommendation. In Proceedings of the 2018 World Wide Web Conference (WWW’18 Lyon, France). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 167–176.

Digital Library

[89]

Sijin Zhou, Xinyi Dai, Haokun Chen, Weinan Zhang, Kan Ren, Ruiming Tang, Xiuqiang He, and Yong Yu. 2020. Interactive recommender system via knowledge graph-enhanced reinforcement learning. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’20 Virtual Event, China). ACM, New York, NY, 179–188.

Digital Library

[90]

Lixin Zou, Long Xia, Zhuoye Ding, Dawei Yin, Jiaxing Song, and Weidong Liu. 2019. Reinforcement learning to diversify top-n recommendation. In Proceedings of the International Conference on Database Systems for Advanced Applications. Springer International Publishing, Cham, 104–120.

Digital Library

[91]

Lixin Zou, Long Xia, Pan Du, Zhuo Zhang, Ting Bai, Weidong Liu, Jian-Yun Nie, and Dawei Yin. 2020. Pseudo Dyna-Q: A reinforcement learning framework for interactive recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining (WSDM’20 Houston, TX, USA). ACM, New York, NY, 816–824.

Digital Library

Cited By

Jeunen OPotapov IUstimenko ABaeza-Yates RBonchi F(2024)On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n RecommendationProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671687(1222-1233)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671687
Rook LZanker MJannach DBoratto LGena CMarras MGermanakos PPopescus E(2024)Are We Losing Interest in Context-Aware Recommender Systems?Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3665190(229-230)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3665190

Index Terms

A Systematic Study on Reproducibility of Reinforcement Learning in Recommendation Systems

Recommendations

Do machine learning platforms provide out-of-the-box reproducibility?
Abstract
Science is experiencing an ongoing reproducibility crisis. In light of this crisis, our objective is to investigate whether machine learning platforms provide out-of-the-box reproducibility. Our method is twofold: First, we survey ...
Highlights
- A framework for comparing the support for reproducibility of machine learning platforms is proposed.
Towards reproducibility in recommender-systems research

Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system ...
Reproducibility in learning
STOC 2022: Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing

We introduce the notion of a reproducible algorithm in the context of learning. A reproducible learning algorithm is resilient to variations in its samples — with high probability, it returns the exact same output when run on two samples from the same ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Recommender Systems

ACM Transactions on Recommender Systems Volume 1, Issue 3

September 2023

118 pages

EISSN:2770-6699

DOI:10.1145/3609309

Editors:
Li Chen
Hong Kong Baptist University, China
,
Dietmar Jannach
University of Klagenfurt, Austria

Issue’s Table of Contents

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 July 2023

Online AM: 17 May 2023

Accepted: 05 April 2023

Revised: 28 March 2023

Received: 18 May 2022

Published in TORS Volume 1, Issue 3

Check for updates

Author Tags

Qualifiers

Survey

Funding Sources

Open Access Publishing Fund of the Free University of Bozen-Bolzano

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
607
Total Downloads

Downloads (Last 12 months)243
Downloads (Last 6 weeks)21

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jeunen OPotapov IUstimenko ABaeza-Yates RBonchi F(2024)On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n RecommendationProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671687(1222-1233)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671687
Rook LZanker MJannach DBoratto LGena CMarras MGermanakos PPopescus E(2024)Are We Losing Interest in Context-Aware Recommender Systems?Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3665190(229-230)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3665190

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents