research-article

An Analysis of Approaches Taken in the ACM RecSys Challenge 2018 for Automatic Music Playlist Continuation

Authors:

Ching-Wei ChenAuthors Info & Claims

ACM Transactions on Intelligent Systems and Technology (TIST), Volume 10, Issue 5

Article No.: 57, Pages 1 - 21

https://doi.org/10.1145/3344257

Published: 18 September 2019 Publication History

Abstract

The ACM Recommender Systems Challenge 2018 focused on the task of automatic music playlist continuation, which is a form of the more general task of sequential recommendation. Given a playlist of arbitrary length with some additional meta-data, the task was to recommend up to 500 tracks that fit the target characteristics of the original playlist. For the RecSys Challenge, Spotify released a dataset of one million user-generated playlists. Participants could compete in two tracks, i.e., main and creative tracks. Participants in the main track were only allowed to use the provided training set, however, in the creative track, the use of external public sources was permitted. In total, 113 teams submitted 1,228 runs to the main track; 33 teams submitted 239 runs to the creative track. The highest performing team in the main track achieved an R-precision of 0.2241, an NDCG of 0.3946, and an average number of recommended songs clicks of 1.784. In the creative track, an R-precision of 0.2233, an NDCG of 0.3939, and a click rate of 1.785 was obtained by the best team. This article provides an overview of the challenge, including motivation, task definition, dataset description, and evaluation. We further report and analyze the results obtained by the top-performing teams in each track and explore the approaches taken by the winners. We finally summarize our key findings, discuss generalizability of approaches and results to domains other than music, and list the open avenues and possible future directions in the area of automatic playlist continuation.

References

[1]

Sebastiano Antenucci, Simone Boglio, Emanuele Chioso, Ervin Dervishaj, Shuwen Kang, Tommaso Scarlatti, and Maurizio Ferrari Dacrema. 2018. Artist-driven layering and user’s behaviour impact on recommendations in a playlist continuation scenario. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[2]

Dmitry Bogdanov, Alastair Porter, Perfecto Herrera, and Xavier Serra. 2016. Cross-collection evaluation for music classification tasks. In Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR’16).

[3]

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606.

[4]

Geoffray Bonnin and Dietmar Jannach. 2015. Automated generation of music playlists: Survey and experiments. Comput. Surveys 47, 2 (2015), 26.

Digital Library

[5]

Chris J. C. Burges. 2010. From RankNet to LambdaRank to LambdaMART: An overview. Technical Report.

[6]

Robin Burke. 2002. Hybrid recommender systems: Survey and experiments. User Model. User-Adapt. Interact. 12, 4 (Nov. 2002), 331--370.

Digital Library

[7]

Ching-Wei Chen, Paul Lamere, Markus Schedl, and Hamed Zamani. 2018. RecSys challenge 2018: Automatic music playlist continuation. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys’18).

Digital Library

[8]

Ruey-Cheng Chen, Luke Gallagher, Roi Blanco, and J. Shane Culpepper. 2017. Efficient cost-aware cascade ranking in multi-stage retrieval. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’17). ACM, New York, NY, 445--454.

Digital Library

[9]

Shuo Chen, Josh L. Moore, Douglas Turnbull, and Thorsten Joachims. 2012. Playlist prediction via metric embedding. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’12). ACM, New York, NY, 714--722.

Digital Library

[10]

Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (KDD’16). ACM, New York, NY, 785--794.

Digital Library

[11]

Van Dang, Michael Bendersky, and W. Bruce Croft. 2013. Two-stage learning to rank for information retrieval. In Advances in Information Retrieval. Springer, Berlin, 423--434.

Digital Library

[12]

Jean-Charles de Borda. 1781. Mémoire sur les élections au scrutin. Histoire de l’Académie Royale des Sciences.

[13]

Michael D. Ekstrand, F. Maxwell Harper, Martijn C. Willemsen, and Joseph A. Konstan. 2014. User perception of differences in recommender algorithms. In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys’14). ACM, New York, NY, 161--168.

Digital Library

[14]

Guglielmo Faggioli, Mirko Polato, and Fabio Aiolli. 2018. Efficient similarity-based methods for the playlist continuation task. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[15]

Andres Ferraro, Dmitry Bogdanov, Jisang Yoon, Kwangseob Kim, and Xavier Serra. 2018. Automatic playlist continuation using a hybrid recommender system combining features from text and audio. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[16]

Jerome H. Friedman. 2001. Greedy function approximation: A gradient boosting machine. Ann. Stat. 29, 5 (2001), 1189--1232.

[17]

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT Press. Retrieved from http://www.deeplearningbook.org.

Digital Library

[18]

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web (WWW’17). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 173--182.

Digital Library

[19]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9, 8 (1997), 1735--1780.

Digital Library

[20]

Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In Proceedings of the 8th IEEE International Conference on Data Mining (ICDM’08). 263--272.

Digital Library

[21]

Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Trans. Info. Syst. 20, 4 (Oct. 2002), 422--446.

Digital Library

[22]

Karen Spärck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation 28 (1972), 11--21.

[23]

Surya Kallumadi, Bhaskar Mitra, and Tereza Iofciu. 2018. A line in the sand: Recommendation or ad-hoc retrieval? In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[24]

Iman Kamehkhosh, Dietmar Jannach, and Geoffray Bonnin. 2018. How automated recommendations affect the playlist creation behavior of users. In Proceedings of the 23rd ACM Conference on Intelligent User Interfaces Workshops: Intelligent Music Interfaces for Listening and Creation (MILC’18).

[25]

Mesut Kaya and Derek Bridge. 2018. Automatic playlist continuation using subprofile-aware diversification. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[26]

Domokos M. Kelen, Daniel Berecz, Ferenc Béres, and Andrés A. Benczur. 2018. Efficient K-NN for playlist continuation. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[27]

Jaehun Kim, Minz Won, Cynthia C. S. Liem, and Alan Hanjalic. 2018. Towards seed-free music playlist generation: Enhancing collaborative filtering with playlist title information. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[28]

Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the user experience of recommender systems. User Model. User-Adapt. Interact. 22, 4--5 (2012), 441--504.

Digital Library

[29]

Maciej Kula. 2015. Metadata embeddings for user and item cold-start recommendations. In Proceedings of the 2nd Workshop on New Trends on Content-based Recommender Systems co-located with 9th ACM Conference on Recommender Systems. 14--21.

[30]

Aristomenis S. Lampropoulos, Paraskevi S. Lampropoulou, and George A. Tsihrintzis. 2012. A cascade-hybrid music recommender system for mobile services based on musical genre classification and personality diagnosis. Multimedia Tools Appl. 59, 1 (July 2012), 241--258.

Digital Library

[31]

Victor Lavrenko and W. Bruce Croft. 2001. Relevance-based language models. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’01). ACM, New York, NY, 120--127.

Digital Library

[32]

Lei Li, Dingding Wang, Tao Li, Daniel Knox, and Balaji Padmanabhan. 2011. SCENE: A scalable two-stage personalized news recommendation system. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’11). ACM, New York, NY, 125--134.

Digital Library

[33]

A. J. Lonsdale and A. C. North. 2011. Why do we listen to music? A uses and gratifications analysis. Brit. J. Psychol. 102 (2011), 108--134.

[34]

Malte Ludewig, Iman Kamehkhosh, Nick Landia, and Dietmar Jannach. 2018. Effective nearest-neighbor music recommendations. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[35]

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2015. Adversarial autoencoders. arXiv preprint arXiv:1511.05644.

[36]

Brian McFee and Gert Lanckriet. 2011. The natural language of playlists. In Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR’11).

[37]

Gabriel Meseguer-Brocal, Geoffroy Peeters, Guillaume Pellerin, Michel Buffa, Elena Cabrio, Catherine Faron Zucker, Alain Giboin, Isabelle Mirbel, Romain Hennequin, Manuel Moussallam, Francesco Piccoli, and Thomas Fillon. 2017. WASABI: A two million song database project with audio and cultural metadata plus webaudio enhanced client applications. In Proceedings of the Web Audio Conference on Collaborative Audio. Queen Mary University of London.

[38]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Curran Associates, 3111--3119.

Digital Library

[39]

Diego Monti, Enrico Palumbo, Giuseppe Rizzo, Pasquale Lisena, RaphaÃ«l Troncy, Michael Fell, Elena Cabrio, and Maurizio Morisio. 2018. An ensemble approach of recurrent neural networks using pre-trained embeddings for playlist completion. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[40]

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI’09). AUAI Press, Arlington, VA, 452--461. Retrieved from http://dl.acm.org/citation.cfm?id=1795114.1795167.

Digital Library

[41]

S. E. Robertson and S. Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’94). Springer-Verlag, New York, NY, 232--241. Retrieved from http://dl.acm.org/citation.cfm?id=188490.188561.

Digital Library

[42]

Vasiliy Rubtsov, Mikhail Kamenshikov, Ilya Valyaev, Vasiliy Leksin, and Dmitry I. Ignatov. 2018. A hybrid two-stage recommender system for automatic playlist continuation. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[43]

Alan Said. 2016. A short history of the RecSys challenge. 37 (12 2016), 102--104.

[44]

Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in automatic text retrieval. Info. Process. Manage. 24, 5 (1988), 513--523.

Digital Library

[45]

Thomas Schäfer, Peter Sedlmeier, Christine Städtler, and David Huron. 2013. The psychological functions of music listening. Front. Psychol. 4 (2013).

[46]

Markus Schedl, Hamed Zamani, Ching-Wei Chen, Yashar Deldjoo, and Mehdi Elahi. 2018. Current challenges and visions in music recommender systems research. Int. J. Multimedia Info. Retriev. 7, 2 (June 2018), 95--116.

[47]

Nava Tintarev, Christoph Lofi, and Cynthia C. S. Liem. 2017. Sequences of diverse song recommendations: An exploratory study in a commercial system. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization (UMAP’17). ACM, New York, NY, 391--392.

Digital Library

[48]

Iacopo Vagliano, Lukas Galke, Florian Mai, and Ansgar Scherp. 2018. Using adversarial autoencoders for automatic playlist continuation. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[49]

Andreu Vall, Massimo Quadrana, Markus Schedl, Gerhard Widmer, and Paolo Cremonesi. 2017. The importance of song context in music playlists. In Proceedings of the 11th ACM Conference on Recommender Systems (RecSys’17).

[50]

Timo van Niedek and Arjen de Vried. 2018. Random walk with restart for automatic playlist continuation and query-specific adaptations. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[51]

Maksims Volkovs, Himanshu Rai, Zhaoyue Cheng, Ga Wu, Yichao Lu, and Scott Sanner. 2018. Two-stage model for automatic playlist continuation at scale. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[52]

Lidan Wang, Jimmy Lin, and Donald Metzler. 2011. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’11). ACM, New York, NY, 105--114.

Digital Library

[53]

Hojin Yang, Yoonki Jeong, Minjin Choi, and Jongwuk Lee. 2018. MMCF: Multimodal collaborative filtering for automatic playlist continuation. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[54]

Xing Zhao, Qingquan Song, James Caverlee, and Xia Hu. 2018. TrailMix: An ensemble recommender system for playlist curation and continuation. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

[55]

Lin Zhu, Bowen He, Mengxin Ji, Cheng Ju, and Yihong Chen. 2018. Automatic music playlist continuation via neighbor-based collaborative filtering and discriminative reweighting/reranking. In Proceedings of the ACM Recommender Systems Challenge (RecSysChallenge’18).

Digital Library

Cited By

Moravec VHynek NGavurova BKubak M(2024)Everyday artificial intelligence unveiled: Societal awareness of technological transformationOeconomia Copernicana10.24136/oc.296115:2(367-406)Online publication date: 30-Jun-2024
https://doi.org/10.24136/oc.2961
Gabbolini GBridge D(2024)Surveying More Than Two Decades of Music Information Retrieval Research on PlaylistsACM Transactions on Intelligent Systems and Technology10.1145/368839815:6(1-68)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3688398
Penha GVardasbi APalumbo EDe Nadai MBouchard H(2024)Bridging Search and Recommendation in Generative Retrieval: Does One Task Help the Other?Proceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688123(340-349)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688123
Show More Cited By

Index Terms

An Analysis of Approaches Taken in the ACM RecSys Challenge 2018 for Automatic Music Playlist Continuation
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Test collections
  2. Information systems applications
    1. Data mining
    2. Multimedia information systems

Recommendations

Recsys challenge 2018: automatic music playlist continuation
RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems

The ACM Recommender Systems Challenge 2018 focused on automatic music playlist continuation, which is a form of the more general task of sequential recommendation. Given a playlist of arbitrary length, the challenge was to recommend up to 500 tracks ...
Using Adversarial Autoencoders for Multi-Modal Automatic Playlist Continuation
RecSys Challenge '18: Proceedings of the ACM Recommender Systems Challenge 2018

The task of automatic playlist continuation is generating a list of recommended tracks that can be added to an existing playlist. By suggesting appropriate tracks, i. e., songs to add to a playlist, a recommender system can increase the user engagement ...
Automatic Playlist Continuation using Subprofile-Aware Diversification
RecSys Challenge '18: Proceedings of the ACM Recommender Systems Challenge 2018

The ACM RecSys Challenge 2018 involves the task of automatic playlist continuation (APC), aiming to help users to create and extend their own music playlists. In this paper, we explain teamrozik's approach to the Challenge. Our approach to APC is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology

ACM Transactions on Intelligent Systems and Technology Volume 10, Issue 5

Special Section on Advances in Causal Discovery and Inference and Regular Papers

September 2019

314 pages

ISSN:2157-6904

EISSN:2157-6912

DOI:10.1145/3360733

Editor:
Yu Zheng
JD Finance, China

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2019

Accepted: 01 July 2019

Revised: 01 June 2019

Received: 01 October 2018

Published in TIST Volume 10, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Center for Intelligent Information Retrieval

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
1,067
Total Downloads

Downloads (Last 12 months)190
Downloads (Last 6 weeks)31

Reflects downloads up to 20 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Moravec VHynek NGavurova BKubak M(2024)Everyday artificial intelligence unveiled: Societal awareness of technological transformationOeconomia Copernicana10.24136/oc.296115:2(367-406)Online publication date: 30-Jun-2024
https://doi.org/10.24136/oc.2961
Gabbolini GBridge D(2024)Surveying More Than Two Decades of Music Information Retrieval Research on PlaylistsACM Transactions on Intelligent Systems and Technology10.1145/368839815:6(1-68)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3688398
Penha GVardasbi APalumbo EDe Nadai MBouchard H(2024)Bridging Search and Recommendation in Generative Retrieval: Does One Task Help the Other?Proceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688123(340-349)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688123
Salganik RLiu XMa YKang JChua TBaeza-Yates RBonchi F(2024)LARP: Language Audio Relational Pre-training for Cold-Start Playlist ContinuationProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671772(2524-2535)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671772
Shigetomi RNishida HSawai KUshiama T(2024)Integrating Repeat Listening Patterns for Enhanced Music Recommendation2024 18th International Conference on Ubiquitous Information Management and Communication (IMCOM)10.1109/IMCOM60618.2024.10418302(1-7)Online publication date: 3-Jan-2024
https://doi.org/10.1109/IMCOM60618.2024.10418302
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Conclusions and Open ChallengesTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_6(143-146)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_6
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Privacy and SecurityTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_5(103-141)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_5
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)TransparencyTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_4(69-102)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_4
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Biases, Fairness, and Non-discriminationTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_3(29-67)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_3
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Regulatory InitiativesTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_2(11-27)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_2
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents