research-article

Towards Universal Sequence Representation Learning for Recommender Systems

Authors:

Wayne Xin Zhao,

Ji-Rong WenAuthors Info & Claims

KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 585 - 593

https://doi.org/10.1145/3534678.3539381

Published: 14 August 2022 Publication History

Abstract

In order to develop effective sequential recommenders, a series of sequence representation learning (SRL) methods are proposed to model historical user behaviors. Most existing SRL methods rely on explicit item IDs for developing the sequence models to better capture user preference. Though effective to some extent, these methods are difficult to be transferred to new recommendation scenarios, due to the limitation by explicitly modeling item IDs. To tackle this issue, we present a novel universal sequence representation learning approach, named UniSRec. The proposed approach utilizes the associated description text of items to learn transferable representations across different recommendation scenarios. For learning universal item representations, we design a lightweight item encoding architecture based on parametric whitening and mixture-of-experts enhanced adaptor. For learning universal sequence representations, we introduce two contrastive pre-training tasks by sampling multi-domain negatives. With the pre-trained universal sequence representation model, our approach can be effectively transferred to new recommendation domains or platforms in a parameter-efficient way, under either inductive or transductive settings. Extensive experiments conducted on real-world datasets demonstrate the effectiveness of the proposed approach. Especially, our approach also leads to a performance improvement in a cross-platform setting, showing the strong transferability of the proposed universal SRL method. The code and pre-trained model are available at: https://github.com/RUCAIBox/UniSRec.

Supplemental Material

MP4 File

Introductory video about the KDD'22 paper "Towards Universal Sequence Representation Learning for Recommender Systems" by Yupeng Hou, Renmin University of China. The whole video is around 17min.

Download
27.30 MB

References

[1]

Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. 2013. Representation Learning: A Review and New Perspectives. TPAMI (2013).

[2]

Jianxin Chang, Chen Gao, Yu Zheng, Yiqun Hui, Yanan Niu, Yang Song, Depeng Jin, and Yong Li. 2021. Sequential Recommendation with Graph Neural Networks. In SIGIR.

[3]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL.

[4]

Hao Ding, Yifei Ma, Anoop Deoras, Yuyang Wang, and Hao Wang. 2021. Zero-Shot Recommender Systems. arXiv preprint arXiv:2105.08318 (2021).

[5]

Zhankui He, Handong Zhao, Zhe Lin, Zhaowen Wang, Ajinkya Kale, and Julian McAuley. 2021. Locally constrained self-attentive sequential recommendation. In CIKM.

[6]

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016. Session-based Recommendations with Recurrent Neural Networks. In ICLR.

[7]

Yupeng Hou, Binbin Hu, Zhiqiang Zhang, and Wayne Xin Zhao. 2022. CORE: Simple and Effective Session-based Recommendation within Consistent Representation Space. In SIGIR.

[8]

Guangneng Hu, Yu Zhang, and Qiang Yang. 2018. CoNet: Collaborative Cross Networks for Cross-Domain Recommendation. In CIKM.

[9]

Junjie Huang, Duyu Tang, Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan Duan. 2021. WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach. In Findings of EMNLP.

[10]

Wang-Cheng Kang and Julian J. McAuley. 2018. Self-Attentive Sequential Recommendation. In ICDM.

[11]

Bohan Li, Hao Zhou, Junxian He, MingxuanWang, Yiming Yang, and Lei Li. 2020. On the Sentence Embeddings from Pre-trained Language Models. In EMNLP.

[12]

Chenglin Li, Mingjun Zhao, Huanming Zhang, Chenyun Yu, Lei Cheng, Guoqiang Shu, Beibei Kong, and Di Niu. 2022. RecGURU: Adversarial Learning of Generalized User Representations for Cross-Domain Recommendation. In WSDM.

[13]

Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017. Neural Attentive Session-based Recommendation. In CIKM.

[14]

Tzu-Heng Lin, Chen Gao, and Yong Li. 2019. CROSS: Cross-platform Recommendation for Social E-Commerce. In SIGIR.

Digital Library

[15]

Zihan Lin, Changxin Tian, Yupeng Hou, and Wayne Xin Zhao. 2022. Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning. In TheWebConf.

[16]

Shanlei Mu, Yupeng Hou, Wayne Xin Zhao, Yaliang Li, and Bolin Ding. 2022. ID-Agnostic User Behavior Pre-training for Sequential Recommendation. arXiv preprint arXiv:2206.02323 (2022).

[17]

Jianmo Ni, Jiacheng Li, and Julian J. McAuley. 2019. Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects. In EMNLP.

[18]

Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factorizing personalized Markov chains for next-basket recommendation. In WWW.

[19]

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In ICLR.

[20]

Ajit Paul Singh and Geoffrey J. Gordon. 2008. Relational learning via collective matrix factorization. In SIGKDD.

[21]

Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening Sentence Representations for Better Semantics and Faster Retrieval. arXiv preprint arXiv:2103.15316 (2021).

[22]

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In CIKM.

[23]

Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020. Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations. In RecSys.

[24]

Jiaxi Tang and Ke Wang. 2018. Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding. In WSDM.

[25]

Jie Tang, Sen Wu, Jimeng Sun, and Hang Su. 2012. Cross-domain collaboration recommendation. In SIGKDD.

[26]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS.

[27]

ShuWu, Yuyuan Tang, Yanqiao Zhu, LiangWang, Xing Xie, and Tieniu Tan. 2019. Session-Based Recommendation with Graph Neural Networks. In AAAI.

[28]

Ruobing Xie, Qi Liu, Liangdong Wang, Shukai Liu, Bo Zhang, and Leyu Lin. 2022. Contrastive Cross-domain Recommendation in Matching. In SIGKDD.

[29]

Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Bolin Ding, and Bin Cui. 2022. Contrastive learning for sequential recommendation. In ICDE.

[30]

Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Jundong Li, and Zi Huang. 2022. Self-Supervised Learning for Recommender Systems: A Survey. arXiv preprint arXiv:2203.15876 (2022).

[31]

Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. 2020. Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation. In SIGIR.

[32]

Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S. Sheng, Jiajie Xu, Deqing Wang, Guanfeng Liu, and Xiaofang Zhou. 2019. Feature-level Deeper Self-Attention Network for Sequential Recommendation. In IJCAI.

[33]

Wayne Xin Zhao, Junhua Chen, Pengfei Wang, Qi Gu, and Ji-Rong Wen. 2020. Revisiting Alternative Experimental Settings for Evaluating Top-N Item Recommendation Algorithms. In CIKM.

[34]

Wayne Xin Zhao, Yanwei Guo, Yulan He, Han Jiang, YuexinWu, and Xiaoming Li. 2014. We know what you want to buy: a demographic-based system for product recommendation on microblogs. In KDD.

[35]

Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, Yingqian Min, Zhichao Feng, Xinyan Fan, Xu Chen, Pengfei Wang, Wendi Ji, Yaliang Li, Xiaoling Wang, and Ji-RongWen. 2021. RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms. In CIKM.

[36]

Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. 2020. S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization. In CIKM.

[37]

Kun Zhou, Hui Yu, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Filter-enhanced MLP is All You Need for Sequential Recommendation. In TheWebConf.

[38]

Feng Zhu, Chaochao Chen, Yan Wang, Guanfeng Liu, and Xiaolin Zheng. 2019. DTCDR: A Framework for Dual-Target Cross-Domain Recommendation. In CIKM.

Digital Library

[39]

Feng Zhu, Yan Wang, Chaochao Chen, Jun Zhou, Longfei Li, and Guanfeng Liu. 2021. Cross-Domain Recommendation: Challenges, Progress, and Prospects. In IJCAI.

Cited By

Boz AZorgdrager WKotti ZHarte JLouridas PKarakoidas VJannach DFragkoulis M(2025)Improving Sequential Recommendations with LLMsACM Transactions on Recommender Systems10.1145/3711667Online publication date: 10-Jan-2025
https://doi.org/10.1145/3711667
Liu QQiu ZZhao XWu XZhang ZXu TTian F(2025)A Contrastive Pretrain Model with Prompt Tuning for Multi-center Medication RecommendationACM Transactions on Information Systems10.1145/370663143:3(1-29)Online publication date: 3-Jan-2025
https://dl.acm.org/doi/10.1145/3706631
Li HDu HLi YFu JLi CZhuang ZLi JNi YNejdl WAuer SKarras OCha MMoens MNajork M(2025)Teach Me How to Denoise: A Universal Framework for Denoising Multi-modal Recommender Systems via Guided CalibrationProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703507(782-791)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703507
Show More Cited By

Index Terms

Towards Universal Sequence Representation Learning for Recommender Systems
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems

Recommendations

Review-Enhanced Universal Sequence Representation Learning for Recommender Systems
With the widespread deployment of recommender systems on various online platforms, researchers are striving to develop transferable recommendation algorithms that can effectively adapt to new task scenarios without requiring the re-training of new ...
Multi-Sequence Attentive User Representation Learning for Side-information Integrated Sequential Recommendation
WSDM '24: Proceedings of the 17th ACM International Conference on Web Search and Data Mining

Side-information integrated sequential recommendation incorporates supplementary information to alleviate the issue of data sparsity. The state-of-the-art works mainly leverage some side information to improve the attention calculation to learn user ...
Tutorial: Sequence-Aware Recommender Systems
WWW '19: Companion Proceedings of The 2019 World Wide Web Conference

Recommender systems are widely used in online applications to help users find items of interest and help them deal with information overload. In this tutorial, we discuss the class of sequence-aware recommender systems. Differently from the traditional ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2022

5033 pages

ISBN:9781450393850

DOI:10.1145/3534678

General Chairs:
Aidong Zhang
University of Virginia
,
Huzefa Rangwala
Amazon/George Mason University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '22

Sponsor:

KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 14 - 18, 2022

Washington DC, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

103
Total Citations
View Citations
1,924
Total Downloads

Downloads (Last 12 months)692
Downloads (Last 6 weeks)58

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Boz AZorgdrager WKotti ZHarte JLouridas PKarakoidas VJannach DFragkoulis M(2025)Improving Sequential Recommendations with LLMsACM Transactions on Recommender Systems10.1145/3711667Online publication date: 10-Jan-2025
https://doi.org/10.1145/3711667
Liu QQiu ZZhao XWu XZhang ZXu TTian F(2025)A Contrastive Pretrain Model with Prompt Tuning for Multi-center Medication RecommendationACM Transactions on Information Systems10.1145/370663143:3(1-29)Online publication date: 3-Jan-2025
https://dl.acm.org/doi/10.1145/3706631
Li HDu HLi YFu JLi CZhuang ZLi JNi YNejdl WAuer SKarras OCha MMoens MNajork M(2025)Teach Me How to Denoise: A Universal Framework for Denoising Multi-modal Recommender Systems via Guided CalibrationProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703507(782-791)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703507
Wang JKaratzoglou AArapakis IJose JNejdl WAuer SKarras OCha MMoens MNajork M(2025)Large Language Model driven Policy Exploration for Recommender SystemsProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703496(107-116)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703496
Lin JDai XXi YLiu WChen BZhang HLiu YWu CLi XZhu CGuo HYu YTang RZhang W(2025)How Can Recommender Systems Benefit from Large Language Models: A SurveyACM Transactions on Information Systems10.1145/367800443:2(1-47)Online publication date: 18-Jan-2025
https://dl.acm.org/doi/10.1145/3678004
Tang MCui SJin ZLiang SLi CZou L(2025)Sequential recommendation by reprogramming pretrained transformerInformation Processing & Management10.1016/j.ipm.2024.10393862:1(103938)Online publication date: Jan-2025
https://doi.org/10.1016/j.ipm.2024.103938
Alshbanat HBenhidour HKerrache S(2025)A survey of latent factor models in recommender systemsInformation Fusion10.1016/j.inffus.2024.102905117:COnline publication date: 1-May-2025
https://dl.acm.org/doi/10.1016/j.inffus.2024.102905
Yang JDai COu DLi DHuang JZhan DZeng XYang Y(2025)COURIER: contrastive user intention reconstruction for large-scale visual recommendationFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-024-3939-x19:7Online publication date: 1-Jul-2025
https://dl.acm.org/doi/10.1007/s11704-024-3939-x
Liu YQi JYu Y(2025)A Review on Deep Learning for Sequential Recommender Systems: Key Technologies and DirectionsBig Data10.1007/978-981-96-1024-2_22(305-318)Online publication date: 24-Jan-2025
https://doi.org/10.1007/978-981-96-1024-2_22
Peng BLing XChen ZSun HNing XSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)eCeLLMProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693702(40215-40257)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693702
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten