research-article

A Generic Network Compression Framework for Sequential Recommender Systems

Authors:

Duo LiuAuthors Info & Claims

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1299 - 1308

https://doi.org/10.1145/3397271.3401125

Published: 25 July 2020 Publication History

Abstract

Sequential recommender systems (SRS) have become the key technology in capturing user's dynamic interests and generating high-quality recommendations. Current state-of-the-art sequential recommender models are typically based on a sandwich-structured deep neural network, where one or more middle (hidden) layers are placed between the input embedding layer and output softmax layer. In general, these models require a large number of parameters to obtain optimal performance. Despite the effectiveness, at some point, further increasing model size may be harder for model deployment in resource-constraint devices. To resolve the issues, we propose a compressed sequential recommendation framework, termed as CpRec, where two generic model shrinking techniques are employed. Specifically, we first propose a block-wise adaptive decomposition to approximate the input and softmax matrices by exploiting the fact that items in SRS obey a long-tailed distribution. To reduce the parameters of the middle layers, we introduce three layer-wise parameter sharing schemes. We instantiate CpRec using deep convolutional neural network with dilated kernels given consideration to both recommendation accuracy and efficiency. By the extensive ablation studies, we demonstrate that the proposed CpRec can achieve up to 4~8 times compression rates in real-world SRS datasets. Meanwhile, CpRec is faster during training & inference, and in most cases outperforms its uncompressed counterpart.

References

[1]

Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853 (2018).

[2]

Alexandre Boulch. 2017. Sharesnet: reducing residual network parameter number by sharing weights. arXiv preprint arXiv:1702.08782 (2017).

[3]

Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2018. Universal Transformers. arXiv e-prints, Article arXiv:1807.03819 (Jul 2018), bibinfonumpagesarXiv:1807.03819 pages.arxiv: cs.CL/1807.03819

[4]

Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, and Nando De Freitas. 2013. Predicting parameters in deep learning. In Advances in neural information processing systems. 2148--2156.

[5]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).

[6]

Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115 (2014).

[7]

Youyang Gu, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Learning to refine text based recommendations. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2103--2108.

[8]

Guibing Guo, Shichang Ouyang, Xiaodong He, Fajie Yuan, and Xiaohua Liu. 2019. Dynamic item block and prediction enhancing block for sequential recommendation. In Proc. Int. Joint Conf. Artif. Intell.(IJCAI).

[9]

Song Han, Huizi Mao, and William J. Dally. 2015. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv e-prints, Article arXiv:1510.00149 (Oct 2015), arXiv:1510.00149 pages.arxiv: cs.CV/1510.00149

[10]

Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015).

[11]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.

[12]

Ruining He and Julian McAuley. 2016. Fusing similarity models with markov chains for sparse sequential recommendation. In 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 191--200.

[13]

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. Proceedings of the 43th International ACM SIGIR conference on Research and Development in Information Retrieval (2020).

Digital Library

[14]

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2015. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939 (2015).

[15]

Longke Hu, Aixin Sun, and Yong Liu. 2014. Your neighbors affect your ratings: on geographical neighborhood influence to rating prediction. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval. ACM, 345--354.

Digital Library

[16]

Bin Jiang. 2013. Head/tail breaks: A new classification scheme for data with a heavy-tailed distribution. The Professional Geographer, Vol. 65, 3 (2013), 482--494.

[17]

Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM). IEEE, 197--206.

[18]

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019).

[19]

Hai-Son Le, Ilya Oparin, Alexandre Allauzen, Jean-Luc Gauvain, and Francc ois Yvon. 2011. Structured output layer neural network language model. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 5524--5527.

[20]

Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 1419--1428.

Digital Library

[21]

Pasquale Lops, Marco De Gemmis, and Giovanni Semeraro. 2011. Content-based recommender systems: State of the art and trends. In Recommender systems handbook. Springer, 73--105.

[22]

Juergen Luettin, Susanne Rothermel, and Mark Andrew. 2019. Future of in-vehicle recommendation systems@ Bosch. In Proceedings of the 13th ACM Conference on Recommender Systems. 524--524.

Digital Library

[23]

Shilin Qu, Fajie Yuan, Guibing Guo, Liguang Zhang, and Wei Wei. 2020. CmnRec: Sequential Recommendations with Chunk-accelerated Memory Network. arXiv preprint arXiv:2004.13401 (2020).

[24]

Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. 2013. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing. IEEE, 6655--6659.

[25]

Badrul Munir Sarwar, George Karypis, Joseph A Konstan, John Riedl, et almbox. 2001. Item-based collaborative filtering recommendation algorithms. Www, Vol. 1 (2001), 285--295.

Digital Library

[26]

Elena Smirnova and Flavian Vasile. 2017. Contextual sequence modeling for recommendation with recurrent neural networks. In Proceedings of the 2nd Workshop on Deep Learning for Recommender Systems. ACM, 2--9.

Digital Library

[27]

Suraj Srinivas and R Venkatesh Babu. 2015. Data-free parameter pruning for deep neural networks. arXiv preprint arXiv:1507.06149 (2015).

[28]

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. arXiv preprint arXiv:1904.06690 (2019).

[29]

Yong Kiam Tan, Xinxing Xu, and Yong Liu. 2016. Improved recurrent neural networks for session-based recommendations. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. ACM, 17--22.

Digital Library

[30]

Jiaxi Tang, Francois Belletti, Sagar Jain, Minmin Chen, Alex Beutel, Can Xu, and Ed H Chi. 2 Towards neural mixture recommender for long range dependent user sequences. In The World Wide Web Conference. ACM, 1782--1793.

[31]

Jiaxi Tang and Ke Wang. 2018a. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 565--573.

Digital Library

[32]

Jiaxi Tang and Ke Wang. 2018b. Ranking distillation: Learning compact ranking models with high performance for recommender system. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2289--2298.

Digital Library

[33]

Wenting Tu, David W Cheung, Nikos Mamoulis, Min Yang, and Ziyu Lu. 2015. Activity-partner recommendation. In PAKDD. 591--604.

[34]

Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. 2011. Improving the speed of neural networks on CPUs. (2011).

[35]

Jingyi Wang, Qiang Liu, Zhaocheng Liu, and Shu Wu. 2019. Towards Accurate and Interpretable Sequential Prediction: A CNN & Attention-Based Feature Extractor. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 1703--1712.

Digital Library

[36]

Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. 2017. Irgan: A minimax game for unifying generative and discriminative information retrieval models. In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 515--524.

Digital Library

[37]

Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. 2016. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4820--4828.

[38]

Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin, and Hongkai Xiong. 2018. Deep neural network compression with single and multiple level quantization. In Thirty-Second AAAI Conference on Artificial Intelligence.

[39]

Haochao Ying, Fuzhen Zhuang, Fuzheng Zhang, Yanchi Liu, Guandong Xu, Xing Xie, Hui Xiong, and Jian Wu. 2018. Sequential recommender system based on hierarchical attention networks. In the 27th International Joint Conference on Artificial Intelligence.

Digital Library

[40]

Fajie Yuan, Guibing Guo, Joemon M Jose, Long Chen, Haitao Yu, and Weinan Zhang. 2016. Lambdafm: learning optimal ranking with factorization machines using lambda surrogates. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. 227--236.

Digital Library

[41]

Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. 2020. Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation. arXiv (2020), arXiv--2001.

[42]

Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M Jose, and Xiangnan He. 2019. A Simple Convolutional Generative Network for Next Item Recommendation. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. ACM, 582--590.

Digital Library

[43]

Fajie Yuan, Xin Xin, Xiangnan He, Guibing Guo, Weinan Zhang, Chua Tat-Seng, and Joemon M Jose. 2018. fBGD: Learning embeddings from positive unlabeled data with BGD. (2018).

[44]

Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. arXiv preprint arXiv:1605.07146 (2016).

Cited By

Vuong Nguyen L(2024)Classifications, evaluation metrics, datasets, and domains in recommendation services: A surveyInternational Journal of Hybrid Intelligent Systems10.3233/HIS-24000320:2(85-100)Online publication date: 11-Jun-2024
https://doi.org/10.3233/HIS-240003
Qu YChen TNguyen QYin HAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Budgeted Embedding Table For Recommender SystemsProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635778(557-566)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635778
Adnan MMaboud YMahajan DNair P(2024)Heterogeneous Acceleration Pipeline for Recommendation System Training2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)10.1109/ISCA59077.2024.00081(1063-1079)Online publication date: 29-Jun-2024
https://doi.org/10.1109/ISCA59077.2024.00081
Show More Cited By

Index Terms

A Generic Network Compression Framework for Sequential Recommender Systems
1. Information systems
  1. Information retrieval

Recommendations

PAC-Bayesian framework based drop-path method for 2D discriminative convolutional network pruning
Abstract
Deep convolutional neural networks (CNNs) have demonstrated its extraordinary power on various visual tasks like object detection and classification. However, it is still challenging to deploy state-of-the-art models into real-world applications, ...
Acquiring User Information Needs for Recommender Systems
WI-IAT '13: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 03

Most recommender systems attempt to use collaborative filtering, content-based filtering or hybrid approach to recommend items to new users. Collaborative filtering recommends items to new users based on their similar neighbours, and content-based ...
A Novel Framework for Improving Recommender Diversity
International Workshop on Behavior and Social Informatics on Behavior and Social Computing - Volume 8178

Recommender systems are being used to assist users in finding relevant items from a large set of alternatives in many online applications. However, while most research up to this point has focused on improving the accuracy of recommender systems, other ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

2548 pages

ISBN:9781450380164

DOI:10.1145/3397271

General Chairs:
Jimmy Huang
York University, Canada
,
Yi Chang
Jilin University, China
,
Xueqi Cheng
Chinese Academy of Sciences, China
,
Program Chairs:
Jaap Kamps
University of Amsterdam, Netherlands
,
Vanessa Murdock
Amazon, U.S.A.
,
Ji-Rong Wen
Renmin University of China, China
,
Yiqun Liu
Tsinghua University, China

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '20

Sponsor:

SIGIR

SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval

July 25 - 30, 2020

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

35
Total Citations
View Citations
550
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)1

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Vuong Nguyen L(2024)Classifications, evaluation metrics, datasets, and domains in recommendation services: A surveyInternational Journal of Hybrid Intelligent Systems10.3233/HIS-24000320:2(85-100)Online publication date: 11-Jun-2024
https://doi.org/10.3233/HIS-240003
Qu YChen TNguyen QYin HAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Budgeted Embedding Table For Recommender SystemsProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635778(557-566)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635778
Adnan MMaboud YMahajan DNair P(2024)Heterogeneous Acceleration Pipeline for Recommendation System Training2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)10.1109/ISCA59077.2024.00081(1063-1079)Online publication date: 29-Jun-2024
https://doi.org/10.1109/ISCA59077.2024.00081
Zhang CHan QChen RZhao XTang PSong H(2024)SSDRec: Self-Augmented Sequence Denoising for Sequential Recommendation2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00067(803-815)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00067
Zhu XLi LLiu WLuo X(2024)Multi-level sequence denoising with cross-signal contrastive learning for sequential recommendationNeural Networks10.1016/j.neunet.2024.106480179(106480)Online publication date: Nov-2024
https://doi.org/10.1016/j.neunet.2024.106480
Li HZhao JHuo HFang SChen JYao LHua Y(2024)T3SRSExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122260238:PEOnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122260
Dantas PSabino da Silva WCordeiro LCarvalho C(2024)A comprehensive review of model compression techniques in machine learningApplied Intelligence10.1007/s10489-024-05747-w54:22(11804-11844)Online publication date: 2-Sep-2024
https://doi.org/10.1007/s10489-024-05747-w
Nguyen LVo QPhan T(2024)A Survey of Recommendation Systems: Datasets, Evaluation Methods, and Application DomainsIntelligent Systems Design and Applications10.1007/978-3-031-64779-6_30(311-322)Online publication date: 25-Jul-2024
https://doi.org/10.1007/978-3-031-64779-6_30
Qin ZJagerman RPasumarthi RZhuang HZhang HBai AHui KYan LWang XOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)RD-SuiteProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3667673(35748-35760)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3667673
Wang FGu HLi DLu TZhang PGu NFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Towards Deeper, Lighter and Interpretable Cross Network for CTR PredictionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615089(2523-2533)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615089
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents