Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article
Free access
Just Accepted

A Privacy Preserving System for Movie Recommendations Using Federated Learning

Online AM: 24 November 2023 Publication History

Abstract

Recommender systems have become ubiquitous in the past years. They solve the tyranny of choice problem faced by many users, and are utilized by many online businesses to drive engagement and sales. Besides other criticisms, like creating filter bubbles within social networks, recommender systems are often reproved for collecting considerable amounts of personal data. However, to personalize recommendations, personal information is fundamentally required. A recent distributed learning scheme called federated learning has made it possible to learn from personal user data without its central collection. Consequently, we present a recommender system for movie recommendations, which provides privacy and thus trustworthiness on multiple levels: First and foremost, it is trained using federated learning and thus, by its very nature, privacy-preserving, while still enabling users to benefit from global insights. Furthermore, a novel federated learning scheme, called FedQ, is employed, which not only addresses the problem of non-i.i.d.-ness and small local datasets, but also prevents input data reconstruction attacks by aggregating client updates early. Finally, to reduce the communication overhead, compression is applied, which significantly compresses the exchanged neural network parametrizations to a fraction of their original size. We conjecture that this may also improve data privacy through its lossy quantization stage.

References

[1]
Md. Hijbul Alam, Woo-Jong Ryu, and SangKeun Lee. 2016. Joint multi-grain topic sentiment: modeling semantic aspects for online reviews. Information Sciences 339(2016), 206–223. https://doi.org/10.1016/j.ins.2016.01.013
[2]
Zareen Alamgir, Farwa K. Khan, and Saira Karim. 2022. Federated Recommenders: Methods, Challenges and Future. Cluster Computing 25, 6 (June 2022), 4075–4096. https://doi.org/10.1007/s10586-022-03644-w
[3]
Muhammad Ammad-ud-din, Elena Ivannikova, Suleiman A. Khan, Were Oyomno, Qiang Fu, Kuan Eeik Tan, and Adrian Flanagan. 2019. Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System. arXiv e-prints abs/1901.09888 (Jan. 2019). arxiv:1901.09888  [cs.IR]
[4]
Muhammad Asad, Saima Shaukat, Ehsan Javanmardi, Jin Nakazato, and Manabu Tsukada. 2023. A Comprehensive Survey on Privacy-Preserving Techniques in Federated Recommendation Systems. Applied Sciences 13, 10 (2023). https://doi.org/10.3390/app13106201
[5]
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2017. Learning Differentially Private Recurrent Language Models. arXiv e-prints abs/1710.06963 (Oct. 2017). arxiv:1710.06963  [cs.LG]
[6]
Sebastian Caldas, J. Konečný, H. Brendan McMahan, and Ameet Talwalkar. 2018. Expanding the Reach of Federated Learning by Reducing Client Resource Requirements. arXiv e-prints abs/1812.07210 (Dec. 2018). arxiv:1812.07210  [cs.LG]
[7]
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. 2018. LEAF: A Benchmark for Federated Settings. CoRR abs/1812.01097(Dec. 2018). https://doi.org/10.48550/arXiv.1812.01097 arxiv:1812.01097  [cs.LG]
[8]
Mei Cao, Yujie Zhang, Zezhong Ma, and Mengying Zhao. 2022. C2S: Class-aware client selection for effective aggregation in federated learning. High-Confidence Computing 2, 3 (2022), 100068. https://doi.org/10.1016/j.hcc.2022.100068
[9]
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to Rank: From Pairwise Approach to Listwise Approach. In Proceedings of the 24th International Conference on Machine Learning (Corvalis, Oregon, USA) (ICML ’07). Association for Computing Machinery, New York, NY, USA, 129–136. https://doi.org/10.1145/1273496.1273513
[10]
Di Chai, Leye Wang, Kai Chen, and Qiang Yang. 2020. FedEval: A Benchmark System with a Comprehensive Evaluation Model for Federated Learning. https://doi.org/10.48550/ARXIV.2011.09655
[11]
D. Chai, L. Wang, K. Chen, and Q. Yang. 2021. Secure Federated Matrix Factorization. IEEE Intelligent Systems 36, 05 (Sept. 2021), 11–20. https://doi.org/10.1109/MIS.2020.3014880
[12]
Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. 2018. Federated Meta-Learning with Fast Convergence and Efficient Communication. arXiv e-prints 1802.07876 (Feb. 2018). https://doi.org/10.48550/arXiv.1802.07876 arxiv:1802.07876  [cs.LG]
[13]
Tianqi Chen, Zhao Zheng, Qiuxia Lu, Weinan Zhang, and Yong Yu. 2011. Feature-Based Matrix Factorization. https://doi.org/10.48550/ARXIV.1109.2271
[14]
Wenlin Chen, Samuel Horvath, and Peter Richtarik. 2020. Optimal Client Sampling for Federated Learning. arXiv e-prints abs/2010.13723 (Oct. 2020). arxiv:2010.13723  [cs.LG]
[15]
Kyunghyuna Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. Association for Computational Linguistics, Doha, Qatar, 103–111. https://doi.org/10.3115/v1/W14-4012
[16]
Byeongjin Choe, Taegwan Kang, and Kyomin Jung. 2021. Recommendation System With Hierarchical Recurrent Neural Network for Long-Term Time Series. IEEE Access 9(2021), 72033–72039. https://doi.org/10.1109/ACCESS.2021.3079922
[17]
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. 2016. Towards the Limit of Network Quantization. arXiv e-prints abs/1612.01543 (Dec. 2016). https://doi.org/10.48550/arXiv.1612.01543 arxiv:1612.01543  [cs.CV]
[18]
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. 2017. EMNIST: Extending MNIST to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN) (Anchorage, Alaska, United States of America). Institute of Electrical and Electronics Engineers (IEEE), 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, 2921–2926. https://doi.org/10.1109/IJCNN.2017.7966217
[19]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (Boston, Massachusetts, USA) (RecSys ’16). ACM (Association for Computer Machinery), New York, NY, USA, 191–198. https://doi.org/10.1145/2959100.2959190
[20]
Ronald Cramer, Ivan Bjerre Damgård, et al. 2015. Secure multiparty computation. Cambridge University Press, Cambridge, United Kingdom.
[21]
Dimitar I. Dimitrov, Mislav Balunović, Nikola Konstantinov, and Martin Vechev. 2022. Data Leakage in Federated Averaging. arXiv e-prints abs/2206.12395 (2022). https://doi.org/10.48550/ARXIV.2206.12395
[22]
Cynthia Dwork. 2008. Differential Privacy: A Survey of Results. In Theory and Applications of Models of Computation, Manindra Agrawal, Dingzhu Du, Zhenhua Duan, and Angsheng Li (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1–19.
[23]
Cynthia Dwork and Aaron Roth. 2014. The Algorithmic Foundations of Differential Privacy. Found. Trends Theor. Comput. Sci. 9, 3-4 (Aug. 2014), 211–407. https://doi.org/10.1561/0400000042
[24]
European Parliament. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). European Union. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679
[25]
Haokun Fang and Qian Quan. 2021. Privacy Preserving Machine Learning with Homomorphic Encryption and Federated Learning. Future Internet 13, 4 (2021), 94. https://doi.org/10.3390/fi13040094
[26]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (Sydney, NSW, Australia) (ICML’17). JMLR.org, 1269 Law Street, San Diego, CA 92109, 1126–1135.
[27]
Adrian Flanagan, Were Oyomno, Alexander Grigorievskiy, Kuan E. Tan, Suleiman A. Khan, and Muhammad Ammad-Ud-Din. 2021. Federated Multi-view Matrix Factorization for Personalized Recommendations. In Machine Learning and Knowledge Discovery in Databases. Springer International Publishing, Ghent, Belgium, 324–347. https://doi.org/10.1007/978-3-030-67661-2_20
[28]
Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. 2021. A General Theory for Client Sampling in Federated Learning. arXiv e-prints abs/2107.12211 (July 2021). arxiv:2107.12211  [cs.LG]
[29]
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. 2020. Inverting Gradients - How Easy is It to Break Privacy in Federated Learning?. In Proceedings of the 34th International Conference on Neural Information Processing Systems(Vancouver, British Columbia, Canada) (NIPS’20). Curran Associates Inc., Red Hook, NY, USA, Article 1421, 11 pages.
[30]
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. 2021. A Survey of Quantization Methods for Efficient Neural Network Inference. arXiv e-prints abs/2103.13630 (March 2021). arxiv:2103.13630  [cs.CV]
[31]
Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter Sentiment Classification using Distant Supervision. CS224N Project Report. Stanford.
[32]
Jennifer Golbeck. 2016. User Privacy Concerns with Common Data Used in Recommender Systems. In Social Informatics, Emma Spiro and Yong-Yeol Ahn (Eds.). Springer International Publishing, Cham, 468–480.
[33]
Carlos A. Gomez-Uribe and Neil Hunt. 2016. The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Trans. Manage. Inf. Syst. 6, 4, Article 13(Dec. 2016), 19 pages. https://doi.org/10.1145/2843948
[34]
Mihajlo Grbovic and Haibin Cheng. 2018. Real-Time Personalization Using Embeddings for Search Ranking at Airbnb. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 311–320. https://doi.org/10.1145/3219819.3219885
[35]
Patrick J. Grother and Kayee K. Hanaoka. 1995. NIST special database 19 handprinted forms and characters database. Technical Report. National Institute of Standards and Technology. https://doi.org/10.18434/T4H01C
[36]
Paul Haase, Daniel Becking, Heiner Kirchhoffer, Karsten Müller, Heiko Schwarz, Wojciech Samek, Detlev Marpe, and Thomas Wiegand. 2021. Encoder Optimizations For The NNR Standard On Neural Network Compression. In 2021 IEEE International Conference on Image Processing (ICIP) (Anchorage, Alaska, USA). Institute of Electrical and Electronics Engineers (IEEE), 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, 3522–3526. https://doi.org/10.1109/ICIP42928.2021.9506655
[37]
Song Han, Huizi Mao, and William J. Dally. 2016. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. In 4th International Conference on Learning Representations, ICLR, May 2-4, 2016, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). ICLR, San Juan, Puerto Rico. http://arxiv.org/abs/1510.00149
[38]
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. 2018. Federated Learning for Mobile Keyboard Prediction. arXiv e-prints abs/1811.03604 (Nov. 2018). arxiv:1811.03604  [cs.CL]
[39]
F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4, Article 19(Dec. 2015), 19 pages. https://doi.org/10.1145/2827872
[40]
Chaoyang He, Keshav Balasubramanian, Emir Ceyani, Carl Yang, Han Xie, Lichao Sun, Lifang He, Liangwei Yang, Philip S. Yu, Yu Rong, Peilin Zhao, Junzhou Huang, Murali Annavaram, and Salman Avestimehr. 2021. FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks. arXiv e-prints abs/2104.07145 (April 2021). arxiv:2104.07145  [cs.LG]
[41]
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. In Proceedings of the 26th International Conference on World Wide Web (Perth, Australia) (WWW ’17). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 173–182. https://doi.org/10.1145/3038912.3052569
[42]
Erik Hermann. 2022. Artificial intelligence and mass personalization of communication content—An ethical and literacy perspective. New Media & Society 24, 5 (2022), 1258–1277. https://doi.org/10.1177/14614448211022702 arXiv:https://doi.org/10.1177/14614448211022702
[43]
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop. Morgan-Kaufmann, Montréal, Québec, Canada. http://arxiv.org/abs/1503.02531
[44]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (Nov. 1997), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
[45]
Ming Hu, Tian Liu, Zhiwei Ling, Zhihao Yue, and Mingsong Chen. 2022. FedCAT: Towards Accurate Federated Learning via Device Concatenation. arXiv e-prints abs/2202.12751 (Feb. 2022). arxiv:2202.12751  [cs.LG]
[46]
International Organization for Standardization (ISO). 2022. Information technology - Multimedia content description interface — Part 17: Compression of neural networks for multimedia content description and analysis. Standard. International Organization for Standardization (ISO), Geneva, Switzerland.
[47]
Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 (Lille, France) (ICML’15). JMLR.org, 1269 Law Street, San Diego, CA 92109, 448–456.
[48]
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. 2018. Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data. arXiv e-prints abs/1811.11479 (Nov. 2018). arxiv:1811.11479  [cs.LG]
[49]
Junjie Jia and Zhipeng Lei. 2021. Personalized Recommendation Algorithm for Mobile Based on Federated Matrix Factorization. Journal of Physics: Conference Series 1802, 3 (March 2021), 032021. https://doi.org/10.1088/1742-6596/1802/3/032021
[50]
Zhiyong Jie, Shuhong Chen, Junqiu Lai, Muhammad Arif, and Zongyuan He. 2022. Personalized federated recommendation system with historical parameter clustering. Journal of Ambient Intelligence and Humanized Computing 14 (02 2022). https://doi.org/10.1007/s12652-022-03709-z
[51]
Michael Kamp, Jonas Fischer, and Jilles Vreeken. 2021. Federated Learning from Small Datasets. arXiv e-prints abs/2110.03469 (Oct. 2021). arxiv:2110.03469  [cs.LG]
[52]
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. In Proceedings of the 37th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.  119), Hal Daumé III and Aarti Singh (Eds.). PMLR, virtual, 5132–5143. https://proceedings.mlr.press/v119/karimireddy20a.html
[53]
J. Kiefer and J. Wolfowitz. 1952. Stochastic Estimation of the Maximum of a Regression Function. The Annals of Mathematical Statistics 23, 3 (1952), 462–466. http://www.jstor.org/stable/2236690
[54]
Jinsu Kim, Dongyoung Koo, Yuna Kim, Hyunsoo Yoon, Junbum Shin, and Sungwook Kim. 2018. Efficient Privacy-Preserving Matrix Factorization for Recommendation via Fully Homomorphic Encryption. ACM Trans. Priv. Secur. 21, 4, Article 17(jun 2018), 30 pages. https://doi.org/10.1145/3212509
[55]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). International Conference on Learning Representations, 2710 E Corridor Drive, Appleton, WI 54913. http://arxiv.org/abs/1412.6980
[56]
Heiner Kirchhoffer, Paul Haase, Wojciech Samek, Karsten Müller, Hamed Rezazadegan-Tavakoli, Francesco Cricri, Emre B. Aksu, Miska M. Hannuksela, Wei Jiang, Wei Wang, Shan Liu, Swayambhoo Jain, Shahab Hamidi-Rad, Fabien Racapé, and Werner Bailer. 2022. Overview of the Neural Network Compression and Representation (NNR) Standard. IEEE Transactions on Circuits and Systems for Video Technology 32, 5(2022), 3203–3216. https://doi.org/10.1109/TCSVT.2021.3095970
[57]
J. Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. 2016. Federated Learning: Strategies for Improving Communication Efficiency. CoRR abs/1610.05492(2016). arXiv:1610.05492 http://arxiv.org/abs/1610.05492
[58]
Jakub Konečný, Hugh Brendan McMahan, Daniel Ramage, and Peter Richtárik. 2016. Federated Optimization: Distributed Machine Learning for On-Device Intelligence. CoRR abs/1610.02527(Oct. 2016). arxiv:1610.02527 http://arxiv.org/abs/1610.02527
[59]
Yehuda Koren. 2008. Factorization Meets the Neighborhood: A Multifaceted Collaborative Filtering Model. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Las Vegas, Nevada, USA) (KDD ’08). Association for Computing Machinery, New York, NY, USA, 426–434. https://doi.org/10.1145/1401890.1401944
[60]
Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix Factorization Techniques for Recommender Systems. Computer 42, 8 (Aug. 2009), 30–37. https://doi.org/10.1109/MC.2009.263
[61]
Anastasia Kozyreva, Philipp Lorenz-Spreen, Ralph Hertwig, Stephan Lewandowsky, and Stefan M Herzog. 2021. Public attitudes towards algorithmic personalization and use of personal data online: Evidence from Germany, Great Britain, and the United States. Humanities and Social Sciences Communications 8, 1(2021), 1–11.
[62]
Shyong K. “Tony” Lam, Dan Frankowski, and John Riedl. 2006. Do You Trust Your Recommendations? An Exploration of Security and Privacy Issues in Recommender Systems. In Emerging Trends in Information and Communication Security, Günter Müller (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 14–29.
[63]
Natalie Lang and Nir Shlezinger. 2022. Joint Privacy Enhancement and Quantization in Federated Learning. In 2022 IEEE International Symposium on Information Theory (ISIT) (Aalto University, Espoo, Finland). Institute of Electrical and Electronics Engineers (IEEE), 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, 2040–2045. https://doi.org/10.1109/ISIT50566.2022.9834551
[64]
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324. https://doi.org/10.1109/5.726791
[65]
Yann LeCun, John Denker, and Sara Solla. 1990. Optimal Brain Damage. In Advances in Neural Information Processing Systems, D. Touretzky (Ed.), Vol.  2. Morgan-Kaufmann, Denver, Colorado, USA. https://proceedings.neurips.cc/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf
[66]
David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau. 2019. Federated Learning for Keyword Spotting. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (Brighton, United Kingdom). Institute of Electrical and Electronics Engineers (IEEE), 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, 6341–6345. https://doi.org/10.1109/ICASSP.2019.8683546
[67]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2018. Federated Optimization in Heterogeneous Networks. https://doi.org/10.48550/ARXIV.1812.06127
[68]
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. 2020. Fair Resource Allocation in Federated Learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, Addis Ababa, Ethiopia. https://openreview.net/forum?id=ByexElSYDr
[69]
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. 2019. On the Convergence of FedAvg on Non-IID Data. https://doi.org/10.48550/ARXIV.1907.02189
[70]
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. 2021. FedBN: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623 abs/2102.07623 (2021).
[71]
Feng Liang, Weike Pan, and Zhong Ming. 2021. Fedrec++: Lossless federated recommendation with explicit feedback. In Proceedings of the AAAI conference on artificial intelligence, Vol.  35. AAAI Press, Washington, DC, USA, 4224–4231.
[72]
Bill Yuchen Lin, Chaoyang He, Zihang Ze, Hulin Wang, Yufen Hua, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. 2022. FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks. In Findings of the Association for Computational Linguistics: NAACL 2022. Association for Computational Linguistics, Seattle, United States, 157–175. https://doi.org/10.18653/v1/2022.findings-naacl.13
[73]
Darryl D. Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. 2016. Fixed Point Quantization of Deep Convolutional Networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (New York, NY, USA) (ICML’16). JMLR.org, 1269 Law Street, San Diego, CA 92109, 2849–2858.
[74]
Guanyu Lin, Feng Liang, Weike Pan, and Zhong Ming. 2021. FedRec: Federated Recommendation With Explicit Feedback. IEEE Intelligent Systems 36, 5 (Sept. 2021), 21–30. https://doi.org/10.1109/MIS.2020.3017205
[75]
Zhaohao Lin, Weike Pan, and Zhong Ming. 2021. FR-FMSS: Federated Recommendation via Fake Marks and Secret Sharing. In Proceedings of the 15th ACM Conference on Recommender Systems (Amsterdam, Netherlands) (RecSys ’21). Association for Computing Machinery, New York, NY, USA, 668–673. https://doi.org/10.1145/3460231.3478855
[76]
Tie-Yan Liu. 2009. Learning to Rank for Information Retrieval. Foundations and Trends in Information Retrieval 3, 3 (March 2009), 225–331. https://doi.org/10.1561/1500000016
[77]
Yang Liu, Yan Kang, Chaoping Xing, Tianjian Chen, and Qiang Yang. 2020. A Secure Federated Transfer Learning Framework. IEEE Intelligent Systems 35, 4 (2020), 70–82. https://doi.org/10.1109/MIS.2020.2988525
[78]
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep Learning Face Attributes in the Wild. In 2015 IEEE International Conference on Computer Vision (ICCV) (Santiago, Chile). Institute of Electrical and Electronics Engineers (IEEE), 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, 3730–3738. https://doi.org/10.1109/ICCV.2015.425
[79]
S. Lloyd. 1982. Least squares quantization in PCM. IEEE Transactions on Information Theory 28, 2 (1982), 129–137. https://doi.org/10.1109/TIT.1982.1056489
[80]
Edward Loper and Steven Bird. 2002. NLTK: The Natural Language Toolkit. arXiv e-prints cs/0205028 (May 2002). https://doi.org/10.48550/arXiv.cs/0205028 arxiv:cs/0205028  [cs.CL]
[81]
Jiahuan Luo, Xueyang Wu, Yun Luo, Anbu Huang, Yunfeng Huang, Yang Liu, and Qiang Yang. 2019. Real-World Image Datasets for Federated Learning. arXiv e-prints abs/1910.11089 (Oct. 2019). arxiv:1910.11089  [cs.CV]
[82]
Ian MacKenzie, Chris Meyer, and Steve Noble. 2013. How retailers can keep up with consumers. McKinsey & Company. https://www.mckinsey.com/industries/retail/our-insights/how-retailers-can-keep-up-with-consumers
[83]
Hugh Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2016. Communication-Efficient Learning of Deep Networks from Decentralized Data. arXiv e-prints abs/1602.05629 (Feb. 2016). arxiv:1602.05629  [cs.LG]
[84]
Lorenzo Minto, Moritz Haller, Benjamin Livshits, and Hamed Haddadi. 2021. Stronger privacy for federated collaborative filtering with implicit feedback. In Proceedings of the 15th ACM Conference on Recommender Systems. ACM (Association for Computer Machinery), New York, NY, USA, 342–350.
[85]
Moving Picture Experts Group (MPEG) working group of ISO/IEC. 2021. MPEG-7: Compression of Neural Networks for Multimedia Content Description and analysis. Standard. Moving Picture Experts Group (MPEG) working group of ISO/IEC, Hannover, DE.
[86]
Khalil Muhammad, Qinqin Wang, Diarmuid O’Reilly-Morgan, Elias Tragos, Barry Smyth, Neil Hurley, James Geraci, and Aonghus Lawlor. 2020. FedFast: Going Beyond Average for Faster Training of Federated Recommender Systems. In KDD ’20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Virtual Event, CA, USA) (KDD ’20). Association for Computing Machinery, New York, New York, USA, 1234–1242. https://doi.org/10.1145/3394486.3403176
[87]
David Neumann, Felix Sattler, Heiner Kirchhoffer, Simon Wiedemann, Karsten Müller, Heiko Schwarz, Thomas Wiegand, Detlev Marpe, and Wojciech Samek. 2020. DeepCABAC: Plug&Play Compression of Neural Network Weights and Weight Updates. In IEEE International Conference on Image Processing, ICIP 2020, October 25-28, 2020. IEEE, Abu Dhabi, United Arab Emirates, 21–25. https://doi.org/10.1109/ICIP40778.2020.9190821
[88]
Pretom Roy Ovi, Emon Dey, Nirmalya Roy, and Aryya Gangopadhyay. 2022. Leveraging Mixed Precision Quantization to Tackle Gradient Leakage Attacks in Federated Learning. arXiv e-prints abs/2210.13457 (Oct. 2022). https://doi.org/10.48550/arXiv.2210.13457 arxiv:2210.13457  [cs.LG]
[89]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., Vancouver, British Columbia, Canada, 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
[90]
Vasileios Perifanis and Pavlos S. Efraimidis. 2022. Federated Neural Collaborative Filtering. Know.-Based Syst. 242, C (April 2022), 16 pages. https://doi.org/10.1016/j.knosys.2022.108441
[91]
Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. 2018. Privacy-Preserving Deep Learning via Additively Homomorphic Encryption. IEEE Transactions on Information Forensics and Security 13, 5(2018), 1333–1345. https://doi.org/10.1109/TIFS.2017.2787987
[92]
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. 2019. FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization. https://doi.org/10.48550/ARXIV.1909.13014
[93]
Mónica Ribero, Jette Henderson, Sinead Williamson, and Haris Vikalo. 2022. Federating Recommendations Using Differentially Private Prototypes. Pattern Recogn. 129, C (Sept. 2022), 14 pages. https://doi.org/10.1016/j.patcog.2022.108746
[94]
Herbert Robbins and Sutton Monro. 1951. A Stochastic Approximation Method. The Annals of Mathematical Statistics 22, 3 (1951), 400–407. http://www.jstor.org/stable/2236626
[95]
Christian Rønn Hansen, Gareth Price, Matthew Field, Nis Sarup, Ruta Zukauskaite, Jørgen Johansen, Jesper Grau Eriksen, Farhannah Aly, Andrew McPartlin, Lois Holloway, David Thwaites, and Carsten Brink. 2022. Larynx cancer survival model developed through open-source federated learning. Radiotherapy and Oncology 176 (2022), 179–186. https://doi.org/10.1016/j.radonc.2022.09.023
[96]
Felix Sattler, Simon Wiedemann, Klaus Robert Müller, and Wojciech Samek. 2019. Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication. In 2019 International Joint Conference on Neural Networks, IJCNN 2019 (Proceedings of the International Joint Conference on Neural Networks). Institute of Electrical and Electronics Engineers Inc., Budapest, Hungary. https://doi.org/10.1109/IJCNN.2019.8852172
[97]
Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek. 2020. Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data. IEEE Transactions on Neural Networks and Learning Systems 31, 9(2020), 3400–3413. https://doi.org/10.1109/TNNLS.2019.2944481
[98]
Michael Schrage. 2017. Great Digital Companies Build Great Recommendation Engines. Harvard Business Review. https://hbr.org/2017/08/great-digital-companies-build-great-recommendation-engines
[99]
Barry Schwartz. 2004. The Tyranny of Choice. Scientific American 290, 4 (April 2004), 70–75. SCAMAC https://doi.org/10.1038/scientificamerican0404-70
[100]
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. 2015. AutoRec: Autoencoders Meet Collaborative Filtering. In Proceedings of the 24th International Conference on World Wide Web (Florence, Italy) (WWW ’15 Companion). Association for Computing Machinery, New York, NY, USA, 111–112. https://doi.org/10.1145/2740908.2742726
[101]
Mihye Seol and Taejoon Kim. 2023. Performance Enhancement in Federated Learning by Reducing Class Imbalance of Non-IID Data. Sensors 23, 3 (2023). https://doi.org/10.3390/s23031152
[102]
William Shakespeare. 1994. The Complete Works of William Shakespeare. Project Gutenberg, Vol.  100. Project Gutenberg, P.O. Box 2782, Champaign, IL 61825-2782, USA. https://www.gutenberg.org/ebooks/100
[103]
Adi Shamir. 1979. How to share a secret. Commun. ACM 22, 11 (1979), 612–613.
[104]
Alex Sherstinsky. 2020. Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network. Physica D: Nonlinear Phenomena 404 (March 2020), 132306. https://doi.org/10.1016/j.physd.2019.132306
[105]
Reza Shokri and Vitaly Shmatikov. 2015. Privacy-Preserving Deep Learning. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (Denver, Colorado, USA) (CCS ’15). Association for Computing Machinery, New York, NY, USA, 1310–1321. https://doi.org/10.1145/2810103.2813687
[106]
Jessie J. Smith, Lucia Jayne, and Robin Burke. 2022. Recommender Systems and Algorithmic Hate. In Proceedings of the 16th ACM Conference on Recommender Systems (Seattle, WA, USA) (RecSys ’22). Association for Computing Machinery, New York, NY, USA, 592–597. https://doi.org/10.1145/3523227.3551480
[107]
Julia Stoll. 2022. Devices used to watch online video on demand (VOD) worldwide in 1st quarter 2022 and 2nd quarter 2022. Statista. https://www.statista.com/statistics/1329449/vod-device-usage-share-worldwide/
[108]
Tao Sun, Dongsheng Li, and Bao Wang. 2022. Adaptive Random Walk Gradient Descent for Decentralized Optimization. In Proceedings of the 39th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.  162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, Baltimore, Maryland, USA, 20790–20809. https://proceedings.mlr.press/v162/sun22b.html
[109]
Zehua Sun, Yonghui Xu, Yong Liu, Wei He, Lanju Kong, Fangzhao Wu, Yali Jiang, and Lizhen Cui. 2022. A Survey on Federated Recommendation Systems. arXiv e-prints 2301.00767 (Dec. 2022). https://doi.org/10.48550/arXiv.2301.00767 arxiv:2301.00767  [cs.IR]
[110]
Jiaxi Tang and Ke Wang. 2018. Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining(Marina Del Rey, CA, USA) (WSDM ’18). Association for Computing Machinery, New York, NY, USA, 565–573. https://doi.org/10.1145/3159652.3159656
[111]
Aleksei Triastcyn, Matthias Reisser, and Christos Louizos. 2022. Decentralized Learning with Random Walks and Communication-Efficient Adaptive Optimization. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022). NeurIPS, New Orleans, LA, USA.
[112]
Aidmar Wainakh, Tim Grube, Jörg Daubert, and Max Mühlhäuser. 2019. Efficient Privacy-Preserving Recommendations Based on Social Graphs. In Proceedings of the 13th ACM Conference on Recommender Systems (Copenhagen, Denmark) (RecSys ’19). Association for Computing Machinery, New York, NY, USA, 78–86. https://doi.org/10.1145/3298689.3347013
[113]
Jianyu Wang, Rudrajit Das, Gauri Joshi, Satyen Kale, Zheng Xu, and Tong Zhang. 2022. On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data. https://doi.org/10.48550/ARXIV.2206.04723
[114]
Shuai Wang, Richard Cornelius Suwandi, and Tsung-Hui Chang. 2021. Demystifying Model Averaging for Communication-Efficient Federated Matrix Factorization. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (Toronto, Ontario, Canada). Institute of Electrical and Electronics Engineers (IEEE), 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, 3680–3684. https://doi.org/10.1109/ICASSP39728.2021.9413927
[115]
Yanmeng Wang, Qingjiang Shi, and Tsung-Hui Chang. 2023. Batch Normalization Damages Federated Learning on NON-IID Data: Analysis and Remedy. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Institute of Electrical and Electronics Engineers (IEEE), 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, 1–5. https://doi.org/10.1109/ICASSP49357.2023.10095399
[116]
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony Q. S. Quek, and H. Vincent Poor. 2020. Federated Learning With Differential Privacy: Algorithms and Performance Analysis. Trans. Info. For. Sec. 15 (Jan. 2020), 3454–3469. https://doi.org/10.1109/TIFS.2020.2988575
[117]
Wenqi Wei, Ling Liu, Margaret Loper, Ka-Ho Chow, Mehmet Emre Gursoy, Stacey Truex, and Yanzhao Wu. 2020. A Framework for Evaluating Client Privacy Leakages in Federated Learning. In Computer Security – ESORICS 2020, Liqun Chen, Ninghui Li, Kaitai Liang, and Steve Schneider (Eds.). Springer International Publishing, Cham, 545–566.
[118]
Davy Weissenbacher, Abeed Sarker, Michael J. Paul, and Graciela Gonzalez-Hernandez. 2018. Overview of the Third Social Media Mining for Health (SMM4H) Shared Tasks at EMNLP 2018. In Proceedings of the 2018 EMNLP Workshop SMM4H: The 3rd Social Media Mining for Health Applications Workshop & Shared Task. Association for Computational Linguistics, Brussels, Belgium, 13–16. https://doi.org/10.18653/v1/W18-5904
[119]
Simon Wiedemann, Heiner Kirchhoffer, Stefan Matlage, Paul Haase, Arturo Marban, Talmaj Marinč, David Neumann, Tung Nguyen, Heiko Schwarz, Thomas Wiegand, Detlev Marpe, and Wojciech Samek. 2020. DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks. IEEE Journal of Selected Topics in Signal Processing 14, 4(2020), 700–714. https://doi.org/10.1109/JSTSP.2020.2969554
[120]
Simon Wiedemann, Heiner Kirchhoffer, Stefan Matlage, Paul Haase, Arturo Marban, Talmaj Marinč, David Neumann, Tung Nguyen, Heiko Schwarz, Thomas Wiegand, Detlev Marpe, and Wojciech Samek. 2020. DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks. IEEE Journal of Selected Topics in Signal Processing 14, 4(2020), 700–714. https://doi.org/10.1109/JSTSP.2020.2969554
[121]
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. 2022. Communication-efficient federated learning via knowledge distillation. Nature Communications 13, 1 (April 2022). https://doi.org/10.1038/s41467-022-29763-x
[122]
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020. MIND: A Large-scale Dataset for News Recommendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 3597–3606. https://doi.org/10.18653/v1/2020.acl-main.331
[123]
Yao Wu, Christopher DuBois, Alice X. Zheng, and Martin Ester. 2016. Collaborative Denoising Auto-Encoders for Top-N Recommender Systems. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining (San Francisco, California, USA) (WSDM ’16). Association for Computing Machinery, New York, NY, USA, 153–162. https://doi.org/10.1145/2835776.2835837
[124]
Yuxin Wu and Kaiming He. 2020. Group Normalization. International Journal of Computer Vision 128, 3 (01 Mar 2020), 742–755. https://doi.org/10.1007/s11263-019-01198-w
[125]
Enyue Yang, Yunfeng Huang, Feng Liang, Weike Pan, and Zhong Ming. 2021. FCMF: Federated collective matrix factorization for heterogeneous collaborative filtering. Knowledge-Based Systems 220 (03 2021), 106946. https://doi.org/10.1016/j.knosys.2021.106946
[126]
Yelp. 2021. Yelp Dataset. Yelp Inc. https://www.yelp.com/dataset
[127]
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018. Graph Convolutional Neural Networks for Web-Scale Recommender Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining(London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 974–983. https://doi.org/10.1145/3219819.3219890
[128]
Kai Yue, Richeng Jin, Chau-Wai Wong, Dror Baron, and Huaiyu Dai. 2022. Gradient Obfuscation Gives a False Sense of Security in Federated Learning. arXiv e-prints abs/2206.04055 (June 2022). https://doi.org/10.48550/arXiv.2206.04055 arxiv:2206.04055  [cs.CR]
[129]
Riccardo Zaccone, Andrea Rizzardi, Debora Caldarola, Marco Ciccone, and Barbara Caputo. 2022. Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients. In 2022 26th International Conference on Pattern Recognition (ICPR) (Montréal, Québec, Canada). Institute of Electrical and Electronics Engineers (IEEE), 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, 3376–3382. https://doi.org/10.1109/ICPR56361.2022.9956084
[130]
Honglei Zhang, Fangyuan Luo, Jun Wu, Xiangnan He, and Yidong Li. 2022. LightFR: Lightweight Federated Recommendation with Privacy-Preserving Matrix Factorization. ACM Trans. Inf. Syst. 41, 2 (Dec. 2022). https://doi.org/10.1145/3578361 Just Accepted.
[131]
JianFei Zhang and YuChen Jiang. 2021. A vertical federation recommendation method based on clustering and latent factor model. In 2021 International Conference on Electronic Information Engineering and Computer Science (EIECS). Institute of Electrical and Electronics Engineers (IEEE), 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, 362–366. https://doi.org/10.1109/EIECS53707.2021.9587935
[132]
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. 2020. iDLG: Improved Deep Leakage from Gradients. arXiv e-prints abs/2001.02610 (Jan. 2020). https://doi.org/10.48550/arXiv.2001.02610 arxiv:2001.02610  [cs.LG]
[133]
Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. 2021. Federated Learning on Non-IID Data: A Survey. Neurocomput. 465, C (Nov. 2021), 371–390. https://doi.org/10.1016/j.neucom.2021.07.098
[134]
Ligeng Zhu, Zhijian Liu, and Song Han. 2019. Deep Leakage from Gradients. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol.  32. Curran Associates, Inc., Vancouver, British Columbia, Canada. https://proceedings.neurips.cc/paper/2019/file/60a6c4002cc7b29142def8871531281a-Paper.pdf

Cited By

View all
  • (2024)Neural Network Coding of Difference Updates for Efficient Distributed Learning CommunicationIEEE Transactions on Multimedia10.1109/TMM.2024.335719826(6848-6863)Online publication date: 23-Jan-2024
  • (2024)Free lunch for federated remote sensing target fine-grained classificationKnowledge-Based Systems10.1016/j.knosys.2024.111694294:COnline publication date: 17-Jul-2024
  • (2023)A Comprehensive Survey on Privacy-Preserving Techniques in Federated Recommendation SystemsApplied Sciences10.3390/app1310620113:10(6201)Online publication date: 18-May-2023

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Recommender Systems
ACM Transactions on Recommender Systems Just Accepted
EISSN:2770-6699
Table of Contents
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Online AM: 24 November 2023
Accepted: 05 November 2023
Revised: 20 August 2023
Received: 06 March 2023

Check for updates

Author Tags

  1. federated learning
  2. distributed learning
  3. federated recommender systems
  4. neural network compression

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)659
  • Downloads (Last 6 weeks)81
Reflects downloads up to 21 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Neural Network Coding of Difference Updates for Efficient Distributed Learning CommunicationIEEE Transactions on Multimedia10.1109/TMM.2024.335719826(6848-6863)Online publication date: 23-Jan-2024
  • (2024)Free lunch for federated remote sensing target fine-grained classificationKnowledge-Based Systems10.1016/j.knosys.2024.111694294:COnline publication date: 17-Jul-2024
  • (2023)A Comprehensive Survey on Privacy-Preserving Techniques in Federated Recommendation SystemsApplied Sciences10.3390/app1310620113:10(6201)Online publication date: 18-May-2023

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media