A Survey of Model Compression and Its Feedback Mechanism in Federated Learning
Pages 37 - 42
Abstract
In this paper, we review various model compression methods used in extensive neural networks, such as Quantization, Pruning, Knowledge Distillation, and Weight Sharing. We also focus on their implementation in federated learning environments. Especially, we delve into the feedback model compression mechanism in federated learning. This survey provides valuable insights into the potential advantages and challenges of this approach. Furthermore, the paper presents forward-looking perspectives, charting potential future developments in this dynamic field. It serves as a guide for researchers and practitioners aiming to refine model compression strategies in federated learning, contributing to the growth and practicality of this field.
References
[1]
Muntadher Qasim Abdulhasan, Mustafa Ismael Salman, Chee Kyun Ng, Nor Kamariah Noordin, Shaiful Jahari Hashim, and Fazirulhisham Hashim. 2015. An adaptive threshold feedback compression scheme based on channel quality indicator (CQI) in long term evolution (LTE) system. Wireless Personal Communications 82 (2015), 2323–2349.
[2]
Nima Aghli and Eraldo Ribeiro. 2021. Combining Weight Pruning and Knowledge Distillation For CNN Compression. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2021), 3185–3192.
[3]
Alyazeed Albasyoni, Mher Safaryan, Laurent Condat, and Peter Richtárik. 2020. Optimal gradient compression for distributed and federated learning. arXiv preprint arXiv:2010.03246 (2020).
[4]
Anthony Berthelier, Thierry Chateau, Stefan Duffner, Christophe Garcia, and Christophe Blanc. 2020. Deep Model Compression and Architecture Optimization for Embedded Systems: A Survey. Journal of Signal Processing Systems 93 (2020), 863 – 878.
[5]
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. ZeroQ: A Novel Zero Shot Quantization Framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 13169–13178.
[6]
Song Cheng, Zixuan Li, Yongsen Wang, Wanbing Zou, Yumei Zhou, Delong Shang, and Shushan Qiao. 2021. Gradient Corrected Approximation for Binary Neural Networks. IEICE TRANSACTIONS on Information and Systems 104, 10 (2021), 1784–1788.
[7]
François Chollet. 2016. Xception: Deep Learning with Depthwise Separable Convolutions. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 1800–1807.
[8]
Wesley Cooke, Zihao Mo, and Weiming Xiang. 2023. Guaranteed Quantization Error Computation for Neural Network Model Compression. 2023 IEEE International Conference on Industrial Technology (ICIT) (2023), 1–4.
[9]
Greg Diamos, Shubho Sengupta, Bryan Catanzaro, Mike Chrzanowski, Adam Coates, Erich Elsen, Jesse Engel, Awni Hannun, and Sanjeev Satheesh. 2016. Persistent rnns: Stashing recurrent weights on-chip. In International Conference on Machine Learning. PMLR, 2024–2033.
[10]
Shiming Ge, Zhao Luo, Shengwei Zhao, Xin Jin, and Xiao-Yu Zhang. 2017. Compressing deep neural networks for efficient visual inference. In 2017 IEEE International Conference on Multimedia and Expo (ICME). 667–672. https://doi.org/10.1109/ICME.2017.8019465
[11]
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision 129 (2021), 1789–1819.
[12]
Xiaotian Han, Tong Zhao, Yozen Liu, Xia Hu, and Neil Shah. 2022. Mlpinit: Embarrassingly simple gnn training acceleration with mlp initialization. arXiv preprint arXiv:2210.00102 (2022).
[13]
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).
[14]
Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, and Yue Liu. 2022. Fedsynth: Gradient compression via synthetic data in federated learning. arXiv preprint arXiv:2204.01273 (2022).
[15]
Berivan Isik, Albert No, and Tsachy Weissman. 2021. Rate-Distortion Theoretic Model Compression: Successive Refinement for Pruning.
[16]
Qinjun Jiang and Matthew D. Sinclair. 2021. Reducing Synchronization Overhead for Persistent RNNs.
[17]
Rui-Yang Ju, Ting-Yu Lin, Jia-Hao Jian, and Jen-Shiun Chiang. 2023. Efficient convolutional neural networks on Raspberry Pi for image classification. Journal of Real-Time Image Processing 20, 2 (2023), 21.
[18]
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, 2021. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning 14, 1–2 (2021), 1–210.
[19]
Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. 2019. Error feedback fixes signsgd and other gradient compression schemes. (2019), 3252–3261.
[20]
Sourabh Katoch, Sumit Singh Chauhan, and Vijay Kumar. 2020. A review on genetic algorithm: past, present, and future. Multimedia Tools and Applications 80 (2020), 8091 – 8126.
[21]
Petros Katsileros, Nikiforos Mandilaras, Dimitrios Mallis, Vassilis Pitsikalis, Stavros Theodorakis, and Gil Chamiel. 2022. An Incremental Learning framework for Large-scale CTR Prediction. (2022), 490–493.
[22]
Duy-Dong Le, Anh-Khoa Tran, Minh-Son Dao, Kieu-Chinh Nguyen-Ly, Hoang-Son Le, Xuan-Dao Nguyen-Thi, Thanh-Qui Pham, Van-Luong Nguyen, and Bach-Yen Nguyen-Thi. 2022. Insights into multi-model federated learning: An advanced approach for air quality index forecasting. Algorithms 15, 11 (2022), 434.
[23]
Zhuo Li, Hengyi Li, and Lin Meng. 2023. Model Compression for Deep Neural Networks: A Survey. Comput. 12 (2023), 60.
[24]
Kai Liang, Huiru Zhong, Haoning Chen, and Youlong Wu. 2021. Wyner-Ziv gradient compression for federated learning. arXiv preprint arXiv:2111.08277 (2021).
[25]
Yuang Liu, Wei Zhang, and Jun Wang. 2021. Zero-shot Adversarial Quantization. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021), 1512–1521.
[26]
Gangzhao Lu, Weizhe Zhang, and Zheng Wang. 2021. Optimizing depthwise separable convolution operations on gpus. IEEE Transactions on Parallel and Distributed Systems 33, 1 (2021), 70–87.
[27]
Yuanhua Lv and ChengXiang Zhai. 2014. Revisiting the Divergence Minimization Feedback Model. Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (2014).
[28]
Xiaojun Ma, Qin Chen, Yuanyi Ren, Guojie Song, and Liang Wang. 2022. Meta-weight graph neural network: Push the limits beyond global homophily. (2022), 1270–1280.
[29]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 1273–1282.
[30]
Luke Melas-Kyriazi and Franklyn Wang. 2021. Intrinisic Gradient Compression for Federated Learning. arXiv preprint arXiv:2112.02656 (2021).
[31]
Georgii Sergeevich Novikov, Daniel Bershatsky, Julia Gusak, Alex Shonenkov, Denis Valerievich Dimitrov, and Ivan Oseledets. 2023. Few-bit backward: Quantized gradients of activation functions for memory footprint reduction. (2023), 26363–26381.
[32]
Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model compression via distillation and quantization. ArXiv abs/1802.05668 (2018).
[33]
Ofir Press and Lior Wolf. 2016. Using the Output Embedding to Improve Language Models. In Conference of the European Chapter of the Association for Computational Linguistics.
[34]
Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletarì, Holger R Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, 2020. The future of digital health with federated learning. NPJ Digital Medicine, 3, 119. (2020).
[35]
Mohammed Saeed and Paolo Papotti. 2022. You Are My Type! Type Embeddings for Pre-trained Language Models. In Conference on Empirical Methods in Natural Language Processing.
[36]
Suhail Mohmad Shah and Vincent KN Lau. 2021. Model compression for communication efficient federated learning. IEEE Transactions on Neural Networks and Learning Systems (2021).
[37]
Sangeetha Siddegowda, Marios Fournarakis, Markus Nagel, Tijmen Blankevoort, Chirag Patel, and Abhijit Khobare. 2022. Neural network quantization with ai model efficiency toolkit (aimet). arXiv preprint arXiv:2201.08442 (2022).
[38]
Suraj Srinivas, Andrey Kuzmin, Markus Nagel, Mart van Baalen, Andrii Skliar, and Tijmen Blankevoort. 2022. Cyclical pruning for sparse neural networks. (2022), 2762–2771.
[39]
Sebastian U Stich and Sai Praneeth Karimireddy. 2020. The error-feedback framework: Better rates for sgd with delayed gradients and compressed updates. The Journal of Machine Learning Research 21, 1 (2020), 9613–9648.
[40]
Ye Tian, Liguo Zhang, Jianguo Sun, Guisheng Yin, and Yuxin Dong. 2022. Consistency regularization teacher–student semi-supervised learning method for target recognition in SAR images. The Visual Computer 38, 12 (2022), 4179–4192.
[41]
Sunil Vadera and Salem Ameen. 2022. Methods for Pruning Deep Neural Networks. IEEE Access 10 (2022), 63280–63300. https://doi.org/10.1109/ACCESS.2022.3182659
[42]
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. (2022), 23965–23998.
[43]
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. 2015. Quantized Convolutional Neural Networks for Mobile Devices. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015), 4820–4828.
[44]
Xiang Wu, Ran He, Yibo Hu, and Zhenan Sun. 2020. Learning an evolutionary embedding via massive knowledge distillation. International Journal of Computer Vision 128 (2020), 2089–2106.
[45]
Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Lanfei Wang, Zhengsu Chen, An Xiao, Jianlong Chang, Xiaopeng Zhang, 2021. Weight-sharing neural architecture search: A battle to shrink the optimization gap. ACM Computing Surveys (CSUR) 54, 9 (2021), 1–37.
[46]
Ye Xue, Liqun Su, and Vincent KN Lau. 2022. FedOComp: Two-timescale online gradient compression for over-the-air federated learning. IEEE Internet of Things Journal 9, 19 (2022), 19330–19345.
[47]
Nakyeong Yang, Yunah Jang, Hwanhee Lee, Seohyeong Jeong, and Kyomin Jung. 2023. Task-specific Compression for Multi-task Language Models using Attribution-based Pruning. In Findings of the Association for Computational Linguistics: EACL 2023. 582–592.
[48]
TJ Yang, Y Xiao, G Motta, F Beaufays, R Mathews, and M Chen. 2022. Online Model Compression for Federated Learning with Large Models. ArXiv abs/2205.03494 (2022).
[49]
Mengyang Yuan, Bo Lang, and Fengnan Quan. 2023. Student-friendly Knowledge Distillation. ArXiv abs/2305.10893 (2023).
[50]
Mingyang Zhang, Xinyi Yu, Jingtao Rong, and Linlin Ou. 2022. Graph pruning for model compression. Applied Intelligence 52, 10 (2022), 11244–11256.
[51]
Tunhou Zhang, Dehua Cheng, Yuchen He, Zhengxing Chen, Xiaoliang Dai, Liang Xiong, Feng Yan, Hai Li, Yiran Chen, and Wei Wen. 2023. NASRec: weight sharing neural architecture search for recommender systems. (2023), 1199–1207.
[52]
Qi Zhao, Shuchang Lyu, Lijiang Chen, Binghao Liu, Ting-Bing Xu, Guangliang Cheng, and Wenquan Feng. 2023. Learn by Oneself: Exploiting Weight-Sharing Potential in Knowledge Distillation Guided Ensemble Network. IEEE Transactions on Circuits and Systems for Video Technology (2023).
[53]
Kai Zhen, Hieu Duy Nguyen, Raviteja Chinta, Nathan Susanj, Athanasios Mouchtaris, Tariq Afzal, and Ariya Rastrow. 2022. Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognition. In Interspeech.
[54]
Qinghe Zheng, Xinyu Tian, Mingqiang Yang, Yulin Wu, and Huake Su. 2019. PAC-Bayesian framework based drop-path method for 2D discriminative convolutional network pruning. Multidimensional Systems and Signal Processing 31 (2019), 793 – 827.
[55]
Michael Zhu and Suyog Gupta. 2017. To prune, or not to prune: exploring the efficacy of pruning for model compression. ArXiv abs/1710.01878 (2017).
Index Terms
- A Survey of Model Compression and Its Feedback Mechanism in Federated Learning
Recommendations
Multimodal federated learning: Concept, methods, applications and future directions
AbstractMultimodal learning mines and analyzes multimodal data in reality to better understand and appreciate the world around people. However, how to exploit this rich multimodal data without violating user privacy is a key issue. Federated learning is ...
Highlights- The three different modes in the multimodal federated learning model are summarized.
- Multimodal fusion based on the federated learning framework is also specified.
- The difficulties and some ideas of multimodal federated learning ...
Model compression and privacy preserving framework for federated learning
AbstractFederated learning (FL) as a collaborative learning paradigm has attracted extensive attention due to its characteristic of privacy preserving, in which the clients train a shared neural network model collaboratively by their local ...
Highlights- A novel model compression scheme for FL framework is constructed.
- The scheme ...
Recent advances on federated learning: A systematic survey
AbstractFederated learning has emerged as an effective paradigm to achieve privacy-preserving collaborative learning among different parties. Compared to traditional centralized learning that requires collecting data from each party, in federated ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2024
48 pages
ISBN:9798400705496
DOI:10.1145/3643488
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICMR '24
Sponsor:
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 109Total Downloads
- Downloads (Last 12 months)109
- Downloads (Last 6 weeks)22
Reflects downloads up to 24 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format