Abstract
In Federated learning (FL), FederatedAveraging(FedAvg) is widely used to compute the weighted mean of local models in the parametric space over time on a central server by exchanging intermediate updates over multiple rounds of communication. However, this approach requires many communication rounds for the central model to learn data generalization. Each local model updates the central model parameters in different directions of the multi-dimensional parameter space. To address this challenge, we propose FedGMMinit: Federated Initialization with Gaussian Mixture Model, which adjusts initial central model gradients by pre-training the model on synthetic data generated from a Gaussian Mixture Model (GMM). For each label in the client’s dataset, a GMM is built. The pre-trained weights are then communicated to the selected clients to initialize FedAvg. To maintain data privacy, only the client’s representation of the Gaussian is passed to the server. Our proposed approach is tested on MNIST digit datasets for image classification. It shows a reduction of 10–15 communication rounds required by the central model to achieve target accuracy for both IID and non-IID distributions. In the scope of the study, we also discovered that clustering clients and training them with global models also contributed to the overall improvement of convergence. We call this clustering method as FedGMMCluster.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., Gao, Y.: A survey on federated learning. Knowl.-Based Syst. 216, 106775 (2021)
McMahan, B., Moore, E., Ramage, D., Hampson, S., Y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)
Kairouz, P., et al.: Advances and open problems in federated learning. Found. Trends® Mach. Learn. 14(1–2), 1–210 (2021)
Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. Proc. Mach. Learn. Syst. 2, 429–450 (2020)
Chai, B., Liu, K., Yang, R.: Cross-domain federated data modeling on non-IID data. Comput. Intell. Neurosci. CIN 2022 (2022)
Shin, M., Hwang, C., Kim, J., Park, M., Bennis, J., Kim, S.L.: XOR Mixup: privacy-preserving data augmentation for one-shot federated learning. arXiv preprint arXiv:2006.05148 (2020)
Weituo, H., et al.: Towards fair federated learning with zero-shot data augmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3310–3319 (2021)
Briggs, C., Fan, Z., Andras, P.: Federated learning with hierarchical clustering of local updates to improve training on non-IID data. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–9. IEEE (2020)
Guo, C., Fu, H., Luk, W.: A fully-pipelined expectation-maximization engine for gaussian mixture models. In: 2012 International Conference on Field-Programmable Technology, pp. 182–189. IEEE (2012)
Zhu, Z., Hong, J., Zhou, J.: Data-free knowledge distillation for heterogeneous federated learning. In: International Conference on Machine Learning, pp. 12878–12889. PMLR (2021)
Jie, Z., et al.: A practical data-free approach to one-shot federated learning with heterogeneity. arXiv preprint arXiv:2112.12371 (2021)
Chen, H., Tu, C., Li, Z., Shen, H.W., Chao, W.L.: On the importance and applicability of pre-training for federated learning. In: The Eleventh International Conference on Learning Representations (2023)
Nguyen, J., Wang, J., Malik, K., Sanjabi, M., Rabbat, M.: Where to begin? On the impact of pre-training and initialization in federated learning. arXiv preprint arXiv:2210.08090 (2022)
Acknowledgement
The authors acknowledge computational and funding support from the project numbered CSE2122001FACEKALI and titled Design and Development of Disaster Response Dashboard for India for carrying out the work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Palit, A., Nandanavanam, S.P., Yeturu, K. (2024). Swift Convergence: Federated Learning Enhanced with GMMs for Image Classification. In: Kaur, H., Jakhetiya, V., Goyal, P., Khanna, P., Raman, B., Kumar, S. (eds) Computer Vision and Image Processing. CVIP 2023. Communications in Computer and Information Science, vol 2009. Springer, Cham. https://doi.org/10.1007/978-3-031-58181-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-58181-6_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-58180-9
Online ISBN: 978-3-031-58181-6
eBook Packages: Computer ScienceComputer Science (R0)