Abstract
Convolutional neural networks (CNNs) have achieved remarkable performance in diverse applications. Nevertheless, the substantial scale and computational intricacy limit the practical implementation of CNNs, particularly on resource- starved devices. This paper presents a compression technique based on graph structure learning (GSL) for CNNs. This method aims to capture the correlations among parameters in each neural network layer and compress the model by leveraging the strength of these correlations. Firstly, the neural network parameters are modeled as a graph structure by adopting a graph construction methodology. Subsequently, the graph is fed into a dual-branch GSL module. This module introduces a constraint that optimize and refine the original graph topology and maximize the difference in feature information obtained between the two channels. Through the process of graph learning, the existing correlations among the parameters of the CNNs are demonstrated. Finally, based on the correlations between the parameters of the CNNs, the parameters with lower relative importance are selected and the neural network parameters are compressed. The proposed method significantly reduces the parameter and floating-point computation complexity of CNNs, thereby diminishing the model’s intricacy. Furthermore, the operational efficiency of the network model is improved without compromising its prediction accuracy. The effectiveness of the proposed method is validated on the VGG-16 and ResNet-101 models. The compressed models’ accuracy, efficiency, and memory consumption are then compared with the original models to demonstrate the effectiveness of the method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Wang, B., et al.: SparG: a sparse GEMM accelerator for deep learning applications. In: Meng, W., Lu, R., Min, G., Vaidya, J. (eds.) Algorithms and Architectures for Parallel Processing: 22nd International Conference, ICA3PP 2022, Copenhagen, Denmark, 10–12 October 2022, Proceedings, pp. 529–547. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-22677-9_28
Aktas, K., Ignjatovic, V., Ilic, D., Marjanovic, M., Anbarjafari, G.: Deep convolutional neural networks for detection of abnormalities in chest X-rays trained on the very large dataset. Signal Image Video Process. 17(4), 1035–1041 (2023). https://doi.org/10.1007/s11760-022-02309-w
Atakishiyev, S., Salameh, M., Yao, H., Goebel, R.: Explainable artificial intelligence for autonomous driving: a comprehensive overview and field guide for future research directions. CoRR abs/2112.11561 (2021). https://arxiv.org/abs/2112.11561
LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, 27–30 November 1989], pp. 598–605. Morgan Kaufmann (1989). http://papers.nips.cc/paper/250-optimal-brain-damage
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., de Freitas, N.: Predicting parameters in deep learning. In: Burges, C.J.C., Bottou, L., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a Meeting Held 5–8 December 2013, Lake Tahoe, Nevada, United States, pp. 2148–2156 (2013). https://proceedings.neurips.cc/paper/2013/hash/7fec306d1e665bc9c748b5d2b99a6e97-Abstract.html
Lin, Y., Wang, C., Chang, C., Sun, H.: An efficient framework for counting pedestrians crossing a line using low-cost devices: the benefits of distilling the knowledge in a neural network. Multim. Tools Appl. 80(3), 4037–4051 (2021). https://doi.org/10.1007/s11042-020-09276-9
Iandola, F.N., Moskewicz, M.W., Ashraf, K., Han, S., Dally, W.J., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and \(<\)1 MB model size. CoRR abs/1602.07360 (2016). http://arxiv.org/abs/1602.07360
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 April 2017, Conference Track Proceedings. OpenReview.net (2017). https://openreview.net/forum?id=rJqFGTslg
Zhang, L., Wei, W., Shi, Q., Shen, C., van den Hengel, A., Zhang, Y.: Accurate tensor completion via adaptive low-rank representation. IEEE Trans. Neural Networks Learn. Syst. 31(10), 4170–4184 (2020). https://doi.org/10.1109/TNNLS.2019.2952427
Kang, H.: Accelerator-aware pruning for convolutional neural networks. IEEE Trans. Circuits Syst. Video Technol. 30(7), 2093–2103 (2020). https://doi.org/10.1109/TCSVT.2019.2911674
Shen, W., Wang, W., Zhu, J., Zhou, H., Wang, S.: Pruning-and quantization-based compression algorithm for number of mixed signals identification network. Electronics 12(7), 1694 (2023)
Yuan, C., Agaian, S.S.: A comprehensive review of binary neural network. CoRR abs/2110.06804 (2021). https://arxiv.org/abs/2110.06804
Zhao, R., et al.: Accelerating binarized convolutional neural networks with software-programmable FPGAs. In: Greene, J.W., Anderson, J.H. (eds.) Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, FPGA 2017, Monterey, CA, USA, 22–24 February 2017, pp. 15–24. ACM (2017). http://dl.acm.org/citation.cfm?id=3021741
Li, E., Zeng, L., Zhou, Z., Chen, X.: Edge AI: on-demand accelerating deep neural network inference via edge computing. IEEE Trans. Wirel. Commun. 19(1), 447–457 (2020). https://doi.org/10.1109/TWC.2019.2946140
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014). https://doi.org/10.5555/2627435.2670313
Han, S., Pool, J., Tran, J., Dally, W.J.: Learning both weights and connections for efficient neural network. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015 December, pp. 7–12, 2015, Montreal, Quebec, Canada, pp. 1135–1143 (2015). https://proceedings.neurips.cc/paper/2015/hash/ae0eb3eed39d2bcef4622b2499a05fe6-Abstract.html
Srinivas, S., Babu, R.V.: Data-free parameter pruning for deep neural networks. In: Xie, X., Jones, M.W., Tam, G.K.L. (eds.) Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK, 7–10 September 2015, pp. 31.1–31.12. BMVA Press (2015). https://doi.org/10.5244/C.29.31
Chen, S., Zhao, Q.: Shallowing deep networks: layer-wise pruning based on feature representations. IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 3048–3056 (2019). https://doi.org/10.1109/TPAMI.2018.2874634
Fiesler, E., Choudry, A., Caulfield, H.J.: Weight discretization paradigm for optical neural networks. In: Optical Interconnections and Networks, vol. 1281, pp. 164–173. SPIE (1990)
Balzer, W., Takahashi, M., Ohta, J., Kyuma, K.: Weight quantization in Boltzmann machines. Neural Netw. 4(3), 405–409 (1991). https://doi.org/10.1016/0893-6080(91)90077-I
Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural message passing for quantum chemistry. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6–11 August 2017. Proceedings of Machine Learning Research, vol. 70, pp. 1263–1272. PMLR (2017)
Dai, H., et al.: Adversarial attack on graph structured data. In: Dy, J.G., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, 10–15 July 2018. Proceedings of Machine Learning Research, vol. 80, pp. 1123–1132. PMLR (2018). http://proceedings.mlr.press/v80/dai18b.html
Zhu, D., Zhang, Z., Cui, P., Zhu, W.: Robust graph convolutional networks against adversarial attacks. In: Teredesai, A., Kumar, V., Li, Y., Rosales, R., Terzi, E., Karypis, G. (eds.) Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, 4–8 August 2019, pp. 1399–1407. ACM (2019). https://doi.org/10.1145/3292500.3330851
Luo, D., et al.: Learning to drop: robust graph neural network via topological denoising. In: Lewin-Eytan, L., Carmel, D., Yom-Tov, E., Agichtein, E., Gabrilovich, E. (eds.) WSDM 2021, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, 8–12 March 2021, pp. 779–787. ACM (2021). https://doi.org/10.1145/3437963.3441734
Newman, M.: Networks. Oxford University Press, Oxford (2018)
Li, R., Wang, S., Zhu, F., Huang, J.: Adaptive graph convolutional neural networks. In: McIlraith, S.A., Weinberger, K.Q. (eds.) Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-2018), the 30th innovative Applications of Artificial Intelligence (IAAI-2018), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-2018), New Orleans, Louisiana, USA, 2–7 February 2018, pp. 3546–3553. AAAI Press (2018). https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16642
Preparata, F.P., Shamos, M.I.: Computational Geometry - An Introduction. Texts and Monographs in Computer Science, Springer, Cham (1985). https://doi.org/10.1007/978-1-4612-1098-6
Song, L., Smola, A.J., Gretton, A., Borgwardt, K.M., Bedo, J.: Supervised feature selection via dependence estimation. In: Ghahramani, Z. (ed.) Machine Learning, Proceedings of the Twenty-Fourth International Conference (ICML 2007), Corvallis, Oregon, USA, 20–24 June 2007. ACM International Conference Proceeding Series, vol. 227, pp. 823–830. ACM (2007). https://doi.org/10.1145/1273496.1273600
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 770–778. IEEE Computer Society (2016). https://doi.org/10.1109/CVPR.2016.90
Acknowledgements
This work is supported by the Natural Science Foundation of Shandong Province China (NO. ZR2020LZH008, ZR2021MF118, ZR2022LZH003), the Key R &D Program of Shandong Province, China (NO. 2021CXGC010506, NO. 2021SFGC0104) and the National Natural Science Foundation of China (NO. 62101311).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, T., Zheng, X., Zhang, L., Zhang, Y. (2024). Graph Structure Learning-Based Compression Method for Convolutional Neural Networks. In: Tari, Z., Li, K., Wu, H. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2023. Lecture Notes in Computer Science, vol 14488. Springer, Singapore. https://doi.org/10.1007/978-981-97-0801-7_8
Download citation
DOI: https://doi.org/10.1007/978-981-97-0801-7_8
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-0800-0
Online ISBN: 978-981-97-0801-7
eBook Packages: Computer ScienceComputer Science (R0)