research-article

FP-AGL: Filter Pruning With Adaptive Gradient Learning for Accelerating Deep Convolutional Neural Networks

Authors:

Hyun KimAuthors Info & Claims

IEEE Transactions on Multimedia, Volume 25

Pages 5279 - 5290

https://doi.org/10.1109/TMM.2022.3189496

Published: 11 July 2022 Publication History

Abstract

Filter pruning is a technique that reduces computational complexity, inference time, and memory footprint by removing unnecessary filters in convolutional neural networks (CNNs) with an acceptable drop in accuracy, consequently accelerating the network. Unlike traditional filter pruning methods utilizing zeroing-out filters, we propose two techniques to achieve the effect of pruning more filters with less performance degradation, inspired by the existing research on centripetal stochastic gradient descent (C-SGD), wherein the filters are removed only when the ones that need to be pruned have the same value. First, to minimize the negative effect of centripetal vectors that gradually make filters come closer to each other, we redesign the vectors by considering the effect of each vector on the loss-function using the Taylor-based method. Second, we propose an adaptive gradient learning (AGL) technique that updates weights while adaptively changing the gradients. Through AGL, performance degradation can be mitigated because some gradients maintain their original direction, and AGL also minimizes the accuracy loss by perfectly converging the filters, which require pruning, to a single point. Finally, we demonstrate the superiority of the proposed method on various datasets and networks. In particular, on the ILSVRC-2012 dataset, our method removed 52.09% FLOPs with a negligible 0.15% top-1 accuracy drop on ResNet-50. As a result, we achieve the most outstanding performance compared to those reported in previous studies in terms of the trade-off between accuracy and computational complexity.

References

[1]

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770–778.

[2]

A. G. Howard et al., “MobileNets: Efficient convolutional neural networks for mobile vision applications,” 2017, arXiv:1704.04861.

[3]

G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 4700–4708.

[4]

X. Zhang, X. Zhou, M. Lin, and J. Sun, “ShuffleNet: An extremely efficient convolutional neural network for mobile devices,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 6848–6856.

[5]

J. Choi, D. Chun, H. Kim, and H.-J. Lee, “Gaussian YOLOv3: An accurate and fast object detector using localization uncertainty for autonomous driving,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 502–511.

[6]

J. Choi, D. Chun, H.-J. Lee, and H. Kim, “Uncertainty-Based object detector for autonomous driving embedded platforms,” in Proc. 2nd IEEE Int. Conf. Artif. Intell. Circuits Syst., 2020, pp. 16–20.

[7]

J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” 2018, arXiv:1804.02767.

[8]

M. Tan, R. Pang, and Q. V. Le, “EfficientDet: Scalable and efficient object detection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 10781–10790.

[9]

D. Bolya, C. Zhou, F. Xiao, and Y. Lee, “YOLACT: Real-time instance segmentation,” in Proc. IEEE Int. Conf. Comput. Vis., 2019, pp. 9157–9166.

[10]

K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2961–2969.

[11]

D. T. Nguyen, H. Kim, H.-J. Lee, and I.-J. Chang, “An approximate memory architecture for a reduction of refresh power consumption in deep learning applications,” in Proc. IEEE Int. Symp. Circuits Syst., 2018, pp. 1–5.

[12]

D. T. Nguyen, T. N. Nguyen, H. Kim, and H.-J. Lee, “A high-throughput and power-efficient FPGA implementation of YOLO CNN for object detection,” IEEE Trans. Very Large Scale Integration Syst., vol. 27, no. 8, pp. 1861–1873, Aug. 2019.

Digital Library

[13]

Y. Xu, W. Dai, Y. Qi, J. Zou, and H. Xiong, “Iterative deep neural network quantization with Lipschitz constraint,” IEEE Trans. Multimedia, vol. 22, no. 7, pp. 1874–1888, Jul. 2020.

[14]

Z. Wang, W. Hong, Y. Tan, and J. Yuan, “Pruning 3D filters for accelerating 3D ConvNets,” IEEE Trans. Multimedia, vol. 22, no. 8, pp. 2126–2137, Aug. 2020.

[15]

S. Kim and H. Kim, “Zero-Centered fixed-point quantization with iterative retraining for deep convolutional neural network-based object detectors,” IEEE Access, vol. 9, pp. 20828–20839, 2021.

[16]

B. Hassibi, D. G. Stork, and G. Wolf, “Optimal brain surgeon: Extensions and performance comparisons,” in Proc. Adv. Neural Inf. Process. Syst., 1993, pp. 263–270.

[17]

J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” in Proc. 7th Int. Conf. Learn. Representations, 2019.

[18]

S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” in Proc. 4th Int. Conf. Learn. Representations, 2016.

[19]

X. Ding et al., “Global sparse momentum SGD for pruning very deep neural networks,” in Proc. Adv. Neural Inf. Process. Syst., 2019, pp. 6379–6391.

[20]

S. Han et al., “DSD: Dense-sparse-dense training for deep neural networks,” in Proc. 5th Int. Conf. Learn. Representations, 2017.

[21]

A. Morcos, H. Yu, M. Paganini, and Y. Tian, “One ticket to win them all: Generalizing lottery ticket initializations across datasets and optimizers,” in Proc. Adv. Neural Inf. Process. Syst., 2019, pp. 4933–4943.

[22]

H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” in Proc. 5th Int. Conf. Learn. Representations, 2017.

[23]

Z. Liu et al., “Learning efficient convolution networks through network slimming,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2736–2744.

[24]

Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang, “Soft filter pruning for accelerating deep convolutional neural networks,” in Proc. 27th Int. Joint Conf. Artif. Intell., 2018, pp. 2234–2240.

[25]

Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang, “Filter pruning via geometric median for deep convolutional neural networks acceleration,” in Proc. IEEE/CVF Conf. comput. Vis. Pattern Recognit., 2019, pp. 4340–4349.

[26]

J. Ye, X. Lu, Z. Lin, and J. Z. Wang, “Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers,” in Proc. 6th Int. Conf. Learn. Representations, 2018.

[27]

X. Gao, Y. Zhao, Ł. Dudziak, R. Mullins, and C. Xu, “Dynamic channel pruning: Feature boosting and suppression,” in Proc. 7th Int. Conf. Learn. Representations, 2019.

[28]

X. Ding, G. Ding, Y. Guo, and J. Han, “Centripetal SGD for pruning very deep convolutional networks with complicated structure,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 4943–4953.

[29]

P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz, “Importance estimation for neural network pruning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 11264–11272.

[30]

C. Zhao et al., “Variational convolutional neural network pruning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 2780–2789.

[31]

I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” J. Mach. Learn. Res., vol. 23, no. 3, pp. 1139–1147, 2013.

[32]

S. Gao, F. Huang, J. Pei, and H. Huang, “Discrete model compression with resource constraint for deep neural networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 1899–1908.

[33]

M. Lin et al., “HRank: Filter pruning using high-rank feature map,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 1529–1538.

[34]

T.-W. Chin, R. Ding, C. Zhang, and D. Marculescu, “Towards efficient model compression via learned global ranking,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 1518–1528.

[35]

M. Lin et al., “Channel pruning via automatic structure search,” in Proc. 29th Int. Joint Conf. Artif. Intell., 2020, pp. 673–679.

[36]

Y. He et al., “Learning filter pruning criteria for deep convolutional neural networks acceleration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 2009–2018.

[37]

Y. He et al., “AMC: AutoML for model compression and acceleration on mobile devices,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 784–800.

[38]

M. Kang and B. Han, “Operation-Aware soft channel pruning using differentiable masks,” in Proc. Int. Conf. Mach. Learn., 2020, pp. 5122–5131.

[39]

N. Kim and H. Kim, “Mask-Soft filter pruning for lightweight CNN inference,” in Proc. IEEE 17th Int. SoC Des. Conf., 2020, pp. 316–317.

[40]

Z. Liu et al., “MetaPruning: Meta learning for automatic neural network channel pruning,” in Proc. IEEE/CVF Conf. comput. Vis. Pattern Recognit., 2019, pp. 3296–3305.

[41]

X. Ding et al., “Lossless CNN channel pruning via gradient resetting and convolutional Re-parameterization,” in Proc. IEEE Int. Conf. Comput. Vis., 2021, pp. 4510–4520.

[42]

J. Shi, J. Xu, K. Tasaka, and Z. Chen, “SASL: Saliency-adaptive sparsity learning for neural network acceleration,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 5, pp. 2008–2019, May 2021.

Digital Library

[43]

X. Ding, G. Ding, Y. Guo, J. Han, and C. Yan, “Approximated oracle filter pruning for destructive CNN width optimization,” in Proc. Int. Conf. Mach. Learn., 2019, pp. 1607–1616.

[44]

S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. Int. Conf. Mach. Learn., pp. 448–456, 2015.

[45]

Y. L. Cun, J. S. Denker, and S. A. Solla, “Optimal brain damage,” in Proc. Adv. Neural Inf. Process. Syst., 1989, pp. 598–605.

[46]

O. Russakovsky et al., “ImageNet large scale visual recognition challenge,” Int. J. Comput. Vis., vol. 115, no. 3, pp. 211–252, 2015.

Digital Library

[47]

Z. You, K. Yan, J. Ye, M. Ma, and P. Wang, “Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks,” in Proc. Adv. Neural Inf. Process. Syst., 2019, pp. 2130–2141.

[48]

A. Krizhevsky, and G. Hinton, “Learning multiple layers of features from tiny images,” Tech. Rep., Univ. Toronto, Toronto, ON, Canada, 2009.

[49]

C.-Y. Wang et al., “CSPNet: A new backbone that can enhance learning capability of CNN,” in Proc. IEEE Conf. comput. Vis. Pattern Recognit. Workshops, 2020, pp. 1571–1580.

[50]

H. Hu, R. Peng, Y. W. Tai, and C. K. Tang, “Network trimming: A data-driven neuron pruning approach towards efficient deep architectures,” 2016, arXiv:1607.03250.

[51]

P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient inference,” in Proc. 5th Int. Conf. Learn. Representations, 2017.

[52]

D. Misra, “Mish: A self regularized non-monotonic activation function,” in Proc. 31st British Mach. Vis. Conf., 2020.

[53]

M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The PASCAL visual object classes challenge results,” Int. J. Comput. Vis., vol. 88, no. 2, pp. 303–338, Jun. 2007.

[54]

H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han, “Once-for-all: Train one network and specialize it for efficient deployment,” in Proc. 8th Int. Conf. Learn. Representations, 2020.

[55]

J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. IEEE Conf. comput. Vis. Pattern Recognit., 2015, pp. 3431–3440.

[56]

X. Ding, Y. Guo, G. Ding, and J. Han, “ACNet: Strengthening the kernel skeletons for powerful CNN via asymmetric convolution blocks,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 1911–1920.

[57]

X. Ding, G. Ding, J. Han, and S. Tang, “Auto-balanced filter pruning for efficient convolutional neural networks,” in Proc. AAAI Conf. Artif. Intell., 2018, vol. 32, pp. 6797–6804.

[58]

G. Ding, S. Zhang, Z. Jia, J. Zhong, and J. Han, “Where to prune: Using LSTM to guide data-dependent soft pruning,” IEEE Trans. Image Process., vol. 30, pp. 293–304, 2021.

Digital Library

[59]

Y. Li, S. Gu, C. Mayer, L. V. Gool, and R. Timofte, “Group sparsity: The hinge between filter pruning and decomposition for network compression,” in Proc. IEEE Conf. comput. Vis. Pattern Recognit., 2020, pp. 8018–8027.

[60]

Y. Aflalo, A. Noy, M. Lin, I. Friedman, and L. Zelnik, “Knapsack pruning with inner distillation,” 2020, arXiv:2002.08258.

[61]

W. Ahmed, A. Zunino, P. Morerio, and V. Murino, “Compact cnn structure learning by knowledge distillation,” in Proc. IEEE Int. Conf. Pattern Recognit., 2021, pp. 6554–6561.

[62]

X. Chen, J. Zhu, J. Jiang, and C.-Y. Tsui, “Tight compression: Compressing CNN through fine-grained pruning and weight permutation for efficient implementation,” IEEE Trans. Comput.-Aided Design Integrated Circuits Syst., to be published.

[63]

Y. Li et al., “Exploiting kernel sparsity and entropy for interpretable CNN compression,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 2800–2809.

[64]

K. Han et al., “GhostNet: More features from cheap operations,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 1580–1589.

[65]

B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, and J. Malik, “Semantic contours from inverse detectors,” in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 991–998.

[66]

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. -C. Chen, “MobileNetV2: Inverted residuals and linear bottlenecks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 4510–4520.

Cited By

Jang JLee HKim HDev KYoo JMeinerzhagen P(2024)EDeN: Enabling Low-Power CNN Inference on Edge Devices Using Prefetcher-assisted NVM SystemsProceedings of the 29th ACM/IEEE International Symposium on Low Power Electronics and Design10.1145/3665314.3670801(1-6)Online publication date: 5-Aug-2024
https://dl.acm.org/doi/10.1145/3665314.3670801
Li XJiao LSun QLiu FLiu XLi LChen PYang S(2024)A Category-Aware Curriculum Learning for Data-Free Knowledge DistillationIEEE Transactions on Multimedia10.1109/TMM.2024.339584426(9603-9618)Online publication date: 2-May-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3395844
Hong HChoi DKim NLee HKang BKang HKim H(2024)Survey of convolutional neural network accelerators on field-programmable gate array platforms: architectures and optimization techniquesJournal of Real-Time Image Processing10.1007/s11554-024-01442-821:3Online publication date: 29-Mar-2024
https://dl.acm.org/doi/10.1007/s11554-024-01442-8
Show More Cited By

Index Terms

FP-AGL: Filter Pruning With Adaptive Gradient Learning for Accelerating Deep Convolutional Neural Networks
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Theory of computation
  1. Theory and algorithms for application domains

Index terms have been assigned to the content through auto-classification.

Recommendations

Pruning Deep Convolutional Neural Networks via Gradient Support Pursuit
Pattern Recognition and Computer Vision
Abstract
In this paper, we propose a filter pruning method, namely, Filter Pruning via Gradient Support Pursuit (FPGraSP), which can accelerate and compress very deep Convolutional Neural Networks effectively in an iterative way. Previous work reports that ...
Pruning convolutional neural networks via filter similarity analysis
Abstract
Deep learning has shown excellent performance in many fields, especially image recognition and retrieval in recent years. The performance of convolutional neural networks (CNNs) is particularly outstanding. CNNs, however, are usually ...
Filter Pruning via Probabilistic Model-based Optimization for Accelerating Deep Convolutional Neural Networks
WSDM '21: Proceedings of the 14th ACM International Conference on Web Search and Data Mining

Accelerating Deep Convolutional Neural Networks(CNNs) has recently received ever-increasing research focus. Among various approaches proposed in the literature, filter pruning has been regarded as a promising solution, which is due to its advantage in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Multimedia

IEEE Transactions on Multimedia Volume 25, Issue

2023

8932 pages

ISSN:1520-9210

Issue’s Table of Contents

1520-9210 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 11 July 2022

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jang JLee HKim HDev KYoo JMeinerzhagen P(2024)EDeN: Enabling Low-Power CNN Inference on Edge Devices Using Prefetcher-assisted NVM SystemsProceedings of the 29th ACM/IEEE International Symposium on Low Power Electronics and Design10.1145/3665314.3670801(1-6)Online publication date: 5-Aug-2024
https://dl.acm.org/doi/10.1145/3665314.3670801
Li XJiao LSun QLiu FLiu XLi LChen PYang S(2024)A Category-Aware Curriculum Learning for Data-Free Knowledge DistillationIEEE Transactions on Multimedia10.1109/TMM.2024.339584426(9603-9618)Online publication date: 2-May-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3395844
Hong HChoi DKim NLee HKang BKang HKim H(2024)Survey of convolutional neural network accelerators on field-programmable gate array platforms: architectures and optimization techniquesJournal of Real-Time Image Processing10.1007/s11554-024-01442-821:3Online publication date: 29-Mar-2024
https://dl.acm.org/doi/10.1007/s11554-024-01442-8
Lee SKoo KLee JLee GJeong SO SKim H(2024)Vision transformer models for mobile/edge devices: a surveyMultimedia Systems10.1007/s00530-024-01312-030:2Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1007/s00530-024-01312-0
Kim NKim H(2023)Trunk Pruning: Highly Compatible Channel Pruning for Convolutional Neural Networks Without Fine-TuningIEEE Transactions on Multimedia10.1109/TMM.2023.333805226(5588-5599)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1109/TMM.2023.3338052

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents