research-article

Filter Pruning via Probabilistic Model-based Optimization for Accelerating Deep Convolutional Neural Networks

Authors:

Hong ChenAuthors Info & Claims

WSDM '21: Proceedings of the 14th ACM International Conference on Web Search and Data Mining

Pages 653 - 661

https://doi.org/10.1145/3437963.3441766

Published: 08 March 2021 Publication History

Abstract

Accelerating Deep Convolutional Neural Networks(CNNs) has recently received ever-increasing research focus. Among various approaches proposed in the literature, filter pruning has been regarded as a promising solution, which is due to its advantage in significant speedup and memory reduction of both network model and intermediate feature maps. Previous works utilized "smaller-norm-less-important" criterion to prune filters with smaller lp-norm values by pruning and retraining alternately. However, they ignore the effects of $feedback: most current approaches that prune filters only consider the statistics of the filters (e.g., prune filter with small lp-norm values), without considering the performance of the pruned model as an important feedback signal in the next iteration of filter pruning. To solve the problem of non-feedback, we propose a novel filter pruning method, namely Filter Pruning via Probabilistic Model-based Optimization (FPPMO). FPPMO solves the problem of non-feedback by pruning filters in a probabilistic manner. We introduce a pruning probability for each filter, and pruning is guided by sampling from the pruning probability distribution. An optimization method is proposed to update the pruning probability based on the performance of the pruned model in the pruning process. When applied to two image classification benchmarks, the effectiveness of our FPPMO is validated. Notably, on CIFAR-10, our FPPMO reduces more than 57% FLOPs on ResNet-110 with even 0.08% relative accuracy improvement. Moreover, on ILSVRC-2012, our FPPMO reduces more than 50% FLOPs on ResNet-101 without top-5 accuracy drop. Which proving that our FPPMO outperforms the state-of-the-art filter pruning method.

References

[1]

Shun-Ichi Amari. 1998. Natural gradient works efficiently in learning. Neural computation, Vol. 10, 2 (1998), 251--276.

[2]

Miguel A Carreira-Perpinán and Yerlan Idelbayev. 2018. ?Learning-Compression? Algorithms for Neural Net Pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8532--8541.

[3]

Miguel Á. Carreira-Perpiñán and Yerlan Idelbayev. 2018. 'Learning-Compression' Algorithms for Neural Net Pruning. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018).

[4]

Giovanna Castellano, Anna Maria Fanelli, and Marcello Pelillo. 1997. An iterative pruning algorithm for feedforward neural networks. IEEE transactions on Neural networks, Vol. 8, 3 (1997), 519--531.

[5]

George E Dahl, Dong Yu, Li Deng, and Alex Acero. 2012. Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition. IEEE Transactions on Audio, Speech, and Language Processing, Vol. 20, 1 (2012), 30--42.

Digital Library

[6]

Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. 2017. More is less: A more complicated network with less inference complexity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5840--5848.

[7]

Yiwen Guo, Anbang Yao, and Yurong Chen. 2016. Dynamic Network Surgery for Efficient DNNs. neural information processing systems (2016), 1379--1387.

[8]

Song Han, Huizi Mao, and William J Dally. 2015a. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015).

[9]

Song Han, Jeff Pool, John Tran, and William J Dally. 2015b. Learning both weights and connections for efficient neural networks. neural information processing systems (2015), 1135--1143.

[10]

Nikolaus Hansen and Andreas Ostermeier. 2001. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, Vol. 9, 2 (2001), 159--195.

[11]

Babak Hassibi and David G Stork. 1993. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information processing systems. 164--171.

[12]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016a. Deep Residual Learning for Image Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[13]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b. Deep Residual Learning for Image Recognition. computer vision and pattern recognition (2016), 770--778.

[14]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016c. Identity mappings in deep residual networks. In European conference on computer vision. Springer, 630--645.

[15]

Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. 2018. Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks. international joint conference on artificial intelligence (2018), 2234--2240.

[16]

Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. 2019. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4340--4349.

[17]

Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision. 1389--1397.

[18]

Babak Hosseini, Romain Montagne, and Barbara Hammer. 2019. Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation. In 19th IEEE International Conference on Data Mining (ICDM 2019) .

[19]

Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. Technical Report. Citeseer.

[20]

Yann Lecun, John S Denker, and Sara A Solla. 1989. Optimal Brain Damage., Vol. 2 (1989), 598--605.

[21]

Guiying Li, Chao Qian, Chunhui Jiang, Xiaofen Lu, and Ke Tang. 2018. Optimization based Layer-wise Magnitude-based Pruning for DNN Compression. In IJCAI. 2383--2389.

[22]

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2017a. Pruning Filters for Efficient ConvNets. international conference on learning representations (2017).

[23]

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2017b. Pruning filters for efficient convnets. ICLR (2017).

[24]

Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall F Tappen, and Marianna Penksy. 2015. Sparse Convolutional Neural Networks. (2015), 806--814.

[25]

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision. 2736--2744.

[26]

Jianhao Luo, Jianxin Wu, and Weiyao Lin. 2017a. ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression. international conference on computer vision (2017), 5068--5076.

[27]

Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. 2017b. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision. 5058--5066.

[28]

Atsushi Miyamae, Yuichi Nagata, Isao Ono, and Shigenobu Kobayashi. 2010. Natural policy gradient methods with parameter-based exploration for control tasks. In Advances in neural information processing systems. 1660--1668.

[29]

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2017. Pruning Convolutional Neural Networks for Resource Efficient Inference. international conference on learning representations (2017).

[30]

Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. 2017. Exploring sparsity in recurrent neural networks. arXiv preprint arXiv:1704.05119 (2017).

[31]

P. Q. Nguyen, T. Do, A. Nguyen-Thi, T. D. Ngo, D. Le, and T. H. Nguyen. 2016. Clustering web video search results with convolutional neural networks. In 2016 3rd National Foundation for Science and Technology Development Conference on Information and Computer Science (NICS). 135--140.

[32]

Yann Ollivier, Ludovic Arnold, Anne Auger, and Nikolaus Hansen. 2017. Information-geometric optimization algorithms: A unifying picture via invariance principles. The Journal of Machine Learning Research, Vol. 18, 1 (2017), 564--628.

Digital Library

[33]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).

[34]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, Vol. 115, 3 (2015), 211--252.

[35]

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A unified embedding for face recognition and clustering. computer vision and pattern recognition (2015), 815--823.

[36]

Abigail See, Minh-Thang Luong, and Christopher D Manning. 2016. Compression of neural machine translation models via pruning. arXiv preprint arXiv:1606.09274 (2016).

[37]

Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. international conference on learning representations (2015).

[38]

Slawomir W Stepniewski and Andy J Keane. 1997. Pruning backpropagation neural networks using modern stochastic optimisation techniques. Neural Computing & Applications, Vol. 5, 2 (1997), 76--98.

[39]

Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. 2014. Natural evolution strategies. The Journal of Machine Learning Research, Vol. 15, 1 (2014), 949--980.

Digital Library

[40]

Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. 2017. Designing energy-efficient convolutional neural networks using energy-aware pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5687--5695.

[41]

Sun Yi, Daan Wierstra, Tom Schaul, and Jürgen Schmidhuber. 2009. Stochastic search using the natural gradient. In Proceedings of the 26th Annual International Conference on Machine Learning. ACM, 1161--1168.

Digital Library

[42]

Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis. 2018. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9194--9203.

[43]

Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. arXiv preprint arXiv:1605.07146 (2016).

[44]

Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. 2018. A systematic dnn weight pruning framework using alternating direction method of multipliers. In Proceedings of the European Conference on Computer Vision (ECCV). 184--199.

Digital Library

Cited By

Sun MDong FHuang ZLuo J(2024)Adaptive Model Compression for Steel Plate Surface Defect Detection: An Expert Knowledge and Working Condition-Based ApproachTsinghua Science and Technology10.26599/TST.2024.901003929:6(1851-1871)Online publication date: Dec-2024
https://doi.org/10.26599/TST.2024.9010039
Min QLuo FDong WGu CDing W(2024)Communication-efficient Federated Learning via Personalized Filter PruningInformation Sciences10.1016/j.ins.2024.121030(121030)Online publication date: Jun-2024
https://doi.org/10.1016/j.ins.2024.121030
Wang XWang YLi J(2023)An Industrial-grade Solution for Convolutional Neural Network Optimization and Deployment2023 4th International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT)10.1109/AINIT59027.2023.10212632(46-50)Online publication date: 16-Jun-2023
https://doi.org/10.1109/AINIT59027.2023.10212632
Show More Cited By

Index Terms

Filter Pruning via Probabilistic Model-based Optimization for Accelerating Deep Convolutional Neural Networks

Recommendations

A Unified-Model via Block Coordinate Descent for Learning the Importance of Filter
ICMR '21: Proceedings of the 2021 International Conference on Multimedia Retrieval

Deep Convolutional Neural Networks (CNNs) are increasingly used in multimedia retrieval, and accelerating Deep CNNs has recently received an ever-increasing research focus. Among various approaches proposed in the literature, filter pruning has been ...
Incremental Filter Pruning via Random Walk for Accelerating Deep Convolutional Neural Networks
WSDM '20: Proceedings of the 13th International Conference on Web Search and Data Mining

Accelerating Deep Convolutional Neural Networks (CNNs) has recently received ever-increasing research focus. Among various approaches proposed in the literature, filter pruning has been regarded as a promising solution, which is due to its advantage in ...
Filter Pruning via Similarity Clustering for Deep Convolutional Neural Networks
Neural Information Processing
Abstract
Network pruning is a technique to obtain a smaller lightweight model by removing the redundant structure from pre-trained models. However, existing methods are mainly based on the importance of filters in the whole network. Unlike previous methods,...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '21: Proceedings of the 14th ACM International Conference on Web Search and Data Mining

March 2021

1192 pages

ISBN:9781450382977

DOI:10.1145/3437963

General Chairs:
Liane Lewin-Eytan
Amazon, Israel
,
David Carmel
Amazon, Israel
,
Elad Yom-Tov
Microsoft, Israel
,
Program Chairs:
Eugene Agichtein
Emory University and Amazon, USA
,
Evgeniy Gabrilovich
Google Health, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 March 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSFC
National Key R & D Program of China

Conference

WSDM '21

Sponsor:

WSDM '21: The Fourteenth ACM International Conference on Web Search and Data Mining

March 8 - 12, 2021

Virtual Event, Israel

Acceptance Rates

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
220
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sun MDong FHuang ZLuo J(2024)Adaptive Model Compression for Steel Plate Surface Defect Detection: An Expert Knowledge and Working Condition-Based ApproachTsinghua Science and Technology10.26599/TST.2024.901003929:6(1851-1871)Online publication date: Dec-2024
https://doi.org/10.26599/TST.2024.9010039
Min QLuo FDong WGu CDing W(2024)Communication-efficient Federated Learning via Personalized Filter PruningInformation Sciences10.1016/j.ins.2024.121030(121030)Online publication date: Jun-2024
https://doi.org/10.1016/j.ins.2024.121030
Wang XWang YLi J(2023)An Industrial-grade Solution for Convolutional Neural Network Optimization and Deployment2023 4th International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT)10.1109/AINIT59027.2023.10212632(46-50)Online publication date: 16-Jun-2023
https://doi.org/10.1109/AINIT59027.2023.10212632
Khan SSajjad MAbbas NRehman A(2022)A Review on Machine Learning-Based WBCs Analysis in Blood Smear Images: Key Challenges, Datasets, and Future DirectionsPrognostic Models in Healthcare: AI and Statistical Approaches10.1007/978-981-19-2057-8_11(293-314)Online publication date: 7-Jul-2022
https://doi.org/10.1007/978-981-19-2057-8_11

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents