Abstract
DNNs are abundantly represented in real-life applications because of their accuracy in challenging problems, yet their demanding memory and computational costs challenge their applicability to resource-constrained environments. Taming computational costs has hitherto focused on first-order techniques, such as eliminating numerically insignificant neurons/filters through numerical contribution metric prioritizations, yielding passable improvements. Yet redundancy in DNNs extends well beyond the limits of numerical insignificance. Modern DNN layers exhibit a significant correlation among output activations; hence, the number of extracted orthogonal features at each layer rarely exceeds a small fraction of the layer size. The exploitation of this observation necessitates the quantification of information content at layer outputs. To this end, we employ practical data analysis techniques coupled with a novel feature elimination algorithm to identify a minimal set of computation units that capture the information content of the layer and squash the rest. Linear transformations on the subsequent layer ensure accuracy retention despite the removal of a significant portion of the computation units. The one-shot application of the outlined technique can shrink the VGG-16 model size \(4.9{\times }\) and speed up its execution by \(3.4{\times }\) with negligible accuracy loss while requiring no additional fine-tuning. The proposed approach, in addition to delivering results overwhelmingly superior to hitherto promulgated heuristics, furthermore promises to spearhead the design of more compact deep learning models through an improved understanding of DNN redundancy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
S and \(V^T\) are statically multiplied to form weight matrix \(SV^T\).
- 2.
Evaluated only on VGG-16 and ResNet-50 with ReLU activation function.
- 3.
Unfortunately, this metric is rarely reported in prior literature.
- 4.
It requires \({\sim }15\) h of computation time.
References
Abadi, M., et al.: TensorFlow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2016), pp. 265–283 (2016)
Ayinde, B.O., Zurada, J.M.: Building efficient ConvNets using redundant feature pruning. arXiv:1802.07653 (2018)
Babaeizadeh, M., Smaragdis, P., Campbell, R.H.: NoiseOut: a simple way to prune neural networks. arXiv:1611.06211 (2016)
Chen, C., Seff, A., Kornhauser, A., Xiao, J.: DeepDriving: learning affordance for direct perception in autonomous driving. In: IEEE International Conference on Computer Vision, pp. 2722–2730 (2015)
Chen, Y.H., Emer, J., Sze, V.: Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks. ACM SIGARCH Comput. Architect. News 44–3, 367–379 (2016)
Chollet, F., et al.: Keras (2015). https://keras.io
Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Han, S., et al.: EIE: efficient inference engine on compressed deep neural network. ACM SIGARCH Compu. Architect. News 44(3), 243–254 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
He, Y., Kang, G., Dong, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 2234–2240 (2018)
He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1389–1397 (2017)
Hu, H., Peng, R., Tai, Y.W., Tang, C.K.: Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. arXiv:1607.03250 (2016)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 (2015)
Kim, Y.D., et al.: Compression of deep convolutional neural networks for fast and low power mobile applications. In: International Conference on Learning Representations (2016)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report. Citeseer (2009)
Lay, D.C., Lay, S.R., McDonald, J.J.: Linear Algebra and Its Applications, 5th edn. Pearson Publishing Co., USA (2016)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient ConvNets. In: International Conference on Learning Representations (2017)
Lin, D., Talathi, S., Annapureddy, S.: Fixed point quantization of deep convolutional networks. In: International Conference on Machine Learning, pp. 2849–2858 (2016)
Lin, S., et al.: Accelerating convolutional networks via global & dynamic filter pruning. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 2425–2432 (2018)
Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. In: International Conference on Learning Representations (2019)
Luo, J.H., Wu, J., Lin, W.: ThiNet: a filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5058–5066 (2017)
Mariet, Z., Sra, S.: Diversity networks. In: International Conference on Learning Representations (2016)
Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. In: International Conference on Learning Representations (2017)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. (IJCV) 115(3), 211–252 (2015)
Samajdar, A., Zhu, Y., Whatmough, P., Mattina, M., Krishna, T.: SCALE-Sim: systolic CNN accelerator simulator. arXiv:1811.02883 (2018)
Shlens, J.: A tutorial on principal component analysis. arXiv preprint arXiv:1404.1100 (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Sze, V., Chen, Y.H., Yang, T.J., Emer, J.S.: Efficient processing of deep neural networks: a tutorial and survey. Proc. IEEE 105(12), 2295–2329 (2017)
Xiao, X., Wang, Z., Rajasekaran, S.: Autoprune: automatic network pruning by regularizing auxiliary parameters. In: Advances in Neural Information Processing Systems, pp. 13681–13691 (2019)
You, Z., Yan, K., Ye, J., Ma, M., Wang, P.: Gate decorator: global filter pruning method for accelerating deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 2130–2141 (2019)
Yu, R., et al.: NISP: pruning networks using neuron importance score propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9194–9203 (2018)
Zhang, X., Zou, J., Ming, X., He, K., Sun, J.: Efficient and accurate approximations of nonlinear convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1984–1992 (2015)
Zhuang, Z., et al.: Discrimination-aware channel pruning for deep neural networks. In: Advances in Neural Information Processing Systems, pp. 875–886 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Ozen, E., Orailoglu, A. (2021). Squeezing Correlated Neurons for Resource-Efficient Deep Neural Networks. In: Hutter, F., Kersting, K., Lijffijt, J., Valera, I. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2020. Lecture Notes in Computer Science(), vol 12458. Springer, Cham. https://doi.org/10.1007/978-3-030-67661-2_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-67661-2_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67660-5
Online ISBN: 978-3-030-67661-2
eBook Packages: Computer ScienceComputer Science (R0)