Abstract
The increasing complexity of artificial intelligence models has given rise to extensive work toward understanding the inner workings of neural networks. Much of that work, however, has focused on manipulating input data feeding the network to assess their effects on network output or pruning model components after the often-extensive time-consuming training. It is shown in this study that model simplification can benefit from investigating the network node, the most fundamental unit of neural networks, during training. Whereas studies on simplification of model structure have mostly required repeated model training, assessing evolving trends in node weights toward model stabilization may circumvent that requirement. Node magnitude stability, defined as the number of epochs where node weights retained their magnitude within a tolerance value, was the central construct in this study. To test evolving trends, a manipulated, a contrived, and two life science data sets were used. Data sets were run on convolutional and deep neural network models. Findings indicated that neural network progress toward stability differed by model, where CNNs tended to add influential nodes early during training. The magnitude stability approach of this study showed superior time efficiencies, which may assist in XAI research toward producing more transparent models and clear outcomes to technical and non-technical audiences.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
The data will be available upon publication of this manuscript.
Code availability
The code for this work will be available upon publication of this manuscript.
References
Aggarwal, C. C. (2018). Neural networks and deep learning: A textbook (1st ed.). Springer.
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020). Explainable machine learning in deployment. In Proceedings of the 2020 conference on fairness, accountability, and transparency. FAT* ’20, pp. 648–657. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3351095.3375624. Accessed 2022–09–01.
Cai, L., An, Z., Yang, C., & Xu, Y. (2021). Softer pruning, incremental regularization. In 2020 25th international conference on pattern recognition (ICPR), pp. 224–230. https://doi.org/10.1109/ICPR48806.2021.9412993
Choudhary, T., Mishra, V., Goswami, A., & Sarangapani, J. (2021). A transfer learning with structured filter pruning approach for improved breast cancer classification on point-of-care devices. Computers in Biology and Medicine, 134, 104432. https://doi.org/10.1016/j.compbiomed.2021.104432
Chouliaras, A., Fragkou, E., & Katsaros, D. (2022). Feed forward neural network sparsification with dynamic pruning. In Proceedings of the 25th pan-hellenic conference on informatics. PCI ’21, pp. 12–17. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3503823.3503826. Accessed 2023–05–21.
Denil, M., Shakibi, B., Dinh, L., Ranzato, M. A., & de Freitas, N. (2013). Predicting parameters in deep learning. In Advances in neural information processing systems, vol. 26. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2013/hash/7fec306d1e665bc9c748b5d2b99a6e97-Abstract.html. Accessed 2022–08–31.
Eilertsen, G., Jönsson, D., Ropinski, T., Unger, J., & Ynnerman, A. (2020). Classifying the classifier: Dissecting the weight space of neural networks. arXiv. arXiv:2002.05688 [cs]. https://doi.org/10.48550/arXiv.2002.05688. Accessed 2023–08–07.
Glorfeld, L. W. (1996). A Methodology for simplification and interpretation of backpropagation-based neural network models. Expert Systems with Applications, 10(1), 37–54. https://doi.org/10.1016/0957-4174(95)00032-1
Golatkar, A., Achille, A., & Soatto, S. (2019). Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. Advances in Neural Information Processing Systems. https://doi.org/10.48550/arXiv.1905.13277. arXiv. arXiv:1905.13277 [cs, stat].
Goodfellow, I., Lee, H., Le, Q., Saxe, A., & Ng, A. (2009). Measuring Invariances in Deep Networks. In Advances in neural information processing systems, vol. 22. Curran Associates, Inc., https://proceedings.neurips.cc/paper/2009/hash/428fca9bc1921c25c5121f9da7815cde-Abstract.html. Accessed 2022–08–30.
Guan, Y., Li, Z., Leng, J., Lin, Z., & Guo, M. (2022). Transkimmer: Transformer Learns to Layer-wise Skim. In Proceedings of the 60th annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp. 7275–7286. Association for Computational Linguistics, Dublin, Ireland. https://doi.org/10.18653/v1/2022.acl-long.502. Accessed 2023–05–28.
Hajabdollahi, M., Esfandiarpoor, R., Najarian, K., Karimi, N., Samavi, S., & Reza Soroushmehr, S.M. (2019). Hierarchical pruning for simplification of convolutional neural networks in diabetic retinopathy classification. In 2019 41st annual international conference of the IEEE engineering in medicine and biology society (EMBC), pp. 970–973. https://doi.org/10.1109/EMBC.2019.8857769 ISSN: 1558-4615.
Hall, C. A. S. (1988). An assessment of several of the historically most influential theoretical models used in ecology and of the data provided in their support. Ecological Modelling, 43(1), 5–31. https://doi.org/10.1016/0304-3800(88)90070-1
Hamet, P., & Tremblay, J. (2017). Artificial intelligence in medicine. Metabolism, 69, 36–40. https://doi.org/10.1016/j.metabol.2017.01.011
Han, S., Mao, H., & Dally, W. J. (2016). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv:1510.00149 [cs]. https://doi.org/10.48550/arXiv.1510.00149. Accessed 2023–01–18.
Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv. arXiv:1503.02531 [cs, stat]. https://doi.org/10.48550/arXiv.1503.02531 Accessed 2023–01–19.
Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., & Keutzer, K. (2016). SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and \(<\)0.5MB model size. arXiv:1602.07360 [cs]. https://doi.org/10.48550/arXiv.1602.07360. Accessed 2023-01-18.
Ivanovs, M., Kadikis, R., & Ozols, K. (2021). Perturbation-based methods for explaining deep neural networks: A survey. Pattern Recognition Letters, 150, 228–234. https://doi.org/10.1016/j.patrec.2021.06.030
Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255–260. https://doi.org/10.1126/science.aaa8415
Kareva, I., & Karev, G. (2018). from experiment to theory: What can we learn from growth curves? Bulletin of Mathematical Biology, 80(1), 151–174. https://doi.org/10.1007/s11538-017-0347-5
Kaul, V., Enslin, S., & Gross, S. A. (2020). History of artificial intelligence in medicine. Gastrointestinal Endoscopy, 92(4), 807–812. https://doi.org/10.1016/j.gie.2020.06.040. Accessed 2021-12-21.
Kaya, Y., Hong, S., & Dumitras, T. (2019). Shallow-deep networks: Understanding and mitigating network overthinking. arXiv:1810.07052 [cs, stat]. https://doi.org/10.48550/arXiv.1810.07052. Accessed 2023-05-28.
Kingsland, S. (1982). The refractory model: The logistic curve and the history of population ecology. The Quarterly Review of Biology, 57(1), 29–52. https://doi.org/10.1086/412574
Liang, T., Glossner, J., Wang, L., Shi, S., & Zhang, X. (2021). Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461, 370–403. https://doi.org/10.1016/j.neucom.2021.07.045
Makino, T., Jastrzȩbski, S., Oleszkiewicz, W., Chacko, C., Ehrenpreis, R., Samreen, N., Chhor, C., Kim, E., Lee, J., Pysarenko, K., Reig, B., Toth, H., Awal, D., Du, L., Kim, A., Park, J., Sodickson, D. K., Heacock, L., Moy, L., … Geras, K. J. (2022). Differences between human and machine perception in medical diagnosis. Scientific Reports, 12(1), 6877. https://doi.org/10.1038/s41598-022-10526-z
Malik, S., Tyagi, A. K., & Mahajan, S. (2022). Architecture, generative model, and deep reinforcement learning for IoT applications: Deep learning perspective. In S. Pal, D. De, & R. Buyya (Eds.), Artificial intelligence-based internet of things systems. Internet of things (pp. 243–265). Springer. https://doi.org/10.1007/978-3-030-87059-1_9
Martin, S.M., Casey, J. R., & Kane, S. (2021). History of artificial intelligence and personalized learning. In Serious games in personalized learning. Routledge.
Martin, C. H., Peng, T. S., & Mahoney, M. W. (2021). Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12(1), 4122. https://doi.org/10.1038/s41467-021-24025-8
Montavon, G., Braun, M. L., & Mueller, K.-R. (2011). Kernel analysis of deep networks. Journal of Machine Learning Research, 12, 19.
Montesinos López, O. A., Montesinos López, A., & Crossa, J. (2022). Fundamentals of artificial neural networks and deep learning. In O. A. Montesinos López, A. Montesinos López, & J. Crossa (Eds.), Multivariate statistical machine learning methods for genomic prediction (pp. 379–425). Springer. https://doi.org/10.1007/978-3-030-89010-0_10
Quinn, T. P., Gupta, S., Venkatesh, S., & Le, V. (2021). A field guide to scientific XAI: Transparent and interpretable deep learning for bioinformatics research. arXiv. arXiv:2110.08253 [cs, q-bio]. Accessed 2022–09–01.
Raghu, M., & Schmidt, E. (2020). A survey of deep learning for scientific discovery. arXiv. arXiv:2003.11755 [cs, stat]. https://doi.org/10.48550/2003.11755. Accessed 2023–01–19.
Saravagi, D., Agrawal, S., Saravagi, M., & Rahman, M. H. (2022). Diagnosis of lumbar spondylolisthesis using a pruned CNN model. Computational and Mathematical Methods in Medicine, 2022, 2722315. https://doi.org/10.1155/2022/2722315
Schirrmeister, R. T., Liu, R., Hooker, S., & Ball, T. (2022). When less is more: Simplifying inputs aids neural network understanding. arXiv. arXiv:2201.05610 [cs]. https://doi.org/10.48550/arXiv.2201.05610. Accessed 2023–01–18.
Singla, S., Pollack, B., Chen, J., & Batmanghelich, K. (2020). Explanation by Progressive Exaggeration. https://openreview.net/forum?id=H1xFWgrFPS Accessed 2023–01–19.
Srinivas, S., & Babu, R.V. (2015). Data-free parameter pruning for Deep Neural Networks. arXiv:1507.06149 [cs]. https://doi.org/10.48550/arXiv.1507.06149. Accessed 2023-01-19.
Sun, M., Song, Z., Jiang, X., Pan, J., & Pang, Y. (2017). Learning Pooling for Convolutional Neural Network. Neurocomputing, 224, 96–104. https://doi.org/10.1016/j.neucom.2016.10.049
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2020). Efficient processing of deep neural networks. Morgan & Claypool Publishers.
Unterthiner, T., Keysers, D., Gelly, S., Bousquet, O., & Tolstikhin, I. (2021). Predicting neural network accuracy from weights. arXiv. arXiv:2002.11448 [cs, stat]. https://doi.org/10.48550/arXiv.2002.11448. Accessed 2023–08–07.
Ye, D., Lin, Y., Huang, Y., & Sun, M. (2021). TR-BERT: Dynamic token reduction for accelerating BERT inference. arXiv:2105.11618 [cs]. https://doi.org/10.48550/arXiv.2105.11618. Accessed 2023–05–28.
Yeom, S.-K., Seegerer, P., Lapuschkin, S., Binder, A., Wiedemann, S., Müller, K.-R., & Samek, W. (2021). Pruning by explaining: A novel criterion for deep neural network pruning. Pattern Recognition, 115, 107899. https://doi.org/10.1016/j.patcog.2021.107899
Yuan, C., & Agaian, S. S. (2023). A comprehensive review of binary neural network. Artificial Intelligence Review. https://doi.org/10.1007/s10462-023-10464-w
Zeng, J., Zhang, M., & Lin, S.-B. (2022). Fully corrective gradient boosting with squared hinge: Fast learning rates and early stopping. Neural Networks, 147, 136–151. https://doi.org/10.1016/j.neunet.2021.12.016
Zou, J., Rui, T., Zhou, Y., Yang, C., & Zhang, S. (2018). Convolutional neural network simplification via feature map pruning. Computers and Electrical Engineering, 70, 950–958. https://doi.org/10.1016/j.compeleceng.2018.01.036
Funding
No funding was provided for this study.
Author information
Authors and Affiliations
Contributions
RR provided the initial draft and analyses; AS provided editing and analytical suggestions.
Corresponding author
Ethics declarations
Conflict of interest
No conflict of interests that are relevant to the content of this manuscript.
Ethics approval
Not Applicable.
Consent to participate
Not Applicable.
Consent for publication
Not Applicable.
Additional information
Editors: Vu Nguyen, Dani Yogatama.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Riedel, R., Segev, A. Neural network structure simplification by assessing evolution in node weight magnitude. Mach Learn 113, 3693–3710 (2024). https://doi.org/10.1007/s10994-023-06438-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10994-023-06438-2