research-article

Architecture search of accurate and lightweight CNNs using genetic algorithm

Authors:

Mingming SuAuthors Info & Claims

Genetic Programming and Evolvable Machines, Volume 25, Issue 1

https://doi.org/10.1007/s10710-024-09484-4

Published: 01 April 2024 Publication History

Abstract

Convolutional neural networks (CNNs) are popularly-used in various AI fields, yet the design of CNN architectures heavily depends on domain expertise. Evolutionary neural architecture search (ENAS) methods can search for neural architectures automatically using evolutionary computation algorithms, e.g. genetic algorithm. However, most existing ENAS methods solely focus on the network accuracy, which leads to large-sized networks to be evolved and huge cost in computation resources and search time. Even though there are ENAS works using multi-objective techniques to optimize both the accuracy and size of CNNs, they are complex and time/resource-consuming. In this work, two new ENAS methods are designed, which aim to evolve both accurate and lightweight CNN architectures efficiently using genetic algorithm (GA). They are termed as GACNN_WS (GA CNN Weighted Sum) and GACNN_LE (GA CNN Local Elitism) respectively. Specifically, GACNN_WS designs a weighted-sum fitness of two items (i.e. accuracy and size) to evaluate candidate networks. GACNN_LE sets the accuracy as its fitness like most other ENAS methods, and designs a local elitism strategy to consider the network size. Thus, GACNN_WS and GACNN_LE can search for both accurate and lightweight CNNs without using multi-objective techniques. Results show that the proposed methods have better search ability than state-of-the-art NAS methods, which consume less time and generate better CNNs with lower error rates and parameter numbers for classification on CIFAR-10. Moreover, the evolved CNNs of the proposed methods generally perform better than eleven hand-designed CNNs.

References

[1]

K. Ahmed, L. Torresani, Maskconnect: connectivity learning by gradient descent, in Proceedings of the European Conference on Computer Vision (ECCV) (2018), pp. 349–365

[2]

B. Baker, O. Gupta, N. Naik, R. Raskar, Designing neural network architectures using reinforcement learning, in International Conference on Learning Representations (2017), pp. 1–18

[3]

T. Chen, I. Goodfellow, J. Shlens, Net2net: Accelerating learning via knowledge transfer, in International Conference on Learning Representations (2016)

[4]

M. Dhouibi, A. Salem, S.B. Saoud, Optimization of CNN model for image classification, in IEEE International Conference on Design and Test of Integrated Micro and Nano-Systems (2021)

[5]

J.K. Duggal, El-Sharkawy, M.: High performance squeezenext for cifar-10, in National Aerospace and Electronics Conference (2019)

[6]

Elsken T, Metzen JH, and Hutter F Neural architecture search: a survey J. Mach. Learn. Res. 2019 20 1 1997-2017

Digital Library

[7]

J. Fang, Y. Sun, Q. Zhang, Y. Li, X. Wang, Densely connected search space for more flexible neural architecture search, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp. 10628–10637

[8]

X. Gastaldi, Shake–shake regularization of 3-branch residual networks, in International Conference on Learning Representations (2017), pp. 770–778

[9]

K. Han, Y. Wang, Q. Tian, J. Guo, C. Xu, C. Xu, Ghostnet: more features from cheap operations, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), pp. 1577–1586

[10]

Hasani B, Negi PS, and Mahoor MH BReG-NeXt: facial affect computing using adaptive residual networks with bounded gradient IEEE Trans. Affect. Comput. 2022 13 2 1023-1036

[11]

K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), pp. 770–778

[12]

A. Howard, M. Sandler, B. Chen, W. Wang, L.C. Chen, M. Tan, G. Chu, V. Vasudevan, Y. Zhu, R. Pang, Searching for mobilenetv3, in 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2020)

[13]

S. Hu, R. Cheng, C. He, Z. Lu, Multi-objective neural architecture search with almost no training, in International Conference on Evolutionary Multi-Criterion Optimization (Springer, 2021), pp. 492–503

[14]

Hu T, Tomassini M, and Banzhaf W A network perspective on genotype–phenotype mapping in genetic programming Genet. Program. Evolvable Mach. 2020 21 375-397

Digital Library

[15]

G. Huang, Z. Liu, L. Van Der Maaten, K.Q. Weinberger, Densely connected convolutional networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 4700–4708

[16]

Jiang P, Xue Y, and Neri F Continuously evolving dropout with multi-objective evolutionary optimisation Eng. Appl. Artif. Intell. 2023 124

Digital Library

[17]

Jiang P, Xue Y, and Neri F Convolutional neural network pruning based on multi-objective feature map selection for image classification Appl. Soft Comput. 2023 139

Digital Library

[18]

A. Krizhevsky, G. Hinton, Learning multiple layers of features from tiny images, in Handbook of Systemic Autoimmune Diseases (2009)

[19]

G. Larsson, M. Maire, G. Shakhnarovich, Fractalnet: ultra-deep neural networks without residuals, in International Conference on Learning Representations (2017), pp. 770–778

[20]

G. Li, G. Qian, I.C. Delgadillo, M. Muller, A. Thabet, B. Ghanem, SGAS: sequential greedy architecture search, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), pp. 1620–1630

[21]

Liang M Figure-ground image segmentation using feature-based multi-objective genetic programming techniques Neural Comput. Appl. 2019 31 7 3075-3094

Digital Library

[22]

T. Liang, Y. Wang, Z. Tang, G. Hu, H. Ling, OPANAS: one-shot path aggregation network architecture search for object detection, pp. 1–9 (2021). arXiv:2103.04507

[23]

Liang Y, Zhang M, and Browne WN Image feature selection using genetic programming for figure-ground segmentation Eng. Appl. Artif. Intell. 2017 62 Jun. 96-108

Digital Library

[24]

M. Lin, P. Wang, Z. Sun, H. Chen, X. Sun, Q. Qian, H. Li, R. Jin, Zen-NAS: a zero-shot NAS for high-performance image recognition, in Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 347–356

[25]

H. Liu, K. Simonyan, O. Vinyals, C. Fernando, K. Kavukcuoglu, Hierarchical representations for efficient architecture search, in International Conference of Learning Representation (2018)

[26]

H. Liu, K. Simonyan, Y. Yang, Darts: differentiable architecture search, in International Conference on Learning Representations (2019), pp. 1–13

[27]

Y. Liu, Y. Sun, B. Xue, M. Zhang, G.G. Yen, K.C. Tan, A survey on evolutionary neural architecture search, in 2021 IEEE Congress on Evolutionary Computation (CEC) (2021)

[28]

Loni M, Sinaei S, Zoljodi A, Daneshtalab M, and Sjödin M Deepmaker: a multi-objective optimization framework for deep neural networks in embedded systems Microprocess. Microsyst. 2020 73

Digital Library

[29]

Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, W. Banzhaf, NSGA-Net: neural architecture search using multi-objective genetic algorithm, in Genetic and Evolutionary Computation Conference 2019 (2019)

[30]

Neri F, Cotta C, and Moscato P Handbook of Memetic Algorithms 2011 Berlin Springer

Digital Library

[31]

W. Peng, X. Hong, H. Chen, G. Zhao, Learning graph convolutional network for skeleton-based human action recognition by neural searching, in National Conference on Artificial Intelligence (2020)

[32]

Y. Peng, A. Song, V. Ciesielski, H.M. Fayek, X. Chang, PRE-NAS: predictor-assisted evolutionary neural architecture search, in Proceedings of the Genetic and Evolutionary Computation Conference (2022), pp. 1066–1074

[33]

A. Piergiovanni, A. Angelova, A. Toshev, M. Ryoo, Evolving space-time neural architectures for video, in 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019)

[34]

E. Real, S. Moore, A. Selle, S. Saxena, Y.L. Suematsu, J. Tan, Q.V. Le, A. Kurakin, Large-scale evolution of image classifiers, in 34th International Conference on Machine Learning (2017), pp. 2902–2911

[35]

D. Sapra, A.D. Pimentel, Constrained evolutionary piecemeal training to design convolutional neural networks, in International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems (Springer, 2020), pp. 709–721

[36]

Scardua LA Genetic Algorithms. Applied Evolutionary Algorithms for Engineers Using Python 2021 Boca Raton CRC Press

[37]

R. Shin, C. Packer, D. Song, Differentiable neural network architecture search, in International Conference on Learning Representations (Workshop Track) (2018), pp. 1–4

[38]

K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in International Conference on Learning Representations (2015)

[39]

N. Sinha, K.W. Chen, Evolving neural architecture using one shot model, in Proceedings of the Genetic and Evolutionary Computation Conference (2021), pp. 910–918

[40]

N. Sinha, K.W. Chen, Neural architecture search using progressive evolution, in Proceedings of the Genetic and Evolutionary Computation Conference (2022), pp. 1093–1101

[41]

Song X, Zhang Y, Gong D, and Sun X Feature selection using bare-bones particle swarm optimization with mutual information Pattern Recognit. J. Pattern Recognit. Soc. 2021 112 1 1-17

[42]

M. Tan, Q. Le, Efficientnetv2: smaller models and faster training, in International Conference on Machine Learning (2021), pp. 10096–10106

[43]

X. Wang, C. Xue, J. Yan, X. Yang, Y. Hu, K. Sun, MergeNAS: merge operations into one for differentiable architecture search, in Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence (2021), pp. 3065–3072

[44]

Wei H, Lee F, Hu C, and Chen Q MOO-DNAS: efficient neural network design via differentiable architecture search based on multi-objective optimization IEEE Access 2022 10 14195-14207

[45]

Williams RJ Simple statistical gradient-following algorithms for connectionist reinforcement learning Mach. Learn. 1992 8 3–4 229-256

Digital Library

[46]

M. Wistuba, Deep learning architecture search by neuro-cell-based evolution with function-preserving mutations, in Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2018)

[47]

Xue Y, Jiang P, Neri F, and Liang J A multi-objective evolutionary approach based on graph-in-graph for neural architecture search of convolutional neural networks Int. J. Neural Syst. 2021 31 09 2150035

[48]

Xue Y, Wang Y, and Liang J A self-adaptive gradient descent search algorithm for fully-connected neural networks Neurocomputing 2022 478 70-80

[49]

Xue Y, Wang Y, Liang J, and Slowik A A self-adaptive mutation neural architecture search algorithm based on blocks IEEE Comput. Intell. Mag. 2021 16 3 67-78

Digital Library

[50]

Z. Yang, Y. Wang, X. Chen, B. Shi, C. Xu, C. Xu, Q. Tian, C. Xu, Cars: continuous evolution for efficient neural architecture search, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), pp. 1829–1838

[51]

Zhang H, Jin Y, Cheng R, and Hao K Efficient evolutionary search of attention convolutional networks via sampled training and node inheritance IEEE Trans. Evol. Comput. 2020 25 2 371-385

[52]

M. Zhang, H. Li, S. Pan, X. Chang, S. Su, Overcoming multi-model forgetting in one-shot NAS with diversity maximization, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), pp. 7809–7818

[53]

Zhao L, Wang L, Jia Y, and Cui Y A lightweight deep neural network with higher accuracy PLoS ONE 2022 17 8

[54]

Y. Zhao, L. Wang, Y. Tian, R. Fonseca, T. Guo, Few-shot neural architecture search, in International Conference on Machine Learning (2021)

[55]

Z. Zhong, J. Yan, C.L. Liu, Practical network blocks design with q-learning, pp. 1–11 (2017). arXiv:1708.05552v3

[56]

Zhou J, He Q, Cheng G, and Lin Z Union-net: lightweight deep neural network model suitable for small data sets J. Supercomput. 2022 79 7228-7243

Digital Library

[57]

B. Zoph, Q.V. Le, Neural architecture search with reinforcement learning, pp. 1–16 (2016). arXiv:1611.01578

[58]

B. Zoph, V. Vasudevan, J. Shlens, Q.V. Le, Learning transferable architectures for scalable image recognition, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), pp. 8697–8710

Cited By

Gerber MPillay NLi XHandl J(2024)Dynamic Neural Architecture Search for Image ClassificationProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3664117(1554-1562)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3664117

Recommendations

Multi-objective Pruning for CNNs Using Genetic Algorithm
Artificial Neural Networks and Machine Learning – ICANN 2019: Deep Learning
Abstract
In this work, we propose a heuristic genetic algorithm (GA) for pruning convolutional neural networks (CNNs) according to the multi-objective trade-off among error, computation and sparsity. In our experiments, we apply our approach to prune pre-...
Evolving convolutional neural networks by symbiotic organisms search algorithm for image classification
Abstract
Convolutional neural networks (CNNs) are important neural networks in the deep learning field. In general, CNN design requires manual iterations and adjustments based on the characteristics of the data and a priori knowledge of the ...
Highlights
- Symbiotic organisms search algorithm is introduced into CNN architecture search.
A Neural Architecture Search Method using Auxiliary Evaluation Metric based on ResNet Architecture
GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary Computation

This paper proposes a neural architecture search space using ResNet as a framework, with search objectives including parameters for convolution, pooling, fully connected layers, and connectivity of the residual network. In addition to recognition ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Genetic Programming and Evolvable Machines

Genetic Programming and Evolvable Machines Volume 25, Issue 1

Jun 2024

371 pages

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 April 2024

Accepted: 29 February 2024

Revision received: 31 January 2024

Received: 23 June 2023

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gerber MPillay NLi XHandl J(2024)Dynamic Neural Architecture Search for Image ClassificationProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3664117(1554-1562)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3664117

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents