research-article

Neural architecture search using progressive evolution

Authors:

Nilotpal Sinha,

Kuan-Wen ChenAuthors Info & Claims

GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference

Pages 1093 - 1101

https://doi.org/10.1145/3512290.3528707

Published: 08 July 2022 Publication History

Abstract

Vanilla neural architecture search using evolutionary algorithms (EA) involves evaluating each architecture by training it from scratch, which is extremely time-consuming. This can be reduced by using a supernet to estimate the fitness of every architecture in the search space due to its weight sharing nature. However, the estimated fitness is very noisy due to the co-adaptation of the operations in the supernet. In this work, we propose a method called pEvoNAS wherein the whole neural architecture search space is progressively reduced to smaller search space regions with good architectures. This is achieved by using a trained supernet for architecture evaluation during the architecture search using genetic algorithm to find search space regions with good architectures. Upon reaching the final reduced search space, the supernet is then used to search for the best architecture in that search space using evolution. The search is also enhanced by using weight inheritance wherein the supernet for the smaller search space inherits its weights from previous trained supernet for the bigger search space. Experimentally, pEvoNAS gives better results on CIFAR-10 and CIFAR-100 while using significantly less computational resources as compared to previous EA-based methods. The code for our paper can be found here.

Supplemental Material

PDF File

Supplemental material.

Download
776.76 KB

References

[1]

Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. 2017. Designing Neural Network Architectures using Reinforcement Learning. International Conference on Learning Representations (2017).

[2]

Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. 2018. Understanding and simplifying one-shot architecture search. In International Conference on Machine Learning. PMLR, 550--559.

[3]

Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. 2019. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE International Conference on Computer Vision. 1294--1303.

[4]

Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. 2017. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819 (2017).

[5]

J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Fei-Fei Li. 2009. ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248--255.

[6]

Terrance DeVries and Graham W Taylor. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017).

[7]

Xuanyi Dong and Yi Yang. 2019. One-shot neural architecture search via self-evaluated template network. In Proceedings of the IEEE International Conference on Computer Vision. 3681--3690.

[8]

Xuanyi Dong and Yi Yang. 2019. Searching for a robust neural architecture in four gpu hours. In Proceedings of the IEEE Conference on computer vision and pattern recognition. 1761--1770.

[9]

Xuanyi Dong and Yi Yang. 2020. NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search. In International Conference on Learning Representations. https://openreview.net/forum?id=HJxyZkBKDr

[10]

Agoston E Eiben, James E Smith, et al. 2003. Introduction to evolutionary computing. Vol. 53. Springer.

[11]

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2018. Neural architecture search: A survey. arXiv preprint arXiv:1808.05377 (2018).

[12]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.

[13]

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4700--4708.

[14]

Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika 30, 1/2 (1938), 81--93.

[15]

Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).

[16]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097--1105.

[17]

Liam Li and Ameet Talwalkar. 2020. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence. PMLR, 367--377.

[18]

Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Fei-Fei Li, Alan Yuille, Jonathan Huang, and Kevin Murphy. 2018. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV). 19--34.

Digital Library

[19]

Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. 2018. Hierarchical representations for eficient architecture search. In International Conference on Learning Representations. https://openreview.net/forum?id=BJQRKzbA-

[20]

Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019. DARTS: Differentiable architecture search. In International Conference on Learning Representations. https://openreview.net/forum?id=S1eYHoC5FX

[21]

Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24--26, 2017, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=Skq89Scxx

[22]

Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. 2019. Nsga-net: neural architecture search using multi-objective genetic algorithm. In Proceedings of the Genetic and Evolutionary Computation Conference. 419--427.

Digital Library

[23]

Zhichao Lu, Ian Whalen, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti. 2020. Multi-objective evolutionary design of deep convolutional neural networks for image classification. IEEE Transactions on Evolutionary Computation (2020).

[24]

Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. 2018. Efficient neural architecture search via parameters sharing. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, Stockholmsmässan, Stockholm Sweden, 4095--4104. http://proceedings.mlr.press/v80/pham18a.html

[25]

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. 2019. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 4780--4789.

Digital Library

[26]

Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. 2017. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2902--2911.

Digital Library

[27]

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4510--4520.

[28]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[29]

Nilotpal Sinha and Kuan-Wen Chen. 2021. Evolving neural architecture using one shot model. In Proceedings of the Genetic and Evolutionary Computation Conference. 910--918.

Digital Library

[30]

Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. 2017. A genetic programming approach to designing convolutional neural network architectures. In Proceedings of the genetic and evolutionary computation conference. 497--504.

Digital Library

[31]

Yanan Sun, Handing Wang, Bing Xue, Yaochu Jin, Gary G Yen, and Mengjie Zhang. 2019. Surrogate-assisted evolutionary deep learning using an end-to-end random forest-based performance predictor. IEEE Transactions on Evolutionary Computation 24, 2 (2019), 350--364.

[32]

Yanan Sun, Bing Xue, Mengjie Zhang, and Gary G Yen. 2019. Completely automated CNN architecture design based on blocks. IEEE transactions on neural networks and learning systems 31, 4 (2019), 1242--1254.

[33]

Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Jiancheng Lv. 2020. Automatically designing CNN architectures using the genetic algorithm for image classification. IEEE transactions on cybernetics 50, 9 (2020), 3840--3854.

[34]

Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. In International conference on machine learning. PMLR, 1139--1147.

[35]

Lingxi Xie and Alan Yuille. 2017. Genetic cnn. In Proceedings of the IEEE International Conference on Computer Vision. 1379--1388.

[36]

Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. 2019. SNAS: stochastic neural architecture search. In International Conference on Learning Representations. https://openreview.net/forum?id=rylqooRqK7

[37]

Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. 2020. Understanding and robustifying differentiable architecture search. In International Conference on Learning Representations. https://openreview.net/forum?id=H1gDNyrKDS

[38]

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. 2018. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6848--6856.

[39]

Barret Zoph and Quoc V Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016).

[40]

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. 2018. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8697--8710.

Cited By

Vo ALuong NLi XHandl J(2024)Efficient Multi-Objective Neural Architecture Search via Pareto Dominance-based Novelty SearchProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654064(1146-1155)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638529.3654064
Sinha NEl Rahman Shabayek AKacem ARostami PShneider CAouada D(2024)Hardware Aware Evolutionary Neural Architecture Search using Representation Similarity Metric2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00261(2616-2625)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00261
Sana ASenhaji KNakib A(2024)Multiobjective Based Strategy for Neural Architecture Search for Segmentation Task2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)10.1109/IPDPSW63119.2024.00164(947-955)Online publication date: 27-May-2024
https://doi.org/10.1109/IPDPSW63119.2024.00164
Show More Cited By

Index Terms

Neural architecture search using progressive evolution
1. Computing methodologies
  1. Artificial intelligence

Recommendations

Novelty driven evolutionary neural architecture search
GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Evolutionary algorithms (EA) based neural architecture search (NAS) involves evaluating each architecture by training it from scratch, which is extremely time-consuming. This can be reduced by using a supernet for estimating the fitness of an ...
A Training-free Genetic Neural Architecture Search
ACM ICEA '21: Proceedings of the 2021 ACM International Conference on Intelligent Computing and its Emerging Applications

The so-called neural architecture search (NAS) provides an alternative way to construct a "good neural architecture," which would normally outperform hand-made architectures, for solving complex problems without domain knowledge. However, a critical ...
A Two-Stage Efficient Evolutionary Neural Architecture Search Method for Image Classification
PRICAI 2021: Trends in Artificial Intelligence
Abstract
Deep convolutional neural networks (DCNNs) have achieved promising performance in different computer vision tasks in recent years. Conventionally, deep learning experts are needed to design convolutional neural network’s (CNN’s) architectures when ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference

July 2022

1472 pages

ISBN:9781450392372

DOI:10.1145/3512290

Editor:
Jonathan E. Fieldsend
University of Exeter
,
General Chair:
Markus Wagner
The University of Adelaide

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Data Availability

Supplemental material. https://dl.acm.org/doi/10.1145/3512290.3528707#p1093-sinha-suppl.pdf

Funding Sources

Ministry of Science and Technology of Taiwan

Conference

GECCO '22

Sponsor:

SIGEVO

GECCO '22: Genetic and Evolutionary Computation Conference

July 9 - 13, 2022

Massachusetts, Boston

Acceptance Rates

Overall Acceptance Rate 1,532 of 4,029 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
85
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Vo ALuong NLi XHandl J(2024)Efficient Multi-Objective Neural Architecture Search via Pareto Dominance-based Novelty SearchProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654064(1146-1155)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638529.3654064
Sinha NEl Rahman Shabayek AKacem ARostami PShneider CAouada D(2024)Hardware Aware Evolutionary Neural Architecture Search using Representation Similarity Metric2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00261(2616-2625)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00261
Sana ASenhaji KNakib A(2024)Multiobjective Based Strategy for Neural Architecture Search for Segmentation Task2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)10.1109/IPDPSW63119.2024.00164(947-955)Online publication date: 27-May-2024
https://doi.org/10.1109/IPDPSW63119.2024.00164
Sinha NRostami PEl Rahman Shabayek AKacem AAouada D(2024)Multi-Objective Hardware Aware Neural Architecture Search using Hardware Cost Diversity2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00802(8032-8039)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00802
Liang JCao HLu YSu M(2024)Architecture search of accurate and lightweight CNNs using genetic algorithmGenetic Programming and Evolvable Machines10.1007/s10710-024-09484-425:1Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1007/s10710-024-09484-4
Siddiqui SKyrkou CTheocharides T(2023)True Rank Guided Efficient Neural Architecture Search for End to End Low-Complexity Network DiscoveryComputer Analysis of Images and Patterns10.1007/978-3-031-44237-7_3(25-34)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-44237-7_3

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents