Abstract
Aiming at the auto-design of powerful neural architectures with a requirement of compromising multiple objectives, this paper introduces a novel approach called Pareto-informed Multi-objective Neural Architecture Search (PiMO-NAS), which employs a solution generator influenced by Tchebycheff decomposition to explore the objective space of multi-objective NAS. Our methodology initiates with a transformation of discrete search space into continuous form, followed by iterative solution optimization, in which Gaussian Process (GP) surrogate models are utilized to establish a mapping from decision space to objective space. Subsequently, a solution generator, directed by preference vectors from the objective space, is designed to generate decision vectors to map the weights from the objective space back to the decision space. This solution generator is further optimized based on the gradients derived from the GP models. To ensure diversity in the solution pool, the solution generator synthesizes new candidate solutions guided by preference vectors generated by a well-designed adaptive sampler. In order to verify the performance of the proposed PiMO-NAS, a series of experiments were conducted within two typical NAS search spaces (i.e., the Once-For-All(OFA) and AutoFormer based ones, covering both convolutional neural networks and vision transformers), and more than 30 state-of-the-art NAS methods and models were employed for performance comparisons. Experimental results showcase that our approach can outperform most peers in terms of search time and solution quality, and has fantastic ability to efficiently discover high-performing neural architectures. In the OFA-based search spaces, compared with the MSuNAS, the proposed PiMO-NAS was able to achieve similar performance in two-thirds of the number of iterations, thereby saving about 24% of the search time. In the AutoFormer-based search space, we successfully approached a strong baseline formed by a single-objective evolutionary algorithm with restricted parameter quantities, approximating the entire Pareto front in a comparable timeframe.
This research is supported by the National Natural Science Foundation of China under Grants 62206313 and 62232008.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baker, B., Gupta, O., Naik, N., Raskar, R.: Designing neural network architectures using reinforcement learning (2016). arXiv preprint arXiv:1611.02167
Brock, A., Lim, T., Ritchie, J.M., Weston, N.: Smash: one-shot model architecture search through hypernetworks (2017). arXiv preprint arXiv:1708.05344
Cai, H., Gan, C., Wang, T., Zhang, Z., Han, S.: Once-for-all: Train one network and specialize it for efficient deployment (2019). arXiv preprint arXiv:1908.09791
Cai, H., Zhu, L., Han, S.: Proxylessnas: Direct neural architecture search on target task and hardware (2018). arXiv preprint arXiv:1812.00332
Chen, B., et al.: Glit: neural architecture search for global and local image transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12–21 (2021)
Chen, M., Peng, H., Fu, J., Ling, H.: Autoformer: searching transformers for visual recognition. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 12270–12280 (2021)
Chen, W., Huang, W., Du, X., Song, X., Wang, Z., Zhou, D.: Auto-scaling vision transformers without training (2022). arXiv preprint arXiv:2202.11921
Chu, X., Lu, S., Li, X., Zhang, B.: Mixpath: a unified approach for one-shot neural architecture search. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5972–5981 (2023)
Chu, X., Zhang, B., Xu, R.: Fairnas: rethinking evaluation fairness of weight sharing neural architecture search. In: Proceedings of the IEEE/CVF International Conference on computer vision, pp. 12239–12248 (2021)
Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling (2014). arXiv preprint arXiv:1412.3555
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. IEEE (2009)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale (2021)
d’Ascoli, S., et al.: Improving vision transformers with soft convolutional inductive biases. In: International conference on machine learning, pp. 2286–2296. PMLR (2021)
Goodfellow, I., Bengio, Y., Courville, A.: Deep learning. MIT press (2016)
Guo, Y., et al.: Pareto-aware neural architecture generation for diverse computational budgets. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2247–2257 (2023)
Guo, Z., et al.: Single path one-shot neural architecture search with uniform sampling. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pp. 544–560. Springer (2020)
Han, K., et al.: A survey on vision transformer. IEEE Trans. Pattern Anal. Mach. Intell. 45(1), 87–110 (2022)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)
Howard, A., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 1314–1324 (2019)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708 (2017)
Kandasamy, K., Neiswanger, W., Schneider, J., Poczos, B., Xing, E.P.: Neural architecture search with Bayesian optimisation and optimal transport. Adv. Neural Inf. Process. Syst. 31 (2018)
Knowles, J.: ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Trans. Evol. Comput. 10(1), 50–66 (2006)
Lin, X., Yang, Z., Zhang, X., Zhang, Q.: Pareto set learning for expensive multi-objective optimization. Adv. Neural. Inf. Process. Syst. 35, 19231–19247 (2022)
Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search (2018). arXiv preprint arXiv:1806.09055
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022 (2021)
Lu, Z., Deb, K., Goodman, E., Banzhaf, W., Boddeti, V.N.: Nsganetv2: evolutionary multi-objective surrogate-assisted neural architecture search. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pp. 35–51. Springer (2020)
Lu, Z., et al.: Nsga-net: neural architecture search using multi-objective genetic algorithm. In: Proceedings of the genetic and evolutionary computation conference, pp. 419–427 (2019)
Luo, G., Li, H., Chen, Z., Zhou, Y.: Supplementary materials of "pareto-informed multi-objective neural architecture search" (2024). https://github.com/SYSU22214881/PiMO-NAS
Ma, N., Zhang, X., Zheng, H.T., Sun, J.: Shufflenet v2: practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV), pp. 116–131 (2018)
Peng, Y., Song, A., Ciesielski, V., Fayek, H.M., Chang, X.: PRE-NAS: predictor-assisted evolutionary neural architecture search. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1066–1074 (2022)
Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: International conference on machine learning, pp. 4095–4104. PMLR (2018)
Real, E., et al.: Large-scale evolution of image classifiers. In: International conference on machine learning, pp. 2902–2911. PMLR (2017)
Su, X., et al.: Vitas: Vision transformer architecture search (2021). arXiv e-prints pp. arXiv–2106
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826 (2016)
Tan, M., et al.: Mnasnet: platform-aware neural architecture search for mobile. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2820–2828 (2019)
Termritthikun, C., Jamtsho, Y., Ieamsaard, J., Muneesawang, P., Lee, I.: EEEA-Net: an early exit evolutionary neural architecture search. Eng. Appl. Artif. Intell. 104, 104397 (2021)
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: International conference on machine learning, pp. 10347–10357. PMLR (2021)
Wang, W., et al.: Pyramid vision transformer: a versatile backbone for dense prediction without convolutions. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 568–578 (2021)
Wang, W., Zhang, X., Cui, H., Yin, H., Zhang, Y.: FP-DARTS: fast parallel differentiable neural architecture search for image classification. Pattern Recogn. 136, 109193 (2023)
Wang, W., et al.: Crossformer: a versatile vision transformer hinging on cross-scale attention. In: International Conference on Learning Representations (2021)
Xue, Y., Chen, C., Słowik, A.: Neural architecture search based on a multi-objective evolutionary algorithm with probability stack. IEEE Trans. Evol. Comput. 27(4), 778–786 (2023)
Yu, W., et al.: Metaformer is actually what you need for vision. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10819–10829 (2022)
Yuan, L., et al.: Tokens-to-token vit: training vision transformers from scratch on imagenet. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 558–567 (2021)
Zhang, J., et al.: Analogous to evolutionary algorithm: designing a unified sequence model. Adv. Neural. Inf. Process. Syst. 34, 26674–26688 (2021)
Zhou, Q., et al.: Training-free transformer architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10894–10903 (2022)
Zong, Z., Cao, Q., Leng, B.: RCNet: reverse feature pyramid and cross-scale shift network for object detection. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 5637–5645 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interest
The author declares no competing interests as defined by Springer Nature, or other interests that might be perceived to influence the results and/or discussion reported in this manuscript.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Luo, G., Li, H., Chen, Z., Zhou, Y. (2024). Pareto-Informed Multi-objective Neural Architecture Search. In: Affenzeller, M., et al. Parallel Problem Solving from Nature – PPSN XVIII. PPSN 2024. Lecture Notes in Computer Science, vol 15150. Springer, Cham. https://doi.org/10.1007/978-3-031-70071-2_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-70071-2_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70070-5
Online ISBN: 978-3-031-70071-2
eBook Packages: Computer ScienceComputer Science (R0)