research-article

Memetic evolution of deep neural networks

Authors:

Pablo Ribalta Lorenzo,

Jakub NalepaAuthors Info & Claims

GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference

Pages 505 - 512

https://doi.org/10.1145/3205455.3205631

Published: 02 July 2018 Publication History

Abstract

Deep neural networks (DNNs) have proven to be effective at solving challenging problems, but their success relies on finding a good architecture to fit the task. Designing a DNN requires expert knowledge and a lot of trial and error, especially as the difficulty of the problem grows. This paper proposes a fully automatic method with the goal of optimizing DNN topologies through memetic evolution. By recasting the mutation step as a series of progressively refined educated local-search moves, this method achieves results comparable to best human designs. Our extensive experimental study showed that the proposed memetic algorithm supports building a real-world solution for segmenting medical images, it exhibits very promising results over a challenging CIFAR-10 benchmark, and works very fast. Given the ever growing availability of data, our memetic algorithm is a very promising avenue for hands-free DNN architecture design to tackle emerging classification tasks.

References

[1]

Sivaram Ambikasaran, Daniel Foreman-Mackey, Leslie Greengard, David W. Hogg, and Michael O'Neil. 2016. Fast Direct Methods for Gaussian Processes. IEEE Trans. Pattern Anal Mach. Intell. 38, 2 (2016), 252--265.

Digital Library

[2]

V. Badrinarayanan, A. Kendall, and R. Cipolla. 2017. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39, 12 (Dec 2017), 2481--2495.

[3]

Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. 2016. Designing Neural Network Architectures using Reinforcement Learning. CoRR abs/1611.02167 (2016), 1--18. arXiv:1611.02167

[4]

James Bergstra and Yoshua Bengio. 2012. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research 13 (2012), 281--305.

Digital Library

[5]

François Chollet et al. 2015. Keras. https://github.com/fchollet/keras. (2015).

[6]

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). CoRR abs/1511.07289 (2015), 1--14.

[7]

Omid E. David and Iddo Greental. 2014. Genetic Algorithms for Evolving Deep Neural Networks. In Proc. GECCO Companion. ACM, USA, 1451--1452.

Digital Library

[8]

Travis Desell. 2017. Large Scale Evolution of Convolutional Neural Networks Using Volunteer Computing. In Proc. GECCO. ACM, USA, 127--128.

Digital Library

[9]

Travis Desell, Sophine Clachar, James Higgins, and Brandon Wild. 2015. Evolving Deep Recurrent Neural Networks Using Ant Colony Optimization. In Proc. EvoCOP, Gabriela Ochoa and Francisco Chicano (Eds.). Springer, Cham, 86--98.

[10]

Chrisantha Fernando, Dylan Banarse, Malcolm Reynolds, Frederic Besse, David Pfau, Max Jaderberg, Marc Lanctot, and Daan Wierstra. 2016. Convolution by Evolution: Differentiable Pattern Producing Networks. In Proc. GECCO. ACM, USA, 109--116.

Digital Library

[11]

Faustino Gomez, Juergen Schmidhuber, and Risto Miikkulainen. 2008. Accelerated Neural Evolution through Cooperatively Coevolved Synapses. Journal of Machine Learning Research (2008), 937--965.

Digital Library

[12]

Delowar Hossain, Genci Capi, and Mitsuru Jindai. 2017. Evolution of Deep Belief Neural Network Parameters for Robot Object Recognition and Grasping. Procedia Computer Science 105 (2017), 153 -- 158.

Digital Library

[13]

Ilija Ilievski, Taimoor Akhtar, Jiashi Feng, and Christine Annette Shoemaker. 2017. Efficient Hyperparameter Optimization for Deep Learning Algorithms Using Deterministic RBF Surrogates. In Proc. AAAI. 822--829.

[14]

Shauharda Khadka, Jen Jen Chung, and Kagan Tumer. 2017. Evolving Memory-augmented Neural Architecture for Deep Memory Problems. In Proc. GECCO. ACM, USA, 441--448.

Digital Library

[15]

Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014), 1--15.

[16]

Jan Koutník, Juergen Schmidhuber, and Faustino Gomez. 2014. Evolving Deep Unsupervised Convolutional Networks for Vision-based Reinforcement Learning. In Proc. GECCO. ACM, USA, 541--548.

Digital Library

[17]

Pablo Ribalta Lorenzo, Jakub Nalepa, Michal Kawulok, Luciano Sanchez Ramos, and José Ranilla Pastor. 2017. Particle Swarm Optimization for Hyper-parameter Selection in Deep Neural Networks. In Proc. GECCO. ACM, USA, 481--488.

Digital Library

[18]

A. Martín, F. Fuentes-Hurtado, V. Naranjo, and D. Camacho. 2017. Evolving Deep Neural Networks architectures for Android malware classification. In Proc. IEEE CEC. 1659--1666.

[19]

Risto Miikkulainen, Jason Zhi Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. 2017. Evolving Deep Neural Networks. CoRR abs/1703.00548 (2017), 1--8.

[20]

Jakub Nalepa and Michal Kawulok. 2016. Adaptive memetic algorithm enhanced with data geometry analysis to select training data for SVMs. Neurocomputing 185 (2016), 113 -- 132.

Digital Library

[21]

Carl Edward Rasmussen and Christopher K. I. Williams. 2005. Gaussian Processes for Machine Learning. The MIT Press. 1--266 pages.

Digital Library

[22]

Esteban Real, Sherry Moore, Andrew Seile, Saurabh Saxena, Yutaka Leon Suematsu, Quoc V. Le, and Alex Kurakin. 2017. Large-Scale Evolution of Image Classifiers. CoRR abs/1703.01041 (2017), 1--18.

Digital Library

[23]

Khalid Salama and Ashraf M. Abdelbar. 2014. A Novel Ant Colony Algorithm for Building Neural Network Topologies. In Proc. ANTS. Springer, 1--12.

[24]

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms. In Proc. NIPS. Curran, 2951--2959.

Digital Library

[25]

Mohammadreza Soltaninejad and et al. 2017. Automated brain tumour detection and segmentation using superpixel-based extremely randomized trees in FLAIR MRI. Int. J. of Computer Assisted Radiol. and Surgery 12, 2 (2017), 183--203.

[26]

Kenneth O. Stanley, David B. D'Ambrosio, and Jason Gauci. 2009. A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks. Artificial Life 15, 2 (2009), 185--212.

Digital Library

[27]

Kenneth O. Stanley and Risto Miikkulainen. 2002. Evolving Neural Networks Through Augmenting Topologies. Evol. Computation 10, 2 (2002), 99--127.

Digital Library

[28]

Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. 2017. Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning. CoRR abs/1712.06567 (2017), 1--15.

[29]

Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. 2017. A Genetic Programming Approach to Designing Convolutional Neural Network Architectures. In Proc. GECCO. ACM, USA, 497--504.

Digital Library

[30]

Yanan Sun, Bing Xue, and Mengjie Zhang. 2017. Evolving Deep Convolutional Neural Networks for Image Classification. CoRR abs/1710.10741 (2017), 1--14.

[31]

Yanan Sun, Gary G. Yen, and Zhang Yi. 2017. Evolving Unsupervised Deep Neural Nets for Learning Meaningful Representations. CoRR abs/1712.05043 (2017), 1--23.

[32]

Lingxi Xie and Alan L. Yuille. 2017. Genetic CNN. In Proc. ICCV. 1388--1397.

[33]

Steven R. Young, Derek C. Rose, Travis Johnston, William T. Heller, Thomas P. Karnowski, Thomas E. Potok, Robert M. Patton, Gabriel N. Perdue, and Jonathan Miller. 2017. Evolving Deep Networks Using HPC. In Proc. MLHPC@SC. 7:1--7:7.

Digital Library

[34]

Fisher Yu and Vladlen Koltun. 2015. Multi-Scale Context Aggregation by Dilated Convolutions. CoRR abs/1511.07122 (2015), 1--13.

[35]

Barret Zoph and Quoc V. Le. 2016. Neural Architecture Search with Reinforcement Learning. CoRR abs/1611.01578 (2016), 1--16.

Cited By

Zhang MLei ZLiu LMa KShang RJiao L(2025)Efficient evolutionary multi-scale spectral-spatial attention fusion network for hyperspectral image classificationExpert Systems with Applications10.1016/j.eswa.2024.125672262(125672)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125672
Liang JZhu KLi YLi YGong Y(2024)Multi-Objective Evolutionary Neural Architecture Search with Weight-Sharing SupernetApplied Sciences10.3390/app1414614314:14(6143)Online publication date: 15-Jul-2024
https://doi.org/10.3390/app14146143
AKGUL AKARACA YPALA MÇIMEN MBOZ AYILDIZ M(2024)CHAOS THEORY, ADVANCED METAHEURISTIC ALGORITHMS AND THEIR NEWFANGLED DEEP LEARNING ARCHITECTURE OPTIMIZATION APPLICATIONS: A REVIEWFractals10.1142/S0218348X2430001032:03Online publication date: 5-Apr-2024
https://doi.org/10.1142/S0218348X24300010
Show More Cited By

Index Terms

Memetic evolution of deep neural networks
1. Computing methodologies
  1. Artificial intelligence
  2. Machine learning
    1. Machine learning approaches
      1. Bio-inspired approaches
      2. Neural networks

Recommendations

Memetic search for the quadratic assignment problem

We present a memetic algorithm (called BMA) for the well-known QAP.BMA integrates BLS within the population-based evolutionary computing framework.BMA is able to attain the best-known results for 133 out of 135 QAP benchmark instances.We provide ...
Distributed memetic differential evolution with the synergy of Lamarckian and Baldwinian learning

As a population-based optimizer, the differential evolution (DE) algorithm has a very good reputation for its competence in global search and numerical robustness. In view of the fact that each member of the population is evaluated individually, DE can ...
Memetic algorithm with double mutation for numerical optimization
IScIDE'11: Proceedings of the Second Sino-foreign-interchange conference on Intelligent Science and Intelligent Data Engineering

A memetic algorithm with double mutation operators is proposed, termed as MADM. In this paper, the algorithm combines two meta-learning systems to improve the ability of global and local exploration. The double mutation operators in our algorithms guide ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference

July 2018

1578 pages

ISBN:9781450356183

DOI:10.1145/3205455

Editor:
Hernan Aguirre
Shinshu University
,
General Chair:
Keiki Takadama
The University of Electro-Communications

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 July 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO '18

Sponsor:

SIGEVO

GECCO '18: Genetic and Evolutionary Computation Conference

July 15 - 19, 2018

Kyoto, Japan

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
649
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)4

Reflects downloads up to 14 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang MLei ZLiu LMa KShang RJiao L(2025)Efficient evolutionary multi-scale spectral-spatial attention fusion network for hyperspectral image classificationExpert Systems with Applications10.1016/j.eswa.2024.125672262(125672)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125672
Liang JZhu KLi YLi YGong Y(2024)Multi-Objective Evolutionary Neural Architecture Search with Weight-Sharing SupernetApplied Sciences10.3390/app1414614314:14(6143)Online publication date: 15-Jul-2024
https://doi.org/10.3390/app14146143
AKGUL AKARACA YPALA MÇIMEN MBOZ AYILDIZ M(2024)CHAOS THEORY, ADVANCED METAHEURISTIC ALGORITHMS AND THEIR NEWFANGLED DEEP LEARNING ARCHITECTURE OPTIMIZATION APPLICATIONS: A REVIEWFractals10.1142/S0218348X2430001032:03Online publication date: 5-Apr-2024
https://doi.org/10.1142/S0218348X24300010
Wu MLin HTsai C(2024)A Training-Free Neural Architecture Search Algorithm Based on Search EconomicsIEEE Transactions on Evolutionary Computation10.1109/TEVC.2023.326453328:2(445-459)Online publication date: Apr-2024
https://doi.org/10.1109/TEVC.2023.3264533
Zhao Z(2024)Automatic search of machine learning models based on intelligent computing2024 IEEE 4th International Conference on Electronic Technology, Communication and Information (ICETCI)10.1109/ICETCI61221.2024.10594313(319-323)Online publication date: 24-May-2024
https://doi.org/10.1109/ICETCI61221.2024.10594313
Wu MTsai C(2024)Training-free neural architecture search: A reviewICT Express10.1016/j.icte.2023.11.00110:1(213-231)Online publication date: Feb-2024
https://doi.org/10.1016/j.icte.2023.11.001
Grabowski BZiaja MKawulok MBosowski PLongépé NLe Saux BNalepa J(2024)Squeezing adaptive deep learning methods with knowledge distillation for on-board cloud detectionEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.107835132:COnline publication date: 18-Jul-2024
https://dl.acm.org/doi/10.1016/j.engappai.2023.107835
Xue YZha JWahib MOuyang TWang X(2024)Neural architecture search via similarity adaptive guidanceApplied Soft Computing10.1016/j.asoc.2024.111821162(111821)Online publication date: Sep-2024
https://doi.org/10.1016/j.asoc.2024.111821
Baratchi MWang CLimmer Svan Rijn JHoos HBäck TOlhofer M(2024)Automated machine learning: past, present and futureArtificial Intelligence Review10.1007/s10462-024-10726-157:5Online publication date: 18-Apr-2024
https://doi.org/10.1007/s10462-024-10726-1
Velazco-Muñoz JAcosta-Mesa HMezura-Montes E(2024)Reducing Parameters by Neuroevolution in CNN for Steering Angle EstimationPattern Recognition10.1007/978-3-031-62836-8_35(377-386)Online publication date: 19-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-62836-8_35
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten