Genetic crossover in the evolution of time-dependent neural networks
Pages 885 - 891
Abstract
Neural networks with temporal characteristics such as asynchronous spiking have made much progress towards biologically plausible artificial intelligence. However, genetic approaches for evolving network structures in this area are still relatively unexplored. In this paper, we examine a specific variant of time-dependent spiking neural networks (NN) in which the spatial and temporal relationships between neurons affect output. First, we built and customized a standard NN implementation to more closely model the time-delay characteristics of biological neurons. Next, we tested this with simulated tasks such as food foraging and image recognition, demonstrating success in multiple domains. We then developed a genetic representation for the network that allows for both scalable network size and compatibility with genetic crossover operations. Finally, we analyzed the effects of genetic crossover algorithms compared to random mutations on the food foraging task. Results showed that crossover operations based on node usage converge on a local maximum more quickly than random mutations, but suffer from genetic defects that reduce overall population performance.
Supplementary Material
MP4 File (p885-orlosky_suppl.mp4)
- Download
- 9.50 MB
References
[1]
Peter J Angeline, Gregory M Saunders, and Jordan B Pollack. 1994. An evolutionary algorithm that constructs recurrent neural networks. IEEE transactions on Neural Networks 5, 1 (1994), 54--65.
[2]
Marco Baioletti, Alfredo Milani, and Valentino Santucci. 2018. Algebraic crossover operators for permutations. In 2018 IEEE Congress on Evolutionary Computation (CEC). IEEE, 1--8.
[3]
Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 1499--1510.
[4]
Egbert JW Boers and Herman Kuiper. 1992. Biological metaphors and the design of modular artificial neural networks. (1992).
[5]
Nicolas Brunel. 2000. Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons. Journal of computational neuroscience 8, 3 (2000), 183--208.
[6]
Erick Cantú-Paz and Chandrika Kamath. 2005. An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 35, 5 (2005), 915--927.
[7]
Amanda Clare, Jacqueline W Daykin, Thomas Mills, and Christine Zarges. 2019. Evolutionary search techniques for the Lyndon factorization of biosequences. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 1543--1550.
[8]
Li Deng. 2012. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine 29, 6 (2012), 141--142.
[9]
Jozef Fekiač, Ivan Zelinka, and Juan C Burguillo. 2011. A review of methods for encoding neural network topologies in evolutionary computation. In Proceedings of 25th European Conference on Modeling and Simulation ECMS. 410--416.
[10]
Adam Gaier and David Ha. 2019. Weight agnostic neural networks. In Advances in Neural Information Processing Systems. 5365--5379.
[11]
Padraig Gleeson, Matteo Cantarelli, Boris Marin, Adrian Quintana, Matt Earnshaw, Sadra Sadeh, Eugenio Piasini, Justas Birgiolas, Robert C Cannon, N Alex Cayco-Gajic, et al. 2019. Open Source Brain: a collaborative resource for visualizing, analyzing, simulating, and developing standardized models of neurons and circuits. Neuron 103, 3 (2019), 395--411.
[12]
Robert Gütig and Haim Sompolinsky. 2006. The tempotron: a neuron that learns spike timing-based decisions. Nature neuroscience 9, 3 (2006), 420--428.
[13]
Geoffrey E Hinton and Steven J Nowlan. 1996. How learning can guide evolution. Adaptive individuals in evolving populations: models and algorithms 26 (1996), 447--454.
[14]
Kurt Hornik. 1991. Approximation capabilities of multilayer feedforward networks. Neural networks 4, 2 (1991), 251--257.
[15]
Philipp Koehn. 1994. Combining Genetic Algorithms and Neural Networks: The Encoding Problem. (1994).
[16]
Taras Kowaliw, Nicolas Bredeche, Sylvain Chevallier, and René Doursat. 2014. Artificial neurogenesis: An introduction and selective review. In Growing Adaptive Machines. Springer, 1--60.
[17]
Nikolaus Kriegeskorte. 2015. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science 1 (2015), 417--446.
[18]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436--444.
[19]
Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Advances in neural information processing systems. 2802--2810.
[20]
Toshifumi Minemoto, Teijiro Isokawa, Haruhiko Nishimura, and Nobuyuki Matsui. 2017. Feed forward neural network with random quaternionic neurons. Signal Processing 136 (2017), 59--68.
[21]
Haruo Mizutani, Michihiko Ueno, Naoya Arakawa, and Hiroshi Yamakawa. 2018. Whole brain connectomic architecture to develop general artificial intelligence. Procedia computer science 123 (2018), 308--313.
[22]
Saber Moradi and Giacomo Indiveri. 2013. An event-based neural network architecture with an asynchronous programmable synaptic memory. IEEE transactions on biomedical circuits and systems 8, 1 (2013), 98--107.
[23]
Markdy Y Orong, Ariel M Sison, and Ruji P Medina. 2018. A new crossover mechanism for genetic algorithm with rank-based selection method. In 2018 5th International Conference on Business and Industrial Research (ICBIR). IEEE, 83--88.
[24]
Rodrigo Perin, Thomas K Berger, and Henry Markram. 2011. A synaptic organizing principle for cortical neuronal groups. Proceedings of the National Academy of Sciences 108, 13 (2011), 5419--5424.
[25]
Marc'Aurelio Ranzato, Fu Jie Huang, Y-Lan Boureau, and Yann LeCun. 2007. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In 2007 IEEE conference on computer vision and pattern recognition. IEEE, 1--8.
[26]
Cristina Savin and Sophie Deneve. 2014. Spatio-temporal representations of uncertainty in spiking neural networks. In Advances in Neural Information Processing Systems. 2024--2032.
[27]
Wouter F Schmidt, Martin A Kraaijveld, Robert PW Duin, et al. 1992. Feed forward neural networks with random weights. In International Conference on Pattern Recognition. IEEE COMPUTER SOCIETY PRESS, 1--1.
[28]
H Sebastian Seung. 2003. Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40, 6 (2003), 1063--1073.
[29]
Wolf Singer. 1999. Neuronal synchrony: a versatile code for the definition of relations? Neuron 24, 1 (1999), 49--65.
[30]
Spencer L Smith, Ikuko T Smith, Tiago Branco, and Michael Häusser. 2013. Dendritic spikes enhance stimulus selectivity in cortical neurons in vivo. Nature 503, 7474 (2013), 115--120.
[31]
Tao Song, Linqiang Pan, and Gheorghe Păun. 2013. Asynchronous spiking neural P systems with local synchronization. Information Sciences 219 (2013), 197--207.
[32]
Kenneth O Stanley and Risto Miikkulainen. 2002. Evolving neural networks through augmenting topologies. Evolutionary computation 10, 2 (2002), 99--127.
[33]
Kenneth O Stanley and Risto Miikkulainen. 2003. A taxonomy for artificial embryogeny. Artificial Life 9, 2 (2003), 93--130.
[34]
Christof Teuscher. 2012. Turing's connectionism: an investigation of neural network architectures. Springer Science & Business Media.
[35]
Anant J Umbarkar and Pranali D Sheth. 2015. Crossover operators in genetic algorithms: a review. ICTACT journal on soft computing 6, 1 (2015).
[36]
Gou-Jen Wang and Chih-Cheng Chen. 1996. A fast multilayer neural-network training algorithm based on the layer-by-layer optimizing procedures. IEEE Transactions on Neural Networks 7, 3 (1996), 768--775.
[37]
Zhigen Xu, Yusong Yan, and Jim X Chen. 2005. Opengl programming in java. Computing in Science & Engineering 7, 1 (2005), 51--55.
[38]
Xin Yao. 1999. Evolving artificial neural networks. Proc. IEEE 87, 9 (1999), 1423--1447.
[39]
Davide Zambrano and Sander M Bohte. 2016. Fast and efficient asynchronous neural computation with adapting spiking neural networks. arXiv preprint arXiv:1609.02053 (2016).
Recommendations
Neural network crossover in genetic algorithms using genetic programming
AbstractThe use of genetic algorithms (GAs) to evolve neural network (NN) weights has risen in popularity in recent years, particularly when used together with gradient descent as a mutation operator. However, crossover operators are often omitted from ...
Genetic algorithms for task scheduling problem
The scheduling and mapping of the precedence-constrained task graph to processors is considered to be the most crucial NP-complete problem in parallel and distributed computing systems. Several genetic algorithms have been developed to solve this ...
Improved Izhikevich neurons for spiking neural networks
Spiking neural networks constitute a modern neural network paradigm that overlaps machine learning and computational neurosciences. Spiking neural networks use neuron models that possess a great degree of biological realism. The most realistic model of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2021
1219 pages
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 June 2021
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
GECCO '21
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,669 of 4,410 submissions, 38%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 261Total Downloads
- Downloads (Last 12 months)85
- Downloads (Last 6 weeks)5
Reflects downloads up to 18 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in