research-article

Towards crossing the reality gap with evolved plastic neurocontrollers

Authors:

Matthew Garratt,

Sreenatha AnavattiAuthors Info & Claims

GECCO '20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference

Pages 130 - 138

https://doi.org/10.1145/3377930.3389843

Published: 26 June 2020 Publication History

Abstract

A critical issue in evolutionary robotics is the transfer of controllers learned in simulation to reality. This is especially the case for small Unmanned Aerial Vehicles (UAVs), as the platforms are highly dynamic and susceptible to breakage. Previous approaches often require simulation models with a high level of accuracy, otherwise significant errors may arise when the well-designed controller is being deployed onto the targeted platform. Here we try to overcome the transfer problem from a different perspective, by designing a spiking neurocontroller which uses synaptic plasticity to cross the reality gap via online adaptation. Through a set of experiments we show that the evolved plastic spiking controller can maintain its functionality by self-adapting to model changes that take place after evolutionary training, and consequently exhibit better performance than its non-plastic counterpart.

References

[1]

L. F. Abbott, Brian DePasquale, and Raoul-Martin Memmesheimer. 2016. Building functional networks of spiking model neurons. Nature Neuroscience 19 (23 Feb 2016), 350 EP -. Perspective.

[2]

A. Alaimo, V. Artale, C. Milazzo, A. Ricciardello, and L. Trefiletti. 2013. Mathematical modeling and control of a hexacopter. In 2013 International Conference on Unmanned Aircraft Systems (ICUAS). 1043--1050.

[3]

Maxence Bouvier, Alexandre Valentian, Thomas Mesquida, Francois Rummens, Marina Reyboz, Elisa Vianello, and Edith Beigne. 2019. Spiking Neural Networks Hardware Implementations and Challenges: A Survey. J. Emerg. Technol. Comput. Syst. 15, 2, Article 22 (April 2019), 35 pages.

Digital Library

[4]

Howard B. Demuth, Mark H. Beale, Orlando De Jess, and Martin T. Hagan. 2014. Neural Network Design (2nd ed.). Martin Hagan, USA. http://hagan.okstate.edu/NNDesign.pdf

[5]

Agoston E Eiben, James E Smith, et al. 2015. Introduction to Evolutionary Computing (second ed.). Springer, Berlin, Heidelberg.

[6]

Matthew Garratt and Sreenatha Anavatti. 2012. Non-linear Control of Heave for an Unmanned Helicopter Using a Neural Network. Journal of Intelligent & Robotic Systems 66, 4 (01 Jun 2012), 495--504.

[7]

Wulfram Gerstner and Werner Kistler. 2002. Spiking Neuron Models: An Introduction. Cambridge University Press, New York, NY, USA. http://icwww.epfl.ch/~gerstner/BUCH.html

[8]

Donald Olding Hebb. 1949. The organization of behavior: A neuropsychological theory. John Wiley & Sons. http://pubman.mpdl.mpg.de/pubman/item/escidoc:2346268/component/escidoc:2346267/Hebb_1949_The_Organization_of_Behavior.pdf

[9]

Nathan V. Hoffer, Calvin Coopmans, Austin M. Jensen, and YangQuan Chen. 2014. A Survey and Categorization of Small Low-Cost Unmanned Aerial Vehicle System Identification. Journal of Intelligent & Robotic Systems 74, 1 (01 Apr 2014), 129--145.

Digital Library

[10]

G. Howard, E. Gale, L. Bull, B. de Lacy Costello, and A. Adamatzky. 2012. Evolution of Plastic Learning in Spiking Networks via Memristive Connections. IEEE Transactions on Evolutionary Computation 16, 5 (Oct 2012), 711--729.

Digital Library

[11]

E. M. Izhikevich. 2003. Simple model of spiking neurons. IEEE Transactions on Neural Networks 14, 6 (Nov 2003), 1569--1572.

Digital Library

[12]

Eugene M. Izhikevich and Niraj S. Desai. 2003. Relating STDP to BCM. Neural Computation 15, 7 (2003), 1511--1523.

Digital Library

[13]

Eric R. Kandel and Robert D. Hawkins. 1992. The Biological Basis of Learning and Individuality. Scientific American 267, 3 (1992), 78--87. http://www.jstor.org/stable/24939215

[14]

Farid Kendoul. 2012. Survey of advances in guidance, navigation, and control of unmanned rotorcraft systems. Journal of Field Robotics 29, 2 (2012), 315--378.

Digital Library

[15]

Wolfgang Maass. 1997. Networks of spiking neurons: The third generation of neural network models. Neural Networks 10, 9 (1997), 1659--1671.

[16]

Zbigniew Michalewicz and Marc Schoenauer. 1996. Evolutionary Algorithms for Constrained Parameter Optimization Problems. Evolutionary Computation 4, 1 (1996), 1--32.

Digital Library

[17]

Andrew Y. Ng, Adam Coates, Mark Diel, Varun Ganapathi, Jamie Schulte, Ben Tse, Eric Berger, and Eric Liang. 2006. Autonomous Inverted Helicopter Flight via Reinforcement Learning. Springer Berlin Heidelberg, Berlin, Heidelberg, 363--372.

[18]

Andrew Y. Ng, H. J. Kim, Michael I. Jordan, and Shankar Sastry. 2004. Autonomous Helicopter Flight via Reinforcement Learning. In Advances in Neural Information Processing Systems 16, S. Thrun, L. K. Saul, and P. B. Schölkopf (Eds.). MIT Press, 799--806. http://papers.nips.cc/paper/2455-autonomous-helicopter-flight-via-reinforcement-learning.pdf

[19]

P. Pounds, R. Mahony, and P. Corke. 2010. Modelling and control of a large quadrotor robot. Control Engineering Practice 18, 7 (2010), 691 -- 699. Special Issue on Aerial Robotics.

[20]

H. Qiu, M. Garratt, D. Howard, and S. Anavatti. 2018. Evolving Spiking Neural Networks for Nonlinear Control Problems. In 2018 IEEE Symposium Series on Computational Intelligence (SSCI). 1367--1373.

[21]

F. Santoso, M. A. Garratt, and S. G. Anavatti. 2017. A self-learning TS-fuzzy system based on the C-means clustering technique for controlling the altitude of a hexacopter unmanned aerial vehicle. In 2017 International Conference on Advanced Mechatronics, Intelligent Manufacture, and Industrial Automation (ICAMIMIA). 46--51.

[22]

A. Soltoggio, P. Durr, C. Mattiussi, and D. Floreano. 2007. Evolving neuromodulatory topologies for reinforcement learning-like problems. In 2007 IEEE Congress on Evolutionary Computation. 2471--2478.

[23]

Kenneth O. Stanley and Risto Miikkulainen. 2002. Evolving Neural Networks Through Augmenting Topologies. Evolutionary Computation 10, 2 (2002), 99--127. http://nn.cs.utexas.edu/?stanley:ec02

Digital Library

[24]

Paul Tonelli and Jean-Baptiste Mouret. 2011. On the Relationships between Synaptic Plasticity and Generative Systems. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation (GECCO '11). Association for Computing Machinery, New York, NY, USA, 1531--1538.

Digital Library

[25]

Joseba Urzelai and Dario Floreano. 2001. Evolution of Adaptive Synapses: Robots with Fast Adaptive Behavior in New Environments. Evolutionary Computation 9, 4 (2001), 495--524.

Digital Library

[26]

Madhavun Candadai Vasu and Eduardo J. Izquierdo. 2017. Evolution and Analysis of Embodied Spiking Neural Networks Reveals Task-specific Clusters of Effective Networks. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '17). ACM, New York, NY, USA, 75--82.

Digital Library

Cited By

Medvet ENadizar GPigozzi FSalvato E(2023)Evolutionary Machine Learning in RoboticsHandbook of Evolutionary Machine Learning10.1007/978-981-99-3814-8_23(657-694)Online publication date: 2-Nov-2023
https://doi.org/10.1007/978-981-99-3814-8_23
Showalter ISchwartz HGivigi S(2021)The Behavioural and Topological Effects of Measurement Noise on Evolutionary Neurocontrollers2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC52423.2021.9659280(2296-2303)Online publication date: 17-Oct-2021
https://doi.org/10.1109/SMC52423.2021.9659280
Qiu HGarratt MHoward DAnavatti S(2020)Evolving Spiking Neurocontrollers for UAVs2020 IEEE Symposium Series on Computational Intelligence (SSCI)10.1109/SSCI47803.2020.9308275(1928-1935)Online publication date: 1-Dec-2020
https://doi.org/10.1109/SSCI47803.2020.9308275

Index Terms

Towards crossing the reality gap with evolved plastic neurocontrollers

Recommendations

Flexible and multistable pattern generation by evolving constrained plastic neurocontrollers

In evolutionary robotics, plastic neural network models proved to be promising for evolving adaptive behaviors. In particular, neurocontrollers incorporating hebbian synapses have been shown to be useful for implementing conflicting sub-behaviors. ...
New biophysical rate-based modeling of long-term plasticity in mean-field neuronal population models
Abstract
The formation of customized neural networks as the basis of brain functions such as receptive field selectivity, learning or memory depends heavily on the long-term plasticity of synaptic connections. However, the current mean-field population ...
Graphical abstract

Display Omitted
Highlights
- Novel rate-based LTP models derived by population density methods.
- Novel mean-field models for simulating neuronal population dynamics with LTP.
- Faithful reproduction of Hebbian plasticity and receptive field development.
- Great ...
Exploration in neo-Hebbian reinforcement learning: Computational approaches to the exploration–exploitation balance with bio-inspired neural networks
Abstract
Recent theoretical and experimental works have connected Hebbian plasticity with the reinforcement learning (RL) paradigm, producing a class of trial-and-error learning in artificial neural networks known as neo-Hebbian plasticity. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference

June 2020

1349 pages

ISBN:9781450371285

DOI:10.1145/3377930

General Chair:
Carlos Artemio Coello Coello
CINVESTAV-IPN

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO '20

Sponsor:

SIGEVO

GECCO '20: Genetic and Evolutionary Computation Conference

July 8 - 12, 2020

Cancún, Mexico

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
125
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Medvet ENadizar GPigozzi FSalvato E(2023)Evolutionary Machine Learning in RoboticsHandbook of Evolutionary Machine Learning10.1007/978-981-99-3814-8_23(657-694)Online publication date: 2-Nov-2023
https://doi.org/10.1007/978-981-99-3814-8_23
Showalter ISchwartz HGivigi S(2021)The Behavioural and Topological Effects of Measurement Noise on Evolutionary Neurocontrollers2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC52423.2021.9659280(2296-2303)Online publication date: 17-Oct-2021
https://doi.org/10.1109/SMC52423.2021.9659280
Qiu HGarratt MHoward DAnavatti S(2020)Evolving Spiking Neurocontrollers for UAVs2020 IEEE Symposium Series on Computational Intelligence (SSCI)10.1109/SSCI47803.2020.9308275(1928-1935)Online publication date: 1-Dec-2020
https://doi.org/10.1109/SSCI47803.2020.9308275

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten