research-article

Evolving deep unsupervised convolutional networks for vision-based reinforcement learning

Authors:

Juergen Schmidhuber,

Faustino GomezAuthors Info & Claims

GECCO '14: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation

Pages 541 - 548

https://doi.org/10.1145/2576768.2598358

Published: 12 July 2014 Publication History

Abstract

Dealing with high-dimensional input spaces, like visual input, is a challenging task for reinforcement learning (RL). Neuroevolution (NE), used for continuous RL problems, has to either reduce the problem dimensionality by (1) compressing the representation of the neural network controllers or (2) employing a pre-processor (compressor) that transforms the high-dimensional raw inputs into low-dimensional features. In this paper, we are able to evolve extremely small recurrent neural network (RNN) controllers for a task that previously required networks with over a million weights. The high-dimensional visual input, which the controller would normally receive, is first transformed into a compact feature vector through a deep, max-pooling convolutional neural network (MPCNN). Both the MPCNN preprocessor and the RNN controller are evolved successfully to control a car in the TORCS racing simulator using only visual input. This is the first use of deep learning in the context evolutionary RL.

References

[1]

D. C. Ciresan, U. Meier, L. M. Gambardella, and J. Schmidhuber. Deep big simple neural nets for handwritten digit recognition. Neural Computation, 22(12):3207--3220, 2010.

Digital Library

[2]

D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber. Flexible, high performance convolutional neural networks for image classification. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 1237--1242, 2011.

Digital Library

[3]

G. Cuccu, M. Luciw, J. Schmidhuber, and F. Gomez. Intrinsically motivated evolutionary search for vision-based reinforcement learning. In Proceedings of the IEEE Conference on Development and Learning, and Epigenetic Robotics, 2011.

[4]

D. B. D'Ambrosio and K. O. Stanley. A novel generative encoding for exploiting neural network sensor and output geometry. In Proceedings of the 9th Conference on Genetic and Evolutionary Computation, (GECCO), pages 974--981, New York, NY, USA, 2007. ACM.

Digital Library

[5]

F. Fernández and D. Borrajo. Two steps reinforcement learning. International Journal of Intelligent Systems, 23(2):213--245, 2008.

Digital Library

[6]

K. Fukushima. Neocognitron: A self-organizing neural network for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193--202, 1980.

[7]

J. Gauci and K. Stanley. Generating large-scale neural networks through discovering geometric regularities. In Proceedings of the Conference on Genetic and Evolutionary Computation, (GECCO), pages 997--1004, New York, NY, USA, 2007. ACM.

Digital Library

[8]

L. Gisslén, M. Luciw, V. Graziano, and J. Schmidhuber. Sequential Constant Size Compressors and Reinforcement Learning. In Proceedings of the Fourth Conference on Artificial General Intelligence, 2011.

Digital Library

[9]

F. Gomez, J. Schmidhuber, and R. Miikkulainen. Accelerated neural evolution through cooperatively coevolved synapses. Journal of Machine Learning Research, 9(May):937--965, 2008.

Digital Library

[10]

F. Gruau. Cellular encoding of genetic neural networks. Technical Report RR-92--21, Ecole Normale Superieure de Lyon, Institut IMAG, Lyon, France, 1992.

[11]

S. R. Jodogne and J. H. Piater. Closed-loop learning of visual control policies. Journal of Artificial Intelligence Research, 28:349--391, 2007.

Digital Library

[12]

H. Kitano. Designing neural networks using genetic algorithms with graph generation system. Complex Systems, 4:461--476, 1990.

[13]

J. Koutník, G. Cuccu, J. Schmidhuber, and F. Gomez. Evolving large-scale neural networks for vision-based reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), Amsterdam, 2013.

Digital Library

[14]

J. Koutník, F. Gomez, and J. Schmidhuber. Evolving neural networks in compressed weight space. In Proceedings of the Conference on Genetic and Evolutionary Computation (GECCO-10), 2010.

Digital Library

[15]

S. Lange and M. Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In International Joint Conference on Neural Networks (IJCNN 2010), Barcelona, Spain, 2010.

[16]

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278--2324, November 1998.

[17]

R. Legenstein, N. Wilbert, and L. Wiskott. Reinforcement Learning on Slow Features of High-Dimensional Input Streams. PLoS Computational Biology, 6(8), 2010.

[18]

D. Pierce and B. Kuipers. Map learning with uninterpreted sensors and effectors. Artificial Intelligence, 92:169--229, 1997.

Digital Library

[19]

M. Riedmiller, S. Lange, and A. Voigtlaender. Autonomous reinforcement learning on raw visual input data in a real world application. In Proceedings of the International Joint Conference on Neural Networks (IJCNN), pages 1--8, Brisbane, Australia, 2012.

[20]

D. Scherer, A. Müller, and S. Behnke. Evaluation of pooling operations in convolutional architectures for object recognition. In Proceedings of the International Conference on Artificial Neural Networks, ICANN, 2010.

Digital Library

[21]

J. Schmidhuber. Discovering neural nets with low Kolmogorov complexity and high generalization capability. Neural Networks, 10(5):857--873, 1997.

Digital Library

[22]

R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12 (NIPS), pages 1057--1063, 1999.

Digital Library

[23]

G. Tesauro. Practical issues in temporal difference learning. In D. S. Lippman, J. E. Moody, and D. S. Touretzky, editors, Advances in Neural Information Processing Systems 4 (NIPS), pages 259--266. Morgan Kaufmann, 1992.

[24]

X. Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423--1447, 1999.

Cited By

Zhang HTang LSong XXu T(2024)NSMD-NAS: Retinal Image Segmentation with Neural Architecture Search and Non-Subsampled Multiscale Decomposition2024 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC60901.2024.10612005(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/CEC60901.2024.10612005
Naulia PWatada JAziz I(2024)A Mathematically Inspired Meta-Heuristic Approach to Parameter (Weight) Optimization of Deep Convolution Neural NetworkIEEE Access10.1109/ACCESS.2024.340968912(83299-83322)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3409689
Bai HCheng RJin Y(2023)Evolutionary Reinforcement Learning: A SurveyIntelligent Computing10.34133/icomputing.00252Online publication date: 10-May-2023
https://doi.org/10.34133/icomputing.0025
Show More Cited By

Index Terms

Evolving deep unsupervised convolutional networks for vision-based reinforcement learning
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Evolving large-scale neural networks for vision-based reinforcement learning
GECCO '13: Proceedings of the 15th annual conference on Genetic and evolutionary computation

The idea of using evolutionary computation to train artificial neural networks, or neuroevolution (NE), for reinforcement learning (RL) tasks has now been around for over 20 years. However, as RL tasks become more challenging, the networks required ...
Deep learning in neural networks

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. ...
Deep reinforcement learning in computer vision: a comprehensive survey
Abstract
Deep reinforcement learning augments the reinforcement learning framework and utilizes the powerful representation of deep neural networks. Recent works have demonstrated the remarkable successes of deep reinforcement learning in various domains ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '14: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation

July 2014

1478 pages

ISBN:9781450326629

DOI:10.1145/2576768

Editor-in-chief:
Christian Igel
Ruhr University of Bochum, University of Copenhagen
,
General Chair:
Dirk V. Arnold
Dalhousie University

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Swiss National Science Foundation

Conference

GECCO '14

Sponsor:

SIGEVO

GECCO '14: Genetic and Evolutionary Computation Conference

July 12 - 16, 2014

BC, Vancouver, Canada

Acceptance Rates

GECCO '14 Paper Acceptance Rate 180 of 544 submissions, 33%;

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

74
Total Citations
View Citations
1,167
Total Downloads

Downloads (Last 12 months)59
Downloads (Last 6 weeks)3

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang HTang LSong XXu T(2024)NSMD-NAS: Retinal Image Segmentation with Neural Architecture Search and Non-Subsampled Multiscale Decomposition2024 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC60901.2024.10612005(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/CEC60901.2024.10612005
Naulia PWatada JAziz I(2024)A Mathematically Inspired Meta-Heuristic Approach to Parameter (Weight) Optimization of Deep Convolution Neural NetworkIEEE Access10.1109/ACCESS.2024.340968912(83299-83322)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3409689
Bai HCheng RJin Y(2023)Evolutionary Reinforcement Learning: A SurveyIntelligent Computing10.34133/icomputing.00252Online publication date: 10-May-2023
https://doi.org/10.34133/icomputing.0025
Praczyk T(2023)Emerging Modularity During the Evolution of Neural NetworksJournal of Artificial Intelligence and Soft Computing Research10.2478/jaiscr-2023-001013:2(107-126)Online publication date: 11-Mar-2023
https://doi.org/10.2478/jaiscr-2023-0010
XUE XHUANG YZHANG Z(2023)Deep Reinforcement Learning Based Ontology Meta-Matching TechniqueIEICE Transactions on Information and Systems10.1587/transinf.2022DLP0050E106.D:5(635-643)Online publication date: 1-May-2023
https://doi.org/10.1587/transinf.2022DLP0050
Li NMa LYu GXue BZhang MJin Y(2023)Survey on Evolutionary Deep Learning: Principles, Algorithms, Applications, and Open IssuesACM Computing Surveys10.1145/360370456:2(1-34)Online publication date: 15-Sep-2023
https://dl.acm.org/doi/10.1145/3603704
Lu XZheng XZhang PLi S(2023)Automobile Emergency Collision Avoidance Control for Pedestrian Crossing Based on Deep Q-Learning2023 China Automation Congress (CAC)10.1109/CAC59555.2023.10450548(234-239)Online publication date: 17-Nov-2023
https://doi.org/10.1109/CAC59555.2023.10450548
Zhang JHuang YHuang QLi YYe X(2023)Hasse sensitivity level: A sensitivity-aware trajectory privacy-enhanced framework with Reinforcement LearningFuture Generation Computer Systems10.1016/j.future.2023.01.008142(301-313)Online publication date: May-2023
https://doi.org/10.1016/j.future.2023.01.008
Watt Ndu Plessis M(2023)Neuro-augmented vision for evolutionary roboticsMachine Vision and Applications10.1007/s00138-023-01453-834:6Online publication date: 2-Sep-2023
https://dl.acm.org/doi/10.1007/s00138-023-01453-8
Granato GCartoni EDa Rold FMattera ABaldassarre G(2022)Integrating unsupervised and reinforcement learning in human categorical perception: A computational modelPLOS ONE10.1371/journal.pone.026783817:5(e0267838)Online publication date: 10-May-2022
https://doi.org/10.1371/journal.pone.0267838
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents