research-article

Evolving autoencoding structures through genetic programming

Authors:

Lino Rodriguez-Coayahuitl,

Alicia Morales-Reyes,

Hugo Jair EscalanteAuthors Info & Claims

Volume 20, Issue 3

Pages 413 - 440

https://doi.org/10.1007/s10710-019-09354-4

Published: 01 September 2019 Publication History

Abstract

We propose a novel method to evolve autoencoding structures through genetic programming (GP) for representation learning on high dimensional data. It involves a partitioning scheme of high dimensional input representations for distributed processing as well as an on-line form of learning that allows GP to efficiently process training datasets composed of hundreds or thousands of samples. The use of this on-line learning approach has important consequences in computational cost given different evolutionary population dynamics, namely steady state evolution and generational replacement. We perform a complete experimental study to compare the evolution of autoencoders (AEs) under different population dynamics and genetic operators useful to evolve GP based AEs’ individuals. Also, we compare the performance of GP based AEs with another representation learning method. Competitive results have been achieved through the proposed method. To the best of the authors’ knowledge, this research work is a precursor within the field of evolutionary deep learning.

References

[1]

C. Anderson, Lfwcrop face dataset. https://conradsanderson.id.au/lfwcrop/ (2014). Accessed 22 May 2019

[2]

D.H. Ballard, Modular learning in neural networks. in AAAI, pp. 279–284 (1987)

[3]

Y. Bengio, A. Courville, P. Vincent, Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013). https://doi.org/10.1109/TPAMI.2013.50

Digital Library

[4]

B.L. Betechuoh, T. Marwala, T. Tettey, Autoencoder networks for HIV classification. Curr. Sci. pp. 1467–1473 (2006)

[5]

M.C. Bot, Feature extraction for the k-nearest neighbour classifier with genetic programming. in European Conference on Genetic Programming, Springer, pp. 256–267 (2001)

[6]

L. Bottou, Large-scale machine learning with stochastic gradient descent. in Proceedings of COMPSTAT’2010, Springer, pp. 177–186 (2010)

[7]

H. Bourlard, Y. Kamp, Auto-association by multilayer perceptrons and singular value decomposition. Biol. Cybern. 59(4–5), 291–294 (1988)

Digital Library

[8]

B. Dolin, F.H. Bennett III, E.G. Rieffel, Co-evolving an effective fitness sample: experiments in symbolic regression and distributed robot control. in Proceedings of the 2002 ACM symposium on Applied computing, ACM, pp. 553–559 (2002)

[9]

J.A. Doucette, A.R. Mcintyre, P. Lichodzijewski, M.I. Heywood, Symbiotic coevolutionary genetic programming: a benchmarking study under large attribute spaces. Genet. Program. Evolv. Mach. 13(1), 71–101 (2012)

Digital Library

[10]

O.J. Dunn, Multiple comparisons among means. J. Am. Stat. Assoc. 56(293), 52–64 (1961)

[11]

A.E. Eiben, J.E. Smith et al., Introduction to Evolutionary Computing, vol. 53 (Springer, Berlin, 2003)

[12]

C. Fernando, D. Banarse, M. Reynolds, F. Besse, D. Pfau, M. Jaderberg, M. Lanctot, D. Wierstra Convolution by evolution: differentiable pattern producing networks. in Proceedings of the Genetic and Evolutionary Computation Conference 2016, ACM, pp. 109–116 (2016)

[13]

P. Gallinari, Y. LeCun, S. Thiria, F. Fogelman-Soulie, Memoires associatives distribuees. Proc. COGNITIVA 87, 93 (1987)

[14]

M. Garcia-Limon, H.J. Escalante, E. Morales, A. Morales-Reyes, Simultaneous generation of prototypes and features through genetic programming. in Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, ACM, pp. 517–524 (2014)

[15]

C. Gathercole, P. Ross, Dynamic training subset selection for supervised learning in genetic programming. in International Conference on Parallel Problem Solving from Nature, Springer, pp. 312–321 (1994)

[16]

C. Gathercole, P. Ross, Tackling the boolean even n parity problem with genetic programming and limited-error fitness. Genet. Program. 97, 119–127 (1997)

[17]

I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep Learning, vol. 1 (MIT Press, Cambridge, 2016)

Digital Library

[18]

G.E. Hinton, R.R. Salakhutdinov, Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

[19]

D.P. Kingma, J. Ba, Adam: a method for stochastic optimization. arXiv preprint arXiv:14126980 (2014)

[20]

J.R. Koza, Genetic Programming II, Automatic Discovery of Reusable Subprograms (MIT Press, Cambridge, 1992)

Digital Library

[21]

J.R. Koza, Genetic programming as a means for programming computers by natural selection. Stat. Comput. 4(2), 87–112 (1994)

[22]

Y. LeCun, The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/ (1998). Accessed 22 May 2019

[23]

Y. LeCun, Y. Bengio, G. Hinton, Deep learning. Nature 521(7553), 436 (2015)

[24]

M.G. Limón, H.J. Escalante, E. Morales, L.V. Pineda, Class-specific feature generation for 1NN through genetic programming. in Power, Electronics and Computing (ROPEC), 2015 IEEE International Autumn Meeting on, IEEE, pp. 1–6 (2015)

[25]

J.Y. Lin, H.R. Ke, B.C. Chien, W.P. Yang, Classifier design with feature selection and feature extraction using layered genetic programming. Expert Syst. Appl. 34(2), 1384–1393 (2008)

Digital Library

[26]

S. Luke, L. Spector, A comparison of crossover and mutation in genetic programming. Genet. Program. 97, 240–248 (1997)

[27]

S. Luke, L. Spector, A revised comparison of crossover and mutation in genetic programming. Genet. Program. 98(208–213), 55 (1998)

[28]

X. Mao, C. Shen, Y.B. Yang, Image restoration using very deep convolutional encoder–decoder networks with symmetric skip connections. in Advances in Neural Information Processing Systems, pp. 2802–2810 (2016)

[29]

Y. Martinez, L. Trujillo, E. Naredo, P. Legrand, A comparison of fitness-case sampling methods for symbolic regression with genetic programming. in EVOLVE-A Bridge Between Probability, Set Oriented Numerics, and Evolutionary Computation V, Springer, pp. 201–212 (2014)

[30]

F.E. Otero, C.G. Johnson, Automated problem decomposition for the boolean domain with genetic programming. in European Conference on Genetic Programming, Springer, pp. 169–180 (2013)

[31]

L. Pagie, P. Hogeweg, Evolutionary consequences of coevolving targets. Evolut. Comput. 5(4), 401–418 (1997)

Digital Library

[32]

A. Parkins, A.K. Nandi, Genetic programming techniques for hand written digit recognition. Signal Process. 84(12), 2345–2365 (2004)

Digital Library

[33]

R. Poli, W.B. Langdon, N.F. McPhee, J.R. Koza, A Field Guide to Genetic Programming (2008)

[34]

L. Rodriguez-Coayahuitl, A. Morales-Reyes, H.J. Escalante, Structurally layered representation learning: towards deep learning through genetic programming. in European Conference on Genetic Programming, Springer, pp. 271–288 (2018)

[35]

F.S. Samaria, A.C. Harter, Parameterisation of a stochastic model for human face identification. in Applications of Computer Vision, 1994., Proceedings of the Second IEEE Workshop on, IEEE, pp. 138–142 (1994)

[36]

J. Schmidhuber, Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)

Digital Library

[37]

M.D. Schmidt, H. Lipson, Coevolution of fitness predictors. IEEE Trans. Evolut. Comput. 12(6), 736–749 (2008)

Digital Library

[38]

L. Shao, L. Liu, X. Li, Feature learning for image classification via multiobjective genetic programming. IEEE Trans. Neural Netw. Learn. Syst. 25(7), 1359–1371 (2014)

[39]

K.O. Stanley, R. Miikkulainen, Evolving neural networks through augmenting topologies. Evolut. Comput. 10(2), 99–127 (2002)

Digital Library

[40]

K.O. Stanley, D.B. D’Ambrosio, J. Gauci, A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)

Digital Library

[41]

L. Theis, W. Shi, A. Cunningham, F. Huszár, Lossy image compression with compressive autoencoders. arXiv preprint arXiv:170300395 (2017)

[42]

B. Tran, B. Xue, M. Zhang, Genetic programming for feature construction and selection in classification on high-dimensional data. Memet. Comput. 8(1), 3–15 (2016)

[43]

B. Tran, B. Xue, M. Zhang, Using feature clustering for GP-based feature construction on high-dimensional data, in European Conference on Genetic Programming, Springer, pp. 210–226 (2017)

[44]

L. Trujillo, G. Olague, Synthesis of interest point detectors through genetic programming. in Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, ACM, pp. 887–894 (2006)

[45]

D.R. White, S. Poulding, A rigorous evaluation of crossover and mutation in genetic programming, in European Conference on Genetic Programming, Springer, pp. 220–231 (2009)

[46]

F. Wilcoxon, Individual comparisons by ranking methods. Biom. Bull. 1(6), 80–83 (1945)

[47]

L. Yann, Modèles connexionnistes de l’apprentissage. Ph.D. thesis, These de Doctorat, Universite Paris 6 (1987)

[48]

K. Zhang, W. Zuo, Y. Chen, D. Meng, L. Zhang, Beyond a gaussian denoiser: residual learning of deep cnn for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)

Digital Library

[49]

Y. Zhang, P.I. Rockett, A generic optimising feature extraction method using multiobjective genetic programming. Appl. Soft Comput. 11(1), 1087–1097 (2011)

Digital Library

Cited By

Hemberg EO'Reilly UToutouh JLi XHandl J(2024)Cooperative Coevolutionary Spatial Topologies for Autoencoder TrainingProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654127(331-339)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638529.3654127
Schofield FSlyfield LLensen A(2023)A Genetic Programming Encoder for Increasing Autoencoder InterpretabilityGenetic Programming10.1007/978-3-031-29573-7_2(19-35)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-29573-7_2
Pereira ETorres Rdos Santos J(2021)A genetic algorithm approach for image representation learning through color quantizationMultimedia Tools and Applications10.1007/s11042-020-10194-z80:10(15315-15350)Online publication date: 1-Apr-2021
https://dl.acm.org/doi/10.1007/s11042-020-10194-z
Show More Cited By

Index Terms

Evolving autoencoding structures through genetic programming
1. Computing methodologies
  1. Artificial intelligence
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Convolutional Genetic Programming
Pattern Recognition
Abstract
In recent years Convolutional Neural Networks (CNN) have come to dominate many machine learning tasks, specially those related to image analysis, such as object recognition. Herein we explore the possibility of developing image denoising filters ...
Evolving dynamic fitness measures for genetic programming
Highlights
- Further research is conducted on dynamic fitness measure genetic programming.
- ...
Abstract
This research builds on the hypothesis that the use of different fitness measures on the different generations of genetic programming (GP) is more effective than the convention of applying the same fitness measure individually ...
Evolving U-Nets Using Genetic Programming for Tree Crown Segmentation
Image and Vision Computing
Abstract
The U-Net deep learning algorithm and its variants have been developed for biomedical image segmentation, and due to their success gained popularity in other science domains including remote sensing. So far no U-Net structure has been specifically ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Genetic Programming and Evolvable Machines

Genetic Programming and Evolvable Machines Volume 20, Issue 3

Sep 2019

154 pages

ISSN:1389-2576

Issue’s Table of Contents

Copyright © 2019 Springer Science+Business Media, LLC, part of Springer Nature.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 September 2019

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hemberg EO'Reilly UToutouh JLi XHandl J(2024)Cooperative Coevolutionary Spatial Topologies for Autoencoder TrainingProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654127(331-339)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638529.3654127
Schofield FSlyfield LLensen A(2023)A Genetic Programming Encoder for Increasing Autoencoder InterpretabilityGenetic Programming10.1007/978-3-031-29573-7_2(19-35)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-29573-7_2
Pereira ETorres Rdos Santos J(2021)A genetic algorithm approach for image representation learning through color quantizationMultimedia Tools and Applications10.1007/s11042-020-10194-z80:10(15315-15350)Online publication date: 1-Apr-2021
https://dl.acm.org/doi/10.1007/s11042-020-10194-z
Ruberto STerragni VMoore J(2020)Image Feature Learning with Genetic ProgrammingParallel Problem Solving from Nature – PPSN XVI10.1007/978-3-030-58115-2_5(63-78)Online publication date: 5-Sep-2020
https://dl.acm.org/doi/10.1007/978-3-030-58115-2_5

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents