Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Evolving autoencoding structures through genetic programming

  • Published:
Genetic Programming and Evolvable Machines Aims and scope Submit manuscript

Abstract

We propose a novel method to evolve autoencoding structures through genetic programming (GP) for representation learning on high dimensional data. It involves a partitioning scheme of high dimensional input representations for distributed processing as well as an on-line form of learning that allows GP to efficiently process training datasets composed of hundreds or thousands of samples. The use of this on-line learning approach has important consequences in computational cost given different evolutionary population dynamics, namely steady state evolution and generational replacement. We perform a complete experimental study to compare the evolution of autoencoders (AEs) under different population dynamics and genetic operators useful to evolve GP based AEs’ individuals. Also, we compare the performance of GP based AEs with another representation learning method. Competitive results have been achieved through the proposed method. To the best of the authors’ knowledge, this research work is a precursor within the field of evolutionary deep learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. This holds only if partitions are built by grouping contiguous features, given problems that allow to do so, such as in image, audio, text and time series problems.

  2. Trying to mix portions of encoder and decoder forests to generate an offspring’ decoder or encoder would be an error, because two random crossover points would need to be picked in order to guarantee that resulting forest has a proper number of trees.

References

  1. C. Anderson, Lfwcrop face dataset. https://conradsanderson.id.au/lfwcrop/ (2014). Accessed 22 May 2019

  2. D.H. Ballard, Modular learning in neural networks. in AAAI, pp. 279–284 (1987)

  3. Y. Bengio, A. Courville, P. Vincent, Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013). https://doi.org/10.1109/TPAMI.2013.50

    Article  Google Scholar 

  4. B.L. Betechuoh, T. Marwala, T. Tettey, Autoencoder networks for HIV classification. Curr. Sci. pp. 1467–1473 (2006)

  5. M.C. Bot, Feature extraction for the k-nearest neighbour classifier with genetic programming. in European Conference on Genetic Programming, Springer, pp. 256–267 (2001)

  6. L. Bottou, Large-scale machine learning with stochastic gradient descent. in Proceedings of COMPSTAT’2010, Springer, pp. 177–186 (2010)

  7. H. Bourlard, Y. Kamp, Auto-association by multilayer perceptrons and singular value decomposition. Biol. Cybern. 59(4–5), 291–294 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  8. B. Dolin, F.H. Bennett III, E.G. Rieffel, Co-evolving an effective fitness sample: experiments in symbolic regression and distributed robot control. in Proceedings of the 2002 ACM symposium on Applied computing, ACM, pp. 553–559 (2002)

  9. J.A. Doucette, A.R. Mcintyre, P. Lichodzijewski, M.I. Heywood, Symbiotic coevolutionary genetic programming: a benchmarking study under large attribute spaces. Genet. Program. Evolv. Mach. 13(1), 71–101 (2012)

    Article  Google Scholar 

  10. O.J. Dunn, Multiple comparisons among means. J. Am. Stat. Assoc. 56(293), 52–64 (1961)

    Article  MathSciNet  MATH  Google Scholar 

  11. A.E. Eiben, J.E. Smith et al., Introduction to Evolutionary Computing, vol. 53 (Springer, Berlin, 2003)

    Book  MATH  Google Scholar 

  12. C. Fernando, D. Banarse, M. Reynolds, F. Besse, D. Pfau, M. Jaderberg, M. Lanctot, D. Wierstra Convolution by evolution: differentiable pattern producing networks. in Proceedings of the Genetic and Evolutionary Computation Conference 2016, ACM, pp. 109–116 (2016)

  13. P. Gallinari, Y. LeCun, S. Thiria, F. Fogelman-Soulie, Memoires associatives distribuees. Proc. COGNITIVA 87, 93 (1987)

    Google Scholar 

  14. M. Garcia-Limon, H.J. Escalante, E. Morales, A. Morales-Reyes, Simultaneous generation of prototypes and features through genetic programming. in Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, ACM, pp. 517–524 (2014)

  15. C. Gathercole, P. Ross, Dynamic training subset selection for supervised learning in genetic programming. in International Conference on Parallel Problem Solving from Nature, Springer, pp. 312–321 (1994)

  16. C. Gathercole, P. Ross, Tackling the boolean even n parity problem with genetic programming and limited-error fitness. Genet. Program. 97, 119–127 (1997)

    Google Scholar 

  17. I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep Learning, vol. 1 (MIT Press, Cambridge, 2016)

    MATH  Google Scholar 

  18. G.E. Hinton, R.R. Salakhutdinov, Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  19. D.P. Kingma, J. Ba, Adam: a method for stochastic optimization. arXiv preprint arXiv:14126980 (2014)

  20. J.R. Koza, Genetic Programming II, Automatic Discovery of Reusable Subprograms (MIT Press, Cambridge, 1992)

    Google Scholar 

  21. J.R. Koza, Genetic programming as a means for programming computers by natural selection. Stat. Comput. 4(2), 87–112 (1994)

    Article  Google Scholar 

  22. Y. LeCun, The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/ (1998). Accessed 22 May 2019

  23. Y. LeCun, Y. Bengio, G. Hinton, Deep learning. Nature 521(7553), 436 (2015)

    Article  Google Scholar 

  24. M.G. Limón, H.J. Escalante, E. Morales, L.V. Pineda, Class-specific feature generation for 1NN through genetic programming. in Power, Electronics and Computing (ROPEC), 2015 IEEE International Autumn Meeting on, IEEE, pp. 1–6 (2015)

  25. J.Y. Lin, H.R. Ke, B.C. Chien, W.P. Yang, Classifier design with feature selection and feature extraction using layered genetic programming. Expert Syst. Appl. 34(2), 1384–1393 (2008)

    Article  Google Scholar 

  26. S. Luke, L. Spector, A comparison of crossover and mutation in genetic programming. Genet. Program. 97, 240–248 (1997)

    Google Scholar 

  27. S. Luke, L. Spector, A revised comparison of crossover and mutation in genetic programming. Genet. Program. 98(208–213), 55 (1998)

    Google Scholar 

  28. X. Mao, C. Shen, Y.B. Yang, Image restoration using very deep convolutional encoder–decoder networks with symmetric skip connections. in Advances in Neural Information Processing Systems, pp. 2802–2810 (2016)

  29. Y. Martinez, L. Trujillo, E. Naredo, P. Legrand, A comparison of fitness-case sampling methods for symbolic regression with genetic programming. in EVOLVE-A Bridge Between Probability, Set Oriented Numerics, and Evolutionary Computation V, Springer, pp. 201–212 (2014)

  30. F.E. Otero, C.G. Johnson, Automated problem decomposition for the boolean domain with genetic programming. in European Conference on Genetic Programming, Springer, pp. 169–180 (2013)

  31. L. Pagie, P. Hogeweg, Evolutionary consequences of coevolving targets. Evolut. Comput. 5(4), 401–418 (1997)

    Article  Google Scholar 

  32. A. Parkins, A.K. Nandi, Genetic programming techniques for hand written digit recognition. Signal Process. 84(12), 2345–2365 (2004)

    Article  Google Scholar 

  33. R. Poli, W.B. Langdon, N.F. McPhee, J.R. Koza, A Field Guide to Genetic Programming (2008)

  34. L. Rodriguez-Coayahuitl, A. Morales-Reyes, H.J. Escalante, Structurally layered representation learning: towards deep learning through genetic programming. in European Conference on Genetic Programming, Springer, pp. 271–288 (2018)

  35. F.S. Samaria, A.C. Harter, Parameterisation of a stochastic model for human face identification. in Applications of Computer Vision, 1994., Proceedings of the Second IEEE Workshop on, IEEE, pp. 138–142 (1994)

  36. J. Schmidhuber, Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  37. M.D. Schmidt, H. Lipson, Coevolution of fitness predictors. IEEE Trans. Evolut. Comput. 12(6), 736–749 (2008)

    Article  Google Scholar 

  38. L. Shao, L. Liu, X. Li, Feature learning for image classification via multiobjective genetic programming. IEEE Trans. Neural Netw. Learn. Syst. 25(7), 1359–1371 (2014)

    Article  Google Scholar 

  39. K.O. Stanley, R. Miikkulainen, Evolving neural networks through augmenting topologies. Evolut. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  40. K.O. Stanley, D.B. D’Ambrosio, J. Gauci, A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)

    Article  Google Scholar 

  41. L. Theis, W. Shi, A. Cunningham, F. Huszár, Lossy image compression with compressive autoencoders. arXiv preprint arXiv:170300395 (2017)

  42. B. Tran, B. Xue, M. Zhang, Genetic programming for feature construction and selection in classification on high-dimensional data. Memet. Comput. 8(1), 3–15 (2016)

    Article  Google Scholar 

  43. B. Tran, B. Xue, M. Zhang, Using feature clustering for GP-based feature construction on high-dimensional data, in European Conference on Genetic Programming, Springer, pp. 210–226 (2017)

  44. L. Trujillo, G. Olague, Synthesis of interest point detectors through genetic programming. in Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, ACM, pp. 887–894 (2006)

  45. D.R. White, S. Poulding, A rigorous evaluation of crossover and mutation in genetic programming, in European Conference on Genetic Programming, Springer, pp. 220–231 (2009)

  46. F. Wilcoxon, Individual comparisons by ranking methods. Biom. Bull. 1(6), 80–83 (1945)

    Article  Google Scholar 

  47. L. Yann, Modèles connexionnistes de l’apprentissage. Ph.D. thesis, These de Doctorat, Universite Paris 6 (1987)

  48. K. Zhang, W. Zuo, Y. Chen, D. Meng, L. Zhang, Beyond a gaussian denoiser: residual learning of deep cnn for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  49. Y. Zhang, P.I. Rockett, A generic optimising feature extraction method using multiobjective genetic programming. Appl. Soft Comput. 11(1), 1087–1097 (2011)

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by CONACyT under Grant No. 436184 - Becas Nacionales 2016, and Grant No. A1-S-26314 - Integración de Visión y Lenguaje mediante Representaciones Multimodales Aprendidas para Clasificación y Recuperación de Imágenes y Videos.

Author information

Authors and Affiliations

  1. Computer Science Department, Instituto Nacional de Astrofísica, Óptica y Electrónica, 72840, Tonantzintla, PUE, Mexico

    Lino Rodriguez-Coayahuitl, Alicia Morales-Reyes & Hugo Jair Escalante

  2. Computer Science Department, Centro de Investigación y Estudios Avanzados del IPN, 07360, Zacatenco, CDMX, Mexico

    Hugo Jair Escalante

Authors
  1. Lino Rodriguez-Coayahuitl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alicia Morales-Reyes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hugo Jair Escalante
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lino Rodriguez-Coayahuitl.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rodriguez-Coayahuitl, L., Morales-Reyes, A. & Escalante, H.J. Evolving autoencoding structures through genetic programming. Genet Program Evolvable Mach 20, 413–440 (2019). https://doi.org/10.1007/s10710-019-09354-4

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10710-019-09354-4

Keywords

Search

Navigation