Abstract
Several evolutionary algorithms make use of hierarchical representations of variable size rather than linear strings of fixed length. Variable complexity of the structures provides an additional representational power which may widen the application domain of evolutionary algorithms. The price for this is, however, that the search space is open-ended and solutions may grow to arbitrarily large size. In this paper we study the effects of structural complexity of the solutions on their generalization performance by analyzing the fitness landscape of sigma-pi neural networks. The analysis suggests that smaller networks achieve, on average, better generalization accuracy than larger ones, thus confirming the usefulness of Occam's razor. A simple method for implementing the Occam's razor principle is described and shown to be effective in improving the generalization accuracy without limiting their learning capacity.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Y. S. Abu-Mostafa: The Vapnik-Chervonenkis dimension: Information versus complexity in learning. Neural Computation 1, 312–317 (1989)
T. Bäck and H.-P. Schwefel: An overview of evolutionary algorithms for parameter optimization. Evolutionary Computation 1, 1–23 (1993)
R. Durbin, D. E. Rumelhart: Product units: A computationally powerful and biologically plausible extension to backpropagation networks. Neural Computation 1, 133–142 (1989)
C. L. Giles, T. Maxwell: Learning, invariance, and generalization in high-order neural networks. Applied Optics 26, 4972–4978 (1987)
D. E. Goldberg: Genetic Algorithms in Search, Optimization & Machine Learning. Addison Wesley, 1989
F. Gruau: Genetic synthesis of Boolean neural networks with a cell rewriting developmental process. Tech. Rep., Laboratoire de l'Informatique du Parallélisme, 1992
S. A. Harp, T. Samad, A. Guha: Towards the genetic synthesis of neural networks. In: D. Schaffer (ed.): Proceedings of the Third International Conference on Genetic Algorithms (ICGA-89). Morgan Kaufmann, 1985, pp. 360–369
H. Iba, T. Kurita, H. de Garis, T. Sato: System identification using structured genetic algorithm. In: S. Forrest (ed.): Proceedings of the Fifth International Conference on Genetic Algorithms (ICGA-93). Morgan Kaufmann, 1993, pp. 279–286
H. Kargupta, R. E. Smith: System identification with evolving polynomial networks. In: R. Belew, L. Booker (eds.): Proceedings of the Fourth International Conference on Genetic Algorithms (ICGA-91). Morgan Kaufmann, 1991, pp. 370–376
K. E. Kinnear Jr.: Generality and difficulty in genetic programming: Evolving a sort. In: S. Forrest (ed.): Proceedings of the Fifth International Conference on Genetic Algorithms (ICGA-93). Morgan Kaufmann, 1993, pp. 287–294
H. Kitano: Designing neural networks using genetic algorithms with graph generation system. Complex Systems 4, 461–476 (1990)
J. R. Koza: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, 1992
H. Mühlenbein: Evolutionary algorithms — Theory and applications. In: E. H. L. Aarts, J. K. Lenstra (eds.): Local Search in Combinatorial Optimization. Wiley, 1993
H. Mühlenbein, D. Schierkamp-Voosen: Predictive models for the breeder genetic algorithm I: Continuous parameter optimization. Evolutionary Computation 1, 25–49 (1993)
I. Rechenberg: Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Stuttgart: Frommann-Holzboog, 1973
J. Rissanen: Stochastic complexity and modeling. The Annuls of Statistics 14, 1080–1100 (1986)
J. D. Schaffer: Multiple objective optimization with vector evaluated genetic algorithm. In: J. D. Schaffer (ed.): Proceedings of the International Conference on Genetic Algorithms and Their Applications. Erlbaum Associates, 1985, pp. 93–100
W. A. Tackett: Genetic programming for feature discovery and image discrimination. In: S. Forrest (ed.): Proceedings of the Fifth International Conference on Genetic Algorithms (ICGA-93). Morgan Kaufmann, 1993, pp. 303–309
D. Whitley, T. Starkweather, C. Bogart: Genetic algorithms and neural networks: Optimizing connections and connectivity. Parallel Computing 14, 347–361 (1990)
B. T. Zhang, H. Mühlenbein: Genetic programming of minimal neural nets using Occam's razor. In: S. Forrest (ed.): Proceedings of the Fifth International Conference on Genetic Algorithms (ICGA-93). Morgan Kaufmann, 1993, pp. 342–349
B. T. Zhang, H. Mühlenbein: Evolving optimal neural networks using genetic algorithms with Occam's razor. Complex Systems (1994)
B. T. Zhang, H. Mühlenbein: Synthesis of sigma-pi neural networks by the breeder genetic programming. In: Proceedings of the IEEE World Congress on Computational Intelligence (WCCI-94) New York: IEEE Press, 1994
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1994 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhang, BT. (1994). Effects of Occam's razor in evolving Sigma-Pi neural nets. In: Davidor, Y., Schwefel, HP., Männer, R. (eds) Parallel Problem Solving from Nature — PPSN III. PPSN 1994. Lecture Notes in Computer Science, vol 866. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-58484-6_289
Download citation
DOI: https://doi.org/10.1007/3-540-58484-6_289
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-58484-1
Online ISBN: 978-3-540-49001-2
eBook Packages: Springer Book Archive