Abstract
Representations of multivariable Boolean functions by one and two-hidden-layer Heaviside perceptron networks are investigated. Sufficient conditions are given for representations with the numbers of network units depending on the input dimension d linearly and polynomially. Functions with such numbers depending on d exponentially or having some weights exponentially large are described in terms of properties of their communication matrices. A mathematical formalization of the concept of “highly-varying functions” is proposed. There is given an example of such function which can be represented by a network with two hidden layers with merely d units.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Fine, T.L.: Feedforward Neural Network Methodology. Springer, Heidelberg (1999)
Chow, T.W.S., Cho, S.Y.: Neural Networks and Computing: Learning Algorithms and Applications. World Scientific (2007)
Leshno, M., Lin, V.Y., Pinkus, A., Schocken, S.: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks 6, 861–867 (1993)
Pinkus, A.: Approximation theory of the MLP model in neural networks. Acta Numerica 8, 143–195 (1999)
Park, J., Sandberg, I.: Approximation and radial-basis-function networks. Neural Computation 5, 305–316 (1993)
Mhaskar, H.N.: Versatile Gaussian networks. In: Proceedings of IEEE Workshop of Nonlinear Image Processing, pp. 70–73 (1995)
Kůrková, V.: Some Comparisons of Networks with Radial and Kernel Units. In: Villa, A.E.P., Duch, W., Érdi, P., Masulli, F., Palm, G. (eds.) ICANN 2012, Part II. LNCS, vol. 7553, pp. 17–24. Springer, Heidelberg (2012)
Ito, Y.: Finite mapping by neural networks and truth functions. Mathematical Scientist 17, 69–77 (1992)
Micchelli, C.A.: Interpolation of scattered data: Distance matrices and conditionally positive definite functions. Constructive Approximation 2, 11–22 (1986)
Kainen, P.C., Kůrková, V., Sanguineti, M.: Dependence of computational models on input dimension: Tractability of approximation and optimization tasks. IEEE Trans. on Information Theory 58(2), 1203–1214 (2012)
Maiorov, V., Pinkus, A.: Lower bounds for approximation by MLP neural networks. Neurocomputing 25, 81–91 (1999)
Maiorov, V.: On best approximation by ridge functions. J. of Approximation Theory 99, 68–94 (1999)
Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Computation (2006)
Bengio, Y.: Learning deep architectures for AI. Foundations and Trends in Machine Learning 2, 1–127 (2009)
Grochowski, M., Duch, W.: Learning Highly Non-separable Boolean Functions Using Constructive Feedforward Neural Network. In: de Sá, J.M., Alexandre, L.A., Duch, W., Mandic, D.P. (eds.) ICANN 2007, Part I. LNCS, vol. 4668, pp. 180–189. Springer, Heidelberg (2007)
Bengio, Y., Delalleau, O., Roux, N.L.: The curse of highly variable functions for local kernel machines. In: Advances in Neural Information Processing Systems, vol. 18, pp. 107–114. MIT Press (2006)
Bengio, Y., Delalleau, O., Roux, N.L.: The curse of dimensionality for local kernel machines. Technical Report 1258, Département d’Informatique et Recherche Opérationnelle, Université de Montréal (2005)
Bartlett, P.L.: The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network. IEEE Trans. on Information Theory 44, 525–536 (1998)
Shläfli, L.: Gesamelte mathematische abhandlungen, Band 1 (1950)
Erdös, P., Spencer, J.H.: Probabilistic Methods in Combinatorics. Academic Press (1974)
Kůrková, V.: Dimension-independent rates of approximation by neural networks. In: Warwick, K., Kárný, M. (eds.) Computer-Intensive Methods in Control and Signal Processing, pp. 261–270. Birkhäuser, Boston (1997)
Barron, A.R.: Neural net approximation. In: Narendra, K. (ed.) Proc. 7th Yale Workshop on Adaptive and Learning Systems, pp. 69–72. Yale University Press (1992)
Kůrková, V.: Complexity estimates based on integral transforms induced by computational units. Neural Networks 33, 160–167 (2012)
Kůrková, V., Savický, P., Hlaváčková, K.: Representations and rates of approximation of real–valued Boolean functions by neural networks. Neural Networks 11, 651–659 (1998)
Knuth, D.E.: Big omicron and big omega and big theta. SIGACT News 8(2), 18–24 (1976)
Kůrková, V., Sanguineti, M.: Comparison of worst-case errors in linear and neural network approximation. IEEE Trans. on Information Theory 48, 264–275 (2002)
Hajnal, A., Maas, W., Pudlák, P., Szegedy, M., Turán, G.: Threshold circuits of bounded depth. In: Proc. 28th Annual Symposium on Foundations of Computer Science, pp. 99–110. IEEE (1987)
Hajnal, A., Maas, W., Pudlák, P., Szegedy, M., Turán, G.: Threshold circuits of bounded depth. J. of Computer and System Sciences 46, 129–154
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kůrková, V., Sanguineti, M. (2013). Can Two Hidden Layers Make a Difference?. In: Tomassini, M., Antonioni, A., Daolio, F., Buesser, P. (eds) Adaptive and Natural Computing Algorithms. ICANNGA 2013. Lecture Notes in Computer Science, vol 7824. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37213-1_4
Download citation
DOI: https://doi.org/10.1007/978-3-642-37213-1_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-37212-4
Online ISBN: 978-3-642-37213-1
eBook Packages: Computer ScienceComputer Science (R0)