Abstract
Networks can be considered as approximation schemes. Multilayer networks of the perceptron type can approximate arbitrarily well continuous functions (Cybenko 1988, 1989; Funahashi 1989; Stinchcombe and White 1989). We prove that networks derived from regularization theory and including Radial Basis Functions (Poggio and Girosi 1989), have a similar property. From the point of view of approximation theory, however, the property of approximating continuous functions arbitrarily well is not sufficient for characterizing good approximation schemes. More critical is the property ofbest approximation. The main result of this paper is that multilayer perceptron networks, of the type used in backpropagation, do not have the best approximation property. For regularization networks (in particular Radial Basis Function networks) we prove existence and uniqueness of best approximation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bertero M (1986) Regularization methods for linear inverse problems. In: Talenti CG (eds) Inverse problems. Springer, Berlin Heidelberg New York
Bertero M, Poggio T, Torre V (1988) Ill-posed problems in early vision. Proc IEEE 76:869–889
Braess D (1986) Nonlinear approximation theory. Springer, Berlin Heidelberg New York
Broomhead DS, Lowe D (1988) Multivariable functional interpolation and adaptive networks. Complex Syst 2:321–355
Carrol SM, Dickinson BW (1989) Construction of neural nets using the Radon transform. In: Proceedings of the International Joint Conference on Neural Networks, pp I-607–I-611. Washington D. C. June 1989. IEEE TAB Neural Network Committee
Cheney EW (1981) Introduction to approximation theory. Chelsea, New York
Cybenko (1988) Continuous valued neural networks with two hidden layers are sufficient. Technical report. Department of Computer Sciences, Tufts University, Medford, Mass
Cybenko G (1989) Approximation by superposition of a sigmoidal function. Math Control Syst Signals (in press)
De Boor C (1969) On the approximation byγ-Polynomials. In: Schoenberg IJ (eds) Approximation with special emphasis on spline functions. Academic Press, New York, pp 157–183
Funahashi K (1989) On the approximate realization of continuous mappings by neural networks. Neural Networks 2:183–192
Gelfand IM, Shilov GE (1964) Generalized functions, vol 1: Properties and operations. Academic Press, New York
Hartman E, Keeler K, Kowalski JM (1989) Layered neural networks with gaussian hidden units as universal approximations, (submitted for publication)
Hobby CR, Rice JR (1967) Approximation from a curve of functions. Arch Rat Mech Anal 27:91–106
Micchelli CA (1986) Interpolation of scattered data: distance matrices and conditionally positive definite functions. Constn Approx 2:11–22
Moody J, Darken C (1989) Fast learning in networks of locally-tuned processing units. Neural Comput 1:281–294
Moore B, Poggio T (1988) Representations properties of multilayer feed-forward networks. In: Abstracts of the First Annual INNS Meeting. Pergamon Press, New York, p 502
Morozov VA (1984) Methods for solving incorrectly posed problems. Springer, Berlin Heidelberg New York
Poggio T, Girosi F (1989) A theory of networks for approximation and learning. A. I. Memo No. 1140. Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge Mass
Rice JR (1964) The approximation of functions, vol 1. Addison-Wesley, Reading, Mass
Rice JR (1969) The approximation of functions, vol. 2. Addison-Wesley, Reading, Mass
Rudin W (1973) Functional analysis. McGraw-Hill, New York
Rumelhart DE, Hinton GE, Williams RJ (1986a) Learning internal representations by error propagation. In: Parallel distributed processing, chap 8. MIT Press, Cambridge Mass, pp 318–362
Rumelhart DE, Hinton GE, Williams RJ (1986b) Learning representations by back-propagating errors. Nature 323:533–536
Sejnowski TJ, Rosenberg CR (1987) Parallel networks that learn to pronounce english text. Complex Syst 1:145–168
Stinchcombe M, White H (1989) Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions. In: Proceedings of the International Joint Conference on Neural Networks. Washington DC, June 1989. IEEE TAB Neural Network Committee, pp I/607–I/611
Stone MH (1937) Applications of the theory of Boolean rings to general topology. AMS Trans 41:375–481
Stone MH (1948) The generalized Weierstrass approximation theorem. Math Mag 21:167–183, 237–254
Tikhonov AN (1963) Solution of incorrectly formulated problems and the regularization method. Soviet Math Dokl 4:1035–1038
Tikhonov AN, Arsenin VY (1977) Solutions of Ill-posed Problems. Winston, Washington, DC
Yosida K (1974) Functional analysis. Springer, Berlin Heidelberg New York
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Girosi, F., Poggio, T. Networks and the best approximation property. Biol. Cybern. 63, 169–176 (1990). https://doi.org/10.1007/BF00195855
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/BF00195855