Abstract
Researchers in the fields of neural networks, statistics, machine learning, and artificial intelligence have followed three basic approaches to developing new pattern classifiers. Probability Density Function (PDF) classifiers include Gaussian and Gaussian Mixture classifiers which estimate distributions or densities of input features separately for each class. Posterior probability classifiers include multilayer perceptron neural networks with sigmoid nonlinearities and radial basis function networks. These classifiers estimate minimum-error Bayesian a posteriori probabilities (hereafter referred to as posterior probabilities) simultaneously for all classes. Boundary forming classifiers include hard-limiting single-layer perceptrons, hypersphere classifiers, and nearest neighbor classifiers. These classifiers have binary indicator outputs which form decision regions that specify the class of any input pattern. Posterior probability and boundary-forming classifiers are trained using discriminant training. All training data is used simultaneously to estimate Bayesian posterior probabilities or minimize overall classification error rates. PDF classifiers are trained using maximum likelihood approaches which individually model class distributions without regard to overall classification performance. Analytic results are presented which demonstrate that many neural network classifiers can accurately estimate posterior probabilities and that these neural network classifiers can sometimes provide lower error rates than PDF classifiers using the same number of trainable parameters. Experiments also demonstrate how interpretation of network outputs as posterior probabilities makes it possible to estimate the confidence of a classification decision, compensate for differences in class prior probabilities between test and training data, and combine outputs of multiple classifiers over time for speech recognition.
This work was sponsored by the Advanced Research Projects Agency. The views expressed are those of the authors and do not reflect the official policy or position of the U.S. Government
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
L. Breiman, J. H. Friedman, et al., Classification and Regression Trees. Belmont, CA: Wadsworth International Group (1984).
Eric I. Chang and Richard P. Lippmann, “A Boundary Hunting Radial Basis Function Classifier Which Allocates Centers Constructively,” in Neural Information Processing Systems 5, Morgan Kaufmann: 139–146, 1993.
John B. Hampshire, A Differential Theory of Learning for Efficient Statistical Pattern Recognition, Carnegie Mellon University, Ph.D., 1993.
John B. Hampshire and Alexander H. Waibel, “A Novel Objective Function for Improved Phoneme Recognition Using Time-Delay Neural Networks,” in IEEE Transactions on Neural Networks, 216–228, 1990.
Don R. Hush and Bill G. Home, “Progress in Supervised Neural Networks,” IEEE Signal Processing Magazine, 10(1), 8–39, 1993.
T. Kohonen, “An Introduction to Neural Computing,” Neural Networks, 1, 3–16, 1988.
Richard P. Lippmann, “An Introduction to Computing with Neural Nets,” IEEE Signal Processing Magazine, 4(2), 4–22, 1987.
Nelson Morgan and Herve Bourlard. “Continuous Speech Recognition Using Multilayer Perceptrons with Hidden Markov Models,” in International Conference on Acoustics Speech and Signal Processing. 1990.
Kenney Ng, A Comparative Study of the Practical Characteristics of Neural Network and Conventional Pattern Classifiers, MIT, Master’s Thesis, 1990.
Gorden E. Peterson and Harold L. Barney, “Control Methods Used in a Study of Vowels,” The Journal of the Acoustical Society of America, 24(2), 175–84, 1952.
D. L. Reilly, L. N. Cooper, and C. Elbaum, “A neural model for category learning,” Biological Cybernetics, 45, 35–41, 1982.
Mike D. Richard and Richard P. Lippmann, “Neural Network Classifiers Estimate Bayesian a Posteriori Probabilities,” Neural Computation, 3, 461–483, 1992.
W. Rogers, N. Armstrong, et al., Report of the Presidential Commission on the Space Shuttle Challenger Accident. Washington D.C.: U.S. Government (1986).
Frank Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Spartan Books (1962).
Elliot Singer and Richard P. Lippmann, “Improved Hidden Markov Model Speech Recognition Using Radial Basis Function Networks,” in Neural Information Processing Systems 4, J. Moody, S. Hanson, and R. Lippmann, (Eds.), Morgan Kaufmann: San Mateo, California, 1992.
Elliot Singer and Richard P. Lippmann. “A Speech Recognizer Using Radial Basis Function Neural Networks in an HMM Framework,” in Proceedings International Conference on Acoustics Speech and Signal Processing. San Francisco: IEEE, 1992.
Torsten Zeppenfeld and Alex Waibel. “A Hybrid Neural Network, Dynamic Programming Word Spotter,” in Proceedings International Conference on Acoustics Speech and Signal Processing. San Francisco: IEEE, 1992.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1994 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lippmann, R.P. (1994). Neural Networks, Bayesian a posteriori Probabilities, and Pattern Classification. In: Cherkassky, V., Friedman, J.H., Wechsler, H. (eds) From Statistics to Neural Networks. NATO ASI Series, vol 136. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-79119-2_4
Download citation
DOI: https://doi.org/10.1007/978-3-642-79119-2_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-79121-5
Online ISBN: 978-3-642-79119-2
eBook Packages: Springer Book Archive