article

Using a Neural Network to Approximate an Ensemble of Classifiers

Authors:

X. Zeng,

T. R. MartinezAuthors Info & Claims

Neural Processing Letters, Volume 12, Issue 3

Pages 225 - 237

https://doi.org/10.1023/A:1026530200837

Published: 01 December 2000 Publication History

Publisher Site

Abstract

Several methods (e.g., Bagging, Boosting) of constructing and combining an ensemble of classifiers have recently been shown capable of improving accuracy of a class of commonly used classifiers (e.g., decision trees, neural networks). The accuracy gain achieved, however, is at the expense of a higher requirement for storage and computation. This storage and computation overhead can decrease the utility of these methods when applied to real-world situations. In this Letter, we propose a learning approach which allows a single neural network to approximate a given ensemble of classifiers. Experiments on a large number of real-world data sets show that this approach can substantially save storage and computation while still maintaining accuracy similar to that of the entire ensemble.

References

[1]

1. Breiman, L.: Bagging predictors, Machine Learning, 24 (1996), 123-140.

Digital Library

Google Scholar

[2]

2. Freund, Y. and Schapire, R.: Experiments with a new boosting algorithm, In: Proc. Thirteenth Nat. Conf. Machine Learning, Morgan Kaufmann, 1996, pp. 148-156

Google Scholar

[3]

3. Quinlan, J. R.: Bagging, boosting, and c4.5, In: Proc. Thirteenth Nat. Conf. Artificial Intelligence, AAAI/MIT Press, 1996, pp. 725-730.

Digital Library

Google Scholar

[4]

4. Bauer, E. and Kohavi, R.: An empirical comparison of voting classification algorithms: bagging, boosting and variants, Machine Learning, 36 (1999), 105-139.

Digital Library

Google Scholar

[5]

5. Maclin, R. and Opitz, D.: An empirical evaluation of bagging and boosting, In: Proc. Fourteenth Nat. Conf. Artificial Intelligence, AAAI/MIT Press, 1997, pp. 546-551.

Digital Library

Google Scholar

[6]

6. Dietterich, T. G.: Machine-learning research - four current directions, AI Magazine, Winter (1997), 97-136.

Google Scholar

[7]

7. Margineantu, D. D. and Dietterich, T. G.: Pruning adaptive boosting, In: Proc. Fourteenth Int. Conf. Machine Learning, 1997, pp. 98-106.

Digital Library

Google Scholar

[8]

8. Dominggos, P.: Knowledge acquisition from examples vis multiple models, In: Proc. Fourteenth Int. Conf. Machine Learning, 1997, pp. 211-218.

Digital Library

Google Scholar

[9]

9. Craven, M. W. and Shavlik, J. W.: Learning symbolic rules using artificial neural networks, In: Proc. 10th Int. Conf. Machine Learning, Amherst, MA, Kaufmann, 1993, pp. 73-80.

Crossref

Google Scholar

[10]

10. Craven, M. W. and Shavlik, J. W.: Extracting tree-structured representation from trained networks, In: D. S. Touretzky, M. C. Mozer and M. Hasselmo, eds., Advances in Neural Information Processing System 8, MIT Press, 1996, pp. 24-30.

Google Scholar

[11]

11. Efron, B. and Tibshirani, R.: An Introduction to the Bootstrap, New York, Chapman and Hall, 1993.

Crossref

Google Scholar

[12]

12. Merz, C. J. and Murphy, P. M.: UCI repository of machine learning databases, http://www.ics.uci.edu/~mlearn/MLRepository.html, 1996.

Google Scholar

[13]

13. Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J.: Classification and Regression Trees, Wadsworth International Group, 1984.

Google Scholar

[14]

14. Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection, In: Proc. Int. Joint Conf. Artificial Intelligence, 1995, pp. 1137-1143.

Digital Library

Google Scholar

Cited By

View all

Stanton SIzmailov PKirichenko PAlemi AWilson ARanzato MBeygelzimer ADauphin YLiang PVaughan J(2021)Does knowledge distillation really work?Proceedings of the 35th International Conference on Neural Information Processing Systems10.5555/3540261.3540790(6906-6919)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.5555/3540261.3540790
Price RIso KShinoda K(2016)Wise teachers train better DNN acoustic modelsEURASIP Journal on Audio, Speech, and Music Processing10.1186/s13636-016-0088-72016:1(1-19)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1186/s13636-016-0088-7
Menke JMartinez T(2009)Artificial neural network reduction through oracle learningIntelligent Data Analysis10.5555/1551758.155176513:1(135-149)Online publication date: 1-Jan-2009
https://dl.acm.org/doi/10.5555/1551758.1551765
Show More Cited By

Index Terms

Using a Neural Network to Approximate an Ensemble of Classifiers

Recommendations

Class-switching neural network ensembles

This article investigates the properties of class-switching ensembles composed of neural networks and compares them to class-switching ensembles of decision trees and to standard ensemble learning methods, such as bagging and boosting. In a class-...
On the Interpretation of Ensemble Classifiers in Terms of Bayes Classifiers

Many of the best classifiers are ensemble methods such as bagging, random forests, boosting, and Bayes model averaging. We give conditions under which each of these four classifiers can be regarded as a Bayes classifier. We also give conditions under ...
Ensembling neural networks: many could be better than all

Neural network ensemble is a learning paradigm where many neural networks are jointly used to solve a problem. In this paper, the relationship between the ensemble and its component neural networks is analyzed from the context of both regression and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

Neural Processing Letters Volume 12, Issue 3

Dec. 1, 2000

101 pages

ISSN:1370-4621

Editors:
François Blayo
PREFIGURE, Lyon, France
,
Mohamad H. Hassoun
Wayne State Univ., Detroit, MI
,
Michel Verleysen
Univ. Catholique de Louvain, Louvain-la-Nueve, Belgium

Issue’s Table of Contents

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 December 2000

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Stanton SIzmailov PKirichenko PAlemi AWilson ARanzato MBeygelzimer ADauphin YLiang PVaughan J(2021)Does knowledge distillation really work?Proceedings of the 35th International Conference on Neural Information Processing Systems10.5555/3540261.3540790(6906-6919)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.5555/3540261.3540790
Price RIso KShinoda K(2016)Wise teachers train better DNN acoustic modelsEURASIP Journal on Audio, Speech, and Music Processing10.1186/s13636-016-0088-72016:1(1-19)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1186/s13636-016-0088-7
Menke JMartinez T(2009)Artificial neural network reduction through oracle learningIntelligent Data Analysis10.5555/1551758.155176513:1(135-149)Online publication date: 1-Jan-2009
https://dl.acm.org/doi/10.5555/1551758.1551765
Cervantes LLee JLee J(2007)Agent-Based Approach to Distributed Ensemble Learning of Fuzzy ARTMAP ClassifiersProceedings of the 1st KES International Symposium on Agent and Multi-Agent Systems: Technologies and Applications10.1007/978-3-540-72830-6_84(805-814)Online publication date: 31-May-2007
https://dl.acm.org/doi/10.1007/978-3-540-72830-6_84
Buciluǎ CCaruana RNiculescu-Mizil AEliassi-Rad TUngar LCraven MGunopulos D(2006)Model compressionProceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining10.1145/1150402.1150464(535-541)Online publication date: 20-Aug-2006
https://dl.acm.org/doi/10.1145/1150402.1150464
Yang SBrowne APicton P(2002)Multistage Neural Network EnsemblesProceedings of the Third International Workshop on Multiple Classifier Systems10.5555/648056.744253(91-97)Online publication date: 24-Jun-2002
https://dl.acm.org/doi/10.5555/648056.744253

Abstract

References

Cited By

Index Terms

Recommendations

Class-switching neural network ensembles

On the Interpretation of Ensemble Classifiers in Terms of Bayes Classifiers

Ensembling neural networks: many could be better than all

Comments

Information

Published In

Publisher

Publication History

Author Tags

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Share

Share this Publication link

Share on social media

Affiliations