Article

Optimizing abstaining classifiers using ROC analysis

Author:

Tadeusz PietraszekAuthors Info & Claims

ICML '05: Proceedings of the 22nd international conference on Machine learning

Pages 665 - 672

https://doi.org/10.1145/1102351.1102435

Published: 07 August 2005 Publication History

Abstract

Classifiers that refrain from classification in certain cases can significantly reduce the misclassification cost. However, the parameters for such abstaining classifiers are often set in a rather ad-hoc manner. We propose a method to optimally build a specific type of abstaining binary classifiers using ROC analysis. These classifiers are built based on optimization criteria in the following three models: cost-based, bounded-abstention and bounded-improvement. We demonstrate the usage and applications of these models to effectively reduce misclassification cost in real classification systems. The method has been validated with a ROC building algorithm and cross-validation on 15 UCI KDD datasets.

References

[1]

Chow, C. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16, 41--46.]]

Digital Library

[2]

Dietterich, T. G. (1998). Approximate statistical test for comparing supervised classification learning algorithms. Neural Computation, 10, 1895 1923.]]

Digital Library

[3]

Fawcett, T. (2003). ROC graphs: Note and practical considerations for researchers (HPL-2003-4) (Technical Report). HP Laboratories.]]

[4]

Ferri, C., Flach, P., & Hernáández-Orallo, J. (2004). Delegating classifiers. Proceedings of 21th International Conference on Machine Leaning (ICML'04) (pp. 106--110). Alberta, Canada: Omnipress.]]

Digital Library

[5]

Ferri, C., & Hernáández-Orallo, J. (2004). Cautious classifiers. Proceedings of ROC Analysis in Artificial Intelligence, 1st International Workshop (ROCAI2004) (pp. 27--36). Valencia Spain.]]

[6]

Flach, P. A., & Wu. S. (2003). Repairing concavities in ROC curves. Proc. 2003 UK Workshop on Computational Intelligence (pp. 38--44). Bristol, UK.]]

[7]

Gamberger, D., & Lavra&ccaron;č, N. (2000). Reducing misclassification costs. Principles of Data Mining and Knowledge Discovery, 4th European Conference (PKDD 2000) (pp. 34--43). Lyon, France: Springer Verlag.]]

Digital Library

[8]

Hettich, S., & Bay, S. D. (1999). The UCI KDD Archive. Web page at http://kdd.ics.uci.edu.]]

[9]

Lewis, D. D., & Catlett, J. (1994). Heterogeneous uncertainty sampling for supervised learning. Proceedings of ICML-94, 11th International Conference on Machine Learning (pp. 148--156). Morgan Kaufmann Publishers, San Francisco, US.]]

[10]

Nedler, J., & Mead, R. (1965). A simplex method for function minimization. Computer Journal, 7, 308--313.]]

[11]

Pazzani, M. J., Murphy, P., Ali, K., & Schulenburg, D. (1994). Trading off coverage for accuracy in forecasts: Applications to clinical data analysis. Proceedings of AAAI Symposium on AI in Medicine (pp. 106--110). Stanford, CA.]]

[12]

Pietraszek, T. (2004). Optimizing abstaining classifiers using ROC analysis (RZ3571) (Technical Report). IBM Zurich Research Laboratory.]]

[13]

Provost, F., & Fawcett, T. (1998). Robust classification systems for imprecise environemnts. Proceedings of the Fifteenth National Conference on Artificial Intellignence (AAAI-98) (pp. 706--713). AAAI Press.]]

Digital Library

[14]

Tortorella, F. (2000). An optimal reject rule for binary classifiers. Advances in Pattern Recognition, Joint IAPR International Workshops SSPR 2000 and SPR 2000 (pp. 611--620). Alicante, Spain: Springer-Verlag.]]

Digital Library

[15]

Witten, I. H., & Frank, E. (2000). Data Mining: Practical machine learning tools with Java implementations. San Francisco: Morgan Kaufmann.]]

Digital Library

Cited By

Dakshit S(2024)Abstaining ECG Classifiers Through Explainable Prototypical Spaces2024 IEEE 12th International Conference on Healthcare Informatics (ICHI)10.1109/ICHI61247.2024.00104(651-659)Online publication date: 3-Jun-2024
https://doi.org/10.1109/ICHI61247.2024.00104
Hendrickx KPerini LVan der Plas DMeert WDavis J(2024)Machine learning with a reject option: a surveyMachine Learning10.1007/s10994-024-06534-x113:5(3073-3110)Online publication date: 29-Mar-2024
https://doi.org/10.1007/s10994-024-06534-x
Buschjäger S(2024)Rejection Ensembles with Online CalibrationMachine Learning and Knowledge Discovery in Databases. Research Track10.1007/978-3-031-70365-2_1(3-20)Online publication date: 22-Aug-2024
https://doi.org/10.1007/978-3-031-70365-2_1
Show More Cited By

Optimizing abstaining classifiers using ROC analysis
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches

Recommendations

On the use of ROC analysis for the optimization of abstaining classifiers

Classifiers that refrain from classification in certain cases can significantly reduce the misclassification cost. However, the parameters for such abstaining classifiers are often set in a rather ad-hoc manner. We propose a method to optimally build a ...
An alternative to ROC and AUC analysis of classifiers
IDA'11: Proceedings of the 10th international conference on Advances in intelligent data analysis X

Performance evaluation of classifiers is a crucial step for selecting the best classifier or the best set of parameters for a classifier. The misclassification rate of a classifier is often too simple because it does not take into account that ...
Classification of intrusion detection alerts using abstaining classifiers

Intrusion Detection Systems have been observed to trigger an abundance of false positives, that is alerts not reporting security problems. Assuming that in real installations most of the alerts are reviewed by human security analysts in a timely manner, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '05: Proceedings of the 22nd international conference on Machine learning

August 2005

1113 pages

ISBN:1595931805

DOI:10.1145/1102351

General Chair:
Saso Dzeroski
Jozef Stefan Institute, Slovenia
,
Program Chairs:
Luc De Raedt,
Stefan Wrobel

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
304
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dakshit S(2024)Abstaining ECG Classifiers Through Explainable Prototypical Spaces2024 IEEE 12th International Conference on Healthcare Informatics (ICHI)10.1109/ICHI61247.2024.00104(651-659)Online publication date: 3-Jun-2024
https://doi.org/10.1109/ICHI61247.2024.00104
Hendrickx KPerini LVan der Plas DMeert WDavis J(2024)Machine learning with a reject option: a surveyMachine Learning10.1007/s10994-024-06534-x113:5(3073-3110)Online publication date: 29-Mar-2024
https://doi.org/10.1007/s10994-024-06534-x
Buschjäger S(2024)Rejection Ensembles with Online CalibrationMachine Learning and Knowledge Discovery in Databases. Research Track10.1007/978-3-031-70365-2_1(3-20)Online publication date: 22-Aug-2024
https://doi.org/10.1007/978-3-031-70365-2_1
Fischer LWollstadt P(2024)Precision and Recall Reject CurvesAdvances in Self-Organizing Maps, Learning Vector Quantization, Interpretable Machine Learning, and Beyond10.1007/978-3-031-67159-3_19(163-173)Online publication date: 1-Aug-2024
https://doi.org/10.1007/978-3-031-67159-3_19
Cheng XCao YWang HWei HAn BFeng LOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Regression with cost-based rejectionProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668080(45172-45196)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3668080
Choe YGangrade ARamdas AOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Counterfactually comparing abstaining classifiersProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3667351(28281-28293)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3667351
Pugnana ARuggieri SWilliams BChen YNeville J(2023)A model-agnostic heuristics for selective classificationProceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v37i8.26133(9461-9469)Online publication date: 7-Feb-2023
https://dl.acm.org/doi/10.1609/aaai.v37i8.26133
Cortes CDeSalvo GMohri M(2023)Theory and algorithms for learning with rejection in binary classificationAnnals of Mathematics and Artificial Intelligence10.1007/s10472-023-09899-292:2(277-315)Online publication date: 13-Dec-2023
https://doi.org/10.1007/s10472-023-09899-2
Kalra BManwani N(2023)CDAN: Cost Dependent Deep Abstention NetworkPRICAI 2023: Trends in Artificial Intelligence10.1007/978-981-99-7022-3_5(53-65)Online publication date: 10-Nov-2023
https://doi.org/10.1007/978-981-99-7022-3_5
Pham TCoustaty MJoseph APoulain d’Andecy VVisani MSidere N(2023)Incremental Learning and Ambiguity Rejection for Document ClassificationDocument Analysis and Recognition - ICDAR 202310.1007/978-3-031-41734-4_2(18-35)Online publication date: 19-Aug-2023
https://doi.org/10.1007/978-3-031-41734-4_2
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents