Article

Volume under the ROC surface for multi-class problems

Authors:

José Hernández-Orallo,

Miguel Angel SalidoAuthors Info & Claims

ECML'03: Proceedings of the 14th European Conference on Machine Learning

Pages 108 - 120

https://doi.org/10.1007/978-3-540-39857-8_12

Published: 22 September 2003 Publication History

Abstract

Operating Characteristic (ROC) analysis has been successfully applied to classifier problems with two classes. The Area Under the ROC Curve (AUC) has been elected as a better way to evaluate classifiers than predictive accuracy or error and has also recently used for evaluating probability estimators. However, the extension of the Area Under the ROC Curve for more than two classes has not been addressed to date, because of the complexity and elusiveness of its precise definition. Some approximations to the real AUC are used without an exact appraisal of their quality. In this paper, we present the real extension to the Area Under the ROC Curve in the form of the Volume Under the ROC Surface (VUS), showing how to compute the polytope that corresponds to the absence of classifiers (given only by the trivial classifiers), to the best classifier and to whatever set of classifiers. We compare the real VUS with "approximations" or "extensions" of the AUC for more than two classes.

References

[1]

Adams, N.M., Hand, D.J.: Comparing classifiers when the misallocation costs are uncertain. Pattern Recognition 32(7), 1139-1147 (1999)

[2]

Barber, C.B., Huhdanpaa, H.: "QHull", The Geometry Center, University of Minnesota, http://www.geom.umn.edu/software/qhull/

[3]

Boissonat, J.D., Yvinec, M.: Algorithmic Geometry. Cambridge University Press, Cambridge (1998)

[4]

Ferri, C., Hernández-Orallo, J., Salido, M.A.: Volume Under the ROC Surface for Multiclass Problems. Exact Computation and Evaluation of Approximations. Technical Report DSIC. Univ. Politèc. València (2003), http://www.dsic.upv.es/users/elp/cferri/VUS.pdf

[5]

Flach, P., Blockeel, H., Ferri, C., Hernández-Orallo, J., Struyf, J.: Decision support for data mining; Introduction to ROC analysis and its applications. In: Data Mining and Decision Support: Integration and Collaboration, Kluwer Publishers, Dordrecht (2003) (to appear)

[6]

Hand, D.J., Till, R.J.: A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems. Machine Learning 45, 171-186 (2001)

[7]

Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29-36 (1982)

[8]

Lane, T.: Extensions of ROC Analysis to Multi-Class Domains. In: ICML 2000 Workshop on cost-sensitive learning (2000)

[9]

Provost, F., Fawcett, T.: Analysis and visualization of classifier performance: Comparison under imprecise class and cost distribution. In: Proc. of The Third International Conference on Knowledge Discovery and Data Mining (KDD 1997), pp. 43-48. AAAI Press, Menlo Park (1997)

[10]

Provost, F., Domingos, P.: Tree Induction for Probability-based Ranking. Machine Learning 52(3), 199-215 (2003)

[11]

Salido, M.A., Giret, A., Barber, F.: Constraint Satisfaction by means of Dynamic Polyhedra. In: Operations Research Proceedings 2001, pp. 405-412. Springer, Heidelberg (2002)

[12]

Srinivasan, A.: Note on the Location of Optimal Classifiers in N-dimensional ROC Space. Technical Report PRG-TR-2-99, Oxford University Computing Laboratory

[13]

Swets, J., Dawes, R., Monahan, J.: Better decisions through science. Scientific American, 82-87 (October 2000)

[14]

Turney, P.: Cost-sensitive classification: Empirical evaluation of a hybrid genetic decision tree induction algorithm. Journal of Artificial Intelligence Research 2, 369-409 (1995)

[15]

Zweig, M.H., Campbell, G.: Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin. Chem. 39, 561-577 (1993)

Cited By

Maser BUhl A(2021)Identifying the Origin of Finger Vein Samples Using Texture DescriptorsComputational Science and Its Applications – ICCSA 202110.1007/978-3-030-86960-1_17(237-250)Online publication date: 13-Sep-2021
https://dl.acm.org/doi/10.1007/978-3-030-86960-1_17
Song YLee JTavares JXu Z(2019)Design of stock price prediction model with various configuration of input featuresProceedings of the International Conference on Artificial Intelligence, Information Processing and Cloud Computing10.1145/3371425.3371432(1-5)Online publication date: 19-Dec-2019
https://dl.acm.org/doi/10.1145/3371425.3371432
Majnik MBosnić Z(2013)ROC analysis of classifiers in machine learningIntelligent Data Analysis10.5555/2595566.259557617:3(531-558)Online publication date: 1-May-2013
https://dl.acm.org/doi/10.5555/2595566.2595576
Show More Cited By

Volume under the ROC surface for multi-class problems
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches

Recommendations

Comparing multi-class classifier performance by multi-class ROC analysis: A nonparametric approach
Abstract
The area under the Receiver Operating Characteristic (ROC) curve (AUC) is a standard metric for quantifying and comparing binary classifiers. Real world applications often require classification into multiple (more than two) classes. For multi-...
Training multiclass classifiers by maximizing the volume under the ROC surface
EUROCAST'07: Proceedings of the 11th international conference on Computer aided systems theory

Receiver operating characteristic (ROC) curves are a plot of a ranking classifier's true-positive rate versus its false-positive rate, as one varies the threshold between positive and negative classifications across the continuum. The area under the ROC ...
Towards maximizing the area under the ROC curve for multi-class classification problems
AAAI'11: Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence

The Area Under the ROC Curve (AUC) metric has achieved a big success in binary classification problems since they measure the performance of classifiers without making any specific assumptions about the class distribution and misclassification costs. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

ECML'03: Proceedings of the 14th European Conference on Machine Learning

September 2003

502 pages

ISBN:3540201211

Editors:
Nada Lavrač
University of Nova Gorica, Nova Gorica, Slovenia
,
Dragan Gamberger
Rudjer Bošković Institute, Zagreb, Croatia
,
Hendrik Blockeel
Leiden Institute of Advanced Computer Science, Leiden University, Leuven, Belgium
,
Ljupčo Todorovski
Jozef Stefan Institute, Ljubljana, Slovenia

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 22 September 2003

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Maser BUhl A(2021)Identifying the Origin of Finger Vein Samples Using Texture DescriptorsComputational Science and Its Applications – ICCSA 202110.1007/978-3-030-86960-1_17(237-250)Online publication date: 13-Sep-2021
https://dl.acm.org/doi/10.1007/978-3-030-86960-1_17
Song YLee JTavares JXu Z(2019)Design of stock price prediction model with various configuration of input featuresProceedings of the International Conference on Artificial Intelligence, Information Processing and Cloud Computing10.1145/3371425.3371432(1-5)Online publication date: 19-Dec-2019
https://dl.acm.org/doi/10.1145/3371425.3371432
Majnik MBosnić Z(2013)ROC analysis of classifiers in machine learningIntelligent Data Analysis10.5555/2595566.259557617:3(531-558)Online publication date: 1-May-2013
https://dl.acm.org/doi/10.5555/2595566.2595576
Wang RTang K(2012)Feature selection for MAUC-oriented classification systemsNeurocomputing10.1016/j.neucom.2012.01.01389(39-54)Online publication date: 1-Jul-2012
https://dl.acm.org/doi/10.1016/j.neucom.2012.01.013
Ivanescu AWichterich MSeidl T(2011)ClasSiProceedings of the 15th international conference on New Frontiers in Applied Data Mining10.1007/978-3-642-28320-8_16(185-196)Online publication date: 24-May-2011
https://dl.acm.org/doi/10.1007/978-3-642-28320-8_16
Landgrebe TPaclik P(2010)The ROC skeleton for multiclass ROC estimationPattern Recognition Letters10.1016/j.patrec.2009.12.03731:9(949-958)Online publication date: 1-Jul-2010
https://dl.acm.org/doi/10.1016/j.patrec.2009.12.037
Waegeman WDe Baets BBoullart L(2008)On the scalability of ordered multi-class ROC analysisComputational Statistics & Data Analysis10.1016/j.csda.2007.12.00152:7(3371-3388)Online publication date: 1-Mar-2008
https://dl.acm.org/doi/10.1016/j.csda.2007.12.001
Drummond CHolte R(2006)Cost curvesMachine Language10.1007/s10994-006-8199-565:1(95-130)Online publication date: 1-Oct-2006
https://dl.acm.org/doi/10.1007/s10994-006-8199-5
Zhang XJiang CLuo M(2006)Training classifiers for unbalanced distribution and cost-sensitive domains with ROC analysisProceedings of the 9th Pacific Rim Knowledge Acquisition international conference on Advances in Knowledge Acquisition and Management10.1007/11961239_8(89-98)Online publication date: 7-Aug-2006
https://dl.acm.org/doi/10.1007/11961239_8
Landgrebe TPaclík PTax DDuin R(2005)Optimising two-stage recognition systemsProceedings of the 6th international conference on Multiple Classifier Systems10.1007/11494683_21(206-215)Online publication date: 13-Jun-2005
https://dl.acm.org/doi/10.1007/11494683_21

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents