research-article

The Multiclass ROC Front method for cost-sensitive classification

Authors:

Clément Chatelain,

Sébastien Adam,

Robert SabourinAuthors Info & Claims

Pattern Recognition, Volume 52, Issue C

Pages 46 - 60

https://doi.org/10.1016/j.patcog.2015.10.010

Published: 01 April 2016 Publication History

Abstract

This paper addresses the problem of learning a multiclass classification system that can suit to any environment. By that we mean that particular (imbalanced) misclassification costs are taken into account by the classifier for predictions. However, these costs are not well known during the learning phase in most cases, or may evolve afterwards. There is a need in that case to learn a classifier that can potentially suit to any of these costs in prediction phase. The learning method proposed in this work, named the Multiclass ROC Front (MROCF) method, responds to this issue by exploiting ROC-based tools through a multiobjective optimization process. While this type of ROC-based multiobjective optimization approach has been successfully used for two-class problems, it has never been proposed in real-world multiclass classification problems. Experiments led on several real-world datasets show that the MROCF method offers a major improvement over a cost-insensitive classifier and is competitive with the state-of-the-art cost-sensitive optimization method on all but one of the 20 datasets. HighlightsWe propose a new method for multiclass cost-sensitive classification when misclassification costs are unknown during training.It is based on a multi-model approach and can suit to any cost-sensitive environment in prediction.It makes use of ROC-based multi-objective optimization algorithms.The method is compared to a cost-insensitive method and a state-of-the-art cost-sensitive optimization method.It outperforms both methods for most of the datasets tested.

References

[1]

P. Cao, D. Zhao, O.R. Zaane, An optimized cost-sensitive svm for imbalanced data learning, in: PAKDD (2), Lecture Notes in Computer Science, vol. 7819, Springer, Berlin, Heidelberg, 2013, pp. 280-292.

[2]

J. Xu, Y. Cao, H. Li, Y. Huang, Cost-sensitive learning of svm for ranking, in: Machine Learning: ECML 2006, Lecture Notes in Computer Science, vol. 4212, Springer, Berlin, Heidelberg, 2006, pp. 833-840.

[3]

S. Lomax, S. Vadera, A survey of cost-sensitive decision tree induction algorithms, ACM Comput. Surv., 45 (2013).

[4]

C. Drummond, R.C. Holte, C4.5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling, in: ICML Workshop on Learning from Imbalanced Datasets II, 2003, pp. 1-8.

[5]

C. Ferri, P. Flach, J. Hernandez-Orallo, Learning decision trees using the area under the ROC curve, in: Proceedings of the 19th International Conference on Machine Learning, 2002, pp. 139-146.

[6]

Y. Freund, R. Iyer, R. Schapire, Y. Singer, An efficient boosting algorithm for combining preferences, J. Mach. Learn. Res., 4 (2003) 933-969.

Digital Library

[7]

A. Rakotomamonjy, Optimizing area under Roc curve with SVMs, in: Workshop on ROC Analysis in Artificial Intelligence, 2004, pp. 469-478.

[8]

U. Brefeld, T. Scheffer, AUC maximizing support vector learning, in: Proceedings of ICML Workshop on ROC Analysis in Machine Learning, 2005.

[9]

C. Chatelain, S. Adam, Y. Lecourtier, L. Heutte, T. Paquet, A multi-model selection framework for unknown and/or evolutive misclassification cost problems, Pattern Recognit., 43 (2010) 815-823.

Digital Library

[10]

J.-C. Levesque, A. Durand, C. Gagne, R. Sabourin, Multi-objective evolutionary optimization for generating ensembles of classifiers in the ROC space, in: Proceedings of the Fourteenth International Conference on Genetic and Evolutionary Computation Conference, 2012, pp. 879-886.

[11]

T. Landgrebe, P. Paclik, The ROC skeleton for multiclass ROC estimation, Pattern Recognit. Lett., 31 (2010) 949-958.

Digital Library

[12]

T. Fawcett, An introduction to roc analysis, Pattern Recognit. Lett., 27 (2006) 861-874.

Digital Library

[13]

T. Fawcett, ROC Graphs: Notes and Practical Considerations for Researchers, Technical Report, HP Laboratories, 2004.

[14]

R. Everson, J. Fieldsend, Multi-class ROC analysis from a multi-objective optimisation perspective, Pattern Recognit. Lett. (2006) 918-927.

[15]

C. Bourke, K. Deng, S.D. Scott, R. Schapire, N. Vinodchandran, On reoptimizing multi-class classifiers, Mach. Learn., 71 (2008) 219-242.

Digital Library

[16]

D. Hand, R. Till, A simple generalisation of the area under the ROC curve for multiple class classification problems, Mach. Learn., 45 (2001) 171-186.

[17]

C. Ferri, J. Hernández-orallo, M. Salido, Volume under the ROC surface for multi-class problems. Exact computation and evaluation of approximations, in: Proceedings of 14th European Conference on Machine Learning, 2003, pp. 108-120.

[18]

T. Landgrebe, R.P.W. Duin, A simplified extension of the area under the ROC to the multiclass domain, in: 17th Annual Symposium of the Pattern Recognition Association of South Africa, 2006, pp. 241-245.

[19]

N. Lachiche, P. Flach, Improving accuracy and cost of two-class and multi-class probabilistic classifiers using ROC curves, in: Proceedings of 20th International Conference on Machine Learning (ICML'03), 2003, pp. 416-423.

[20]

T. Landgrebe, R. Duin, Approximating the multiclass ROC by pairwise analysis, Pattern Recognit. Lett., 28 (2007) 1747-1758.

Digital Library

[21]

J. Fieldsend, R. Everson, Multiobjective supervised learning, in: Multiobjective Problem Solving from Nature, Natural Computing Series, Springer, Berlin, Heidelberg, 2008, pp. 155-176.

[22]

M. Kupinski, M. Anastasio, Optimization and FROC analysis of rule-based detection schemes using a multiobjective approach, IEEE Trans. Med. Imaging, 18 (1999) 675-685.

[23]

J. Horn, N. Nafpliotis, D. Goldberg, A niched pareto genetic algorithm for multiobjective optimization, in: Proceedings of the First IEEE Conference on Evolutionary Computation, 1994. IEEE World Congress on Computational Intelligence, vol. 1, 1994, pp. 82-87.

[24]

K. Deb, A. Pratap, S. Agarwal, T. Meyarivan, A fast elitist multi-objective genetic algorithm, IEEE Trans. Evolut. Comput., 6 (2002) 182-197.

Digital Library

[25]

W. Khreich, E. Granger, A. Miri, R. Sabourin, Iterative boolean combination of classifiers in the roc space: an application to anomaly detection with hmms, Pattern Recognit., 43 (2010) 2732-2752.

Digital Library

[26]

P.V. Radtke, E. Granger, R. Sabourin, D.O. Gorodnichy, Skew-sensitive boolean combination for adaptive ensembles an application to face recognition in video surveillance, Inf. Fusion, 20 (2014) 31-48.

[27]

R.P.W. Duin, P. Juszczak, D. de Ridder, P. Paclik, E. Pekalska, D.M. Tax, PRTools, a Matlab Toolbox for Pattern Recognition. URL {http://www.prtools.org}, 2004.

[28]

K. Deb, Multi-Objective Optimization Using Evolutionary Algorithms, John Wiley & Sons, Inc., New York, NY, USA, 2001.

[29]

J. Bergstra, Y. Bengio, Random search for hyper-parameter optimization, J. Mach. Learn. Res., 13 (2012) 281-305.

Digital Library

[30]

Perclass 4.3: Matlab Toolbox for Pattern Recognition. URL {http://perclass.com/index.php/html/}, 2014.

[31]

K. Bache, M. Lichman, UCI Machine Learning Repository. URL {http://archive.ics.uci.edu/ml}, 2013.

[32]

S. Bernard, S. Adam, L. Heutte, Using random forests for handwritten digit recognition, in: Ninth International Conference on Document Analysis and Recognition, 2007. ICDAR 2007, vol. 2, 2007, pp. 1043-1047.

[33]

T.-F. Wu, C.-J. Lin, R.C. Weng, Probability estimates for multi-class classification by pairwise coupling, J. Mach. Learn. Res., 5 (2004) 975-1005.

Digital Library

[34]

R.A. McDonald, The mean subjective utility score, a novel metric for cost-sensitive classifier evaluation, Pattern Recognit. Lett., 27 (2006) 1472-1477.

Digital Library

[35]

J. Demšar, Statistical comparisons of classifiers over multiple data sets, J. Mach. Learn. Res., 7 (2006) 1-30.

Digital Library

Cited By

Karl FPielok TMoosbauer JPfisterer FCoors SBinder MSchneider LThomas JRichter JLang MGarrido-Merchán EBranke JBischl B(2023)Multi-Objective Hyperparameter Optimization in Machine Learning – An OverviewACM Transactions on Evolutionary Learning and Optimization10.1145/3610536Online publication date: 5-Sep-2023
https://dl.acm.org/doi/10.1145/3610536
Salari NLiu SShen Z(2022)Real-Time Delivery Time Forecasting and Promising in Online RetailingManufacturing & Service Operations Management10.1287/msom.2022.108124:3(1421-1436)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1287/msom.2022.1081
Krawczyk BCano A(2021)Locally Linear Support Vector Machines for Imbalanced Data ClassificationAdvances in Knowledge Discovery and Data Mining10.1007/978-3-030-75762-5_49(616-628)Online publication date: 11-May-2021
https://dl.acm.org/doi/10.1007/978-3-030-75762-5_49
Show More Cited By

The Multiclass ROC Front method for cost-sensitive classification
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches

Recommendations

Advances in Cost-sensitive Multiclass and Multilabel Classification
KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Classification is an important problem for data mining and knowledge discovery and comes with a wide range of applications. Different applications usually evaluate the classification performance with different criteria. The variety of criteria calls for ...
Comparative study of classifier ensembles for cost-sensitive credit risk assessment

Ensemble is a recently emerged computing technique to provide promising decisions by a consensus of multiple classifiers. The benefit of classifier ensembles has been demonstrated in a vast number of studies in the scope of credit risk management. Yet ...
Learning ECOC Code Matrix for Multiclass Classification with Application to Glaucoma Diagnosis

Classification of different mechanisms of angle closure glaucoma (ACG) is important for medical diagnosis. Error-correcting output code (ECOC) is an effective approach for multiclass classification. In this study, we propose a new ensemble learning ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Pattern Recognition

Pattern Recognition Volume 52, Issue C

April 2016

477 pages

ISSN:0031-3203

Issue’s Table of Contents

Copyright © Elsevier Ltd.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 April 2016

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Karl FPielok TMoosbauer JPfisterer FCoors SBinder MSchneider LThomas JRichter JLang MGarrido-Merchán EBranke JBischl B(2023)Multi-Objective Hyperparameter Optimization in Machine Learning – An OverviewACM Transactions on Evolutionary Learning and Optimization10.1145/3610536Online publication date: 5-Sep-2023
https://dl.acm.org/doi/10.1145/3610536
Salari NLiu SShen Z(2022)Real-Time Delivery Time Forecasting and Promising in Online RetailingManufacturing & Service Operations Management10.1287/msom.2022.108124:3(1421-1436)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1287/msom.2022.1081
Krawczyk BCano A(2021)Locally Linear Support Vector Machines for Imbalanced Data ClassificationAdvances in Knowledge Discovery and Data Mining10.1007/978-3-030-75762-5_49(616-628)Online publication date: 11-May-2021
https://dl.acm.org/doi/10.1007/978-3-030-75762-5_49
Lei LSong YLuo X(2020)A new re-encoding ECOC using reject optionApplied Intelligence10.1007/s10489-020-01642-250:10(3090-3100)Online publication date: 1-Oct-2020
https://dl.acm.org/doi/10.1007/s10489-020-01642-2
Singh VSharma TVerma NCui Y(2019)Feature Ranking using Robust Fuzzy Score Function for Gene Expression Data2019 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)10.1109/FUZZ-IEEE.2019.8858907(1-6)Online publication date: 23-Jun-2019
https://dl.acm.org/doi/10.1109/FUZZ-IEEE.2019.8858907
Krawczyk BWozniak M(2019)On the Role of Cost-Sensitive Learning in Imbalanced Data OversamplingComputational Science – ICCS 201910.1007/978-3-030-22744-9_14(180-191)Online publication date: 12-Jun-2019
https://dl.acm.org/doi/10.1007/978-3-030-22744-9_14
Zhao ZWang X(2018)Cost-sensitive SVDD models based on a sample selection approachApplied Intelligence10.5555/3288064.328809448:11(4247-4266)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.5555/3288064.3288094
Li FZhang XZhang XDu CXu YTian Y(2018)Cost-sensitive and hybrid-attribute measure multi-decision tree over imbalanced data setsInformation Sciences: an International Journal10.1016/j.ins.2017.09.013422:C(242-256)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1016/j.ins.2017.09.013
Cruz RSabourin RCavalcanti G(2018)Dynamic classifier selectionInformation Fusion10.1016/j.inffus.2017.09.01041:C(195-216)Online publication date: 1-May-2018
https://dl.acm.org/doi/10.1016/j.inffus.2017.09.010
Borlikova GO'Neill MSmith LPhillips M(2017)Development of a multi-model system to accommodate unknown misclassification costs in prediction of patient recruitment in multicentre clinical trialsProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3067695.3076062(263-264)Online publication date: 15-Jul-2017
https://dl.acm.org/doi/10.1145/3067695.3076062

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents