Abstract
We investigate the problem of using past performance information to select an algorithm for a given classification problem. We present three ranking methods for that purpose: average ranks, success rate ratios and significant wins. We also analyze the problem of evaluating and comparing these methods. The evaluation technique used is based on a leave-one-out procedure. On each iteration, the method generates a ranking using the results obtained by the algorithms on the training datasets. This ranking is then evaluated by calculating its distance from the ideal ranking built using the performance information on the test dataset. The distance measure adopted here, average correlation, is based on Spearman’s rank correlation coefficient. To compare ranking methods, a combination of Friedman’s test and Dunn’s multiple comparison procedure is adopted. When applied to the methods presented here, these tests indicate that the success rate ratios and average ranks methods perform better than significant wins.
Chapter PDF
Similar content being viewed by others
References
D.W. Aha. Generalizing from case studies: A case study. In D. Sleeman and P. Edwards, editors, Proceedings of the Ninth International Workshop on Machine Learning (ML92), pages 1–10. Morgan Kaufmann, 1992. 72
C. Blake, E. Keogh, and C.J. Merz. Repository of machine learning databases, 1998. http://www.ics.uci.edu/~mlearn/MLRepository.html. 65
R.J. Brachman and T. Anand. The process of knowledge discovery in databases. In U.M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, editors, Advances in Knowledge Discovery and Data Mining, chapter 2, pages 37–57. AAAI Press/The MIT Press, 1996. 64
P. Brazdil, J. Gama, and B. Henery. Characterizing the applicability of classification algorithms using meta-level learning. In F. Bergadano and L. de Raedt, editors, Proceedings of the European Conference on Machine Learning (ECML-94), pages 83–102. Springer-Verlag, 1994. 72
C.E. Brodley. Addressing the selective superiority problem: Automatic Algorithm/Model class selection. In P. Utgoff, editor, Proceedings of the 10th International Conference on Machine Learning, pages 17–24. Morgan Kaufmann, 1993. 72
W. Daelemans, J. Zavrel, K. Van der Sloot, and A. Van Den Bosch. TiMBL: Tilburg memory based learner. Technical Report 99-01, ILK, 1999. 64
T.G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895–1924, 1998. ftp://ftp.cs.orst.edu/pub/tgd/papers/nc-stats.ps.gz. 72, 73
U.M. Fayyad, G. Piatetsky-Shapiro, and P. Smyth. From data mining to knowledge discovery: An overview. In U.M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, editors, Advances in Knowledge Discovery and Data Mining, chapter 1, pages 1–34. AAAI Press/The MIT Press, 1996. 73
J. Gama. Probabilistic linear tree. In D. Fisher, editor, Proceedings of the 14th International Machine Learning Conference (ICML97), Morgan Kaufmann, 1997. 64
J. Gama and P. Brazdil. Characterization of classification algorithms. In C. Pinto-Ferreira and N.J. Mamede, editors, Progress in Artificial Intelligence, pages 189–200. Springer-Verlag, 1995. 72, 73
A. Kalousis and T. Theoharis. NOEMON: Design, implementation and performance results of an intelligent assistant for classifier selection. Intelligent Data Analysis, 3(5):319–337, November 1999. 72
D. Michie, D.J. Spiegelhalter, and C.C. Taylor. Machine Learning, Neural and Statistical Classification. Ellis Horwood, 1994. 63, 64, 72, 73
H.R. Neave and P.L. Worthington. Distribution-Free Tests. Routledge, 1992. 65, 67, 68, 70
F. Provost and D. Jensen. Evaluating knowledge discovery and data mining. Tutorial Notes, Fourth International Conference on Knowledge Discovery and Data Mining, 1998. 64
R. Quinlan. C5.0: An Informal Tutorial. RuleQuest, 1998. 64
C. Soares. Ranking classification algorithms on past performance. Master’s thesis, Faculty of Economics, University of Porto, 1999. http://www.ncc.up.pt/~csoares/miac/thesis_revised.zip. 70, 73
C. Soares, P. Brazdil, and J. Costa. Measures to compare rankings of classification algorithms. In Proceedings of the 7th IFCS, 2000. 72, 73
L. Todorovski and S. Dzeroski. Experiments in meta-level learning with ILP. In Proceedings of PKDD99, 1999. 72
D.H. Wolpert and W.G. Macready. No free lunch theorems for search. Technical Report SFI-TR-95-02-010, The Santa Fe Institute, 1996. 63, 73
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Brazdil, P.B., Soares, C. (2000). A Comparison of Ranking Methods for Classification Algorithm Selection. In: López de Mántaras, R., Plaza, E. (eds) Machine Learning: ECML 2000. ECML 2000. Lecture Notes in Computer Science(), vol 1810. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45164-1_8
Download citation
DOI: https://doi.org/10.1007/3-540-45164-1_8
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-67602-7
Online ISBN: 978-3-540-45164-8
eBook Packages: Springer Book Archive