Abstract
Cross-validation is a popular technique for model selection and evaluation. The purpose is to provide an estimate of generalization error using mean error over test folds. Typical recommendation is to use ten-fold stratified cross-validation in classification problems. In this paper, we perform a set of experiments to explore the characteristics of cross-validation, when dealing with model evaluation of Multilayer Perceptron neural network. We test two variants of stratification, where the nonstandard one takes into account classwise data density in addition to pure class frequency. Based on computational experiments, many common beliefs are challenged and some interesting conclusions drawn.
Chapter PDF
Similar content being viewed by others
References
Elisseeff, A., Pontil, M.: Leave-one-out error and stability of learning algorithms with applications. NATO Science Series, Sub Series III: Computer and Systems Sciences 190, 111–130 (2003)
Pinkus, A.: Approximation theory of the MLP model in neural networks. Acta Numerica, 143–195 (1999)
Kohavi, R.: Study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 1995), pp. 1137–1143 (1995)
Witten, I.H., Frank, E., Hall, M.A.: Data Mining: Practical Machine Learning Tools and Techniques, 3rd edn. Morgan Kaufmann, Burlington (2011)
Borra, S., Ciaccio, A.D.: Measuring the prediction error. a comparison of cross-validation, bootstrap and covariance penalty methods. Computational Statistics and Data Analysis 54, 2976–2989 (2010)
Davison, A.C., Hinkley, D.V.: Bootstrap Methods and their Applications. Cambridge Series on Statistical and Probabilistic Mathematics. Cambridge University Press (1997)
Breiman, L.: Heuristics of instability and stabilization in model selection. The Annals of Statistics 24(6), 2350–2383 (1996)
Andersen, T., Martinez, T.: Cross validation and MLP architecture selection. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN 1999), pp. 1614–1619 (1999)
Last, M.: The uncertainty principle of cross-validation. In: Proceedings of the IEEE International Conference on Granular Computing (GrC 2006), pp. 275–280 (2006)
Arlot, S., Celisse, A.: A survey of cross-validation procedures for model selection. Statistics Surveys 4, 40–79 (2010)
Moreno-Torres, J.G., Sáez, J.A., Herrera, F.: Study on the impact of partition-induced dataset shift on k-fold cross-validation. IEEE Transactions on Neural Networks and Learning Systems 23(8), 1304–1312 (2012)
Kärkkäinen, T.: MLP in layer-wise form with applications to weight decay. Neural Computation 14, 1451–1480 (2002)
López, V., Fernández, A., Herrera, F.: On the importance of the validation technique for classification with imbalanced datasets: Addressing covariate shift when data is skewed. Information Sciences 257, 1–13 (2014)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kärkkäinen, T. (2014). On Cross-Validation for MLP Model Evaluation. In: Fränti, P., Brown, G., Loog, M., Escolano, F., Pelillo, M. (eds) Structural, Syntactic, and Statistical Pattern Recognition. S+SSPR 2014. Lecture Notes in Computer Science, vol 8621. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-44415-3_30
Download citation
DOI: https://doi.org/10.1007/978-3-662-44415-3_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-44414-6
Online ISBN: 978-3-662-44415-3
eBook Packages: Computer ScienceComputer Science (R0)