Abstract
We review linear statistical models for the analysis of computational experiments on optimization algorithms. The models offer the mathematical framework to separate the effects of algorithmic components and instance features included in the analysis. We regard test instances as drawn from a population and we focus our interest not on those single instances but on the whole population. Hence, instances are treated as a random factor. Overall these experimental designs lead to mixed effects linear models. We present both the theory to justify these models and a computational example in which we analyze and comment on several possible experimental designs. The example is a component-wise analysis of local search algorithms for the 2-edge-connectivity augmentation problem. We use standard statistical software to perform the analysis and report the R commands. Data sets and the analysis in SAS are available in an online compendium.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Bang-Jensen J, Chiarandini M, Morling P (2009) A computational investigation of heuristic algorithms for 2-edge-connectivity augmentation.Networks (In print.)
Barr R, Golden B, Kelly J, Resende M, Stewart W (1995) Designing and reporting on computational experiments with heuristic methods.Journal of Heuristics 1(1):9–32
Bates D (2007) Personal Communication
Bates D, Maechler M, Dai B (2008) lme4:Linear mixed-effects models using S4 classes. URL http://lme4.r-forge.r-project.org/, R package version 0.999375-28
Birattari M (2004) On the estimation of the expected performance of a metaheuristic on a class of instances. How many instances, how many runs? Tech. Rep. TR/IRIDIA/2004-01, IRIDIA, Université Libre de Bruxelles, Brussels, Belgium
Bondy J, Murty U (2008) Graph Theory. Graduate Texts in Mathematics, Vol. 244,Springer
Chiarandini M (2005) Stochastic local search methods for highly constrained combinatorial optimisation problems. PhD thesis, Computer Science Department, Darmstadt University of Technology, Darmstadt, Germany
Coffin M, Saltzman MJ (2000)Statistical analysis of computational tests of algorithms and heuristics. INFORMS Journal on Computing 12(1):24–44
Conforti M, Galluccio A, Proietti G (2004) Edge-connectivity augmentation and network matrices. In:Workshop on Graph-Theoretic Concepts in Computer Science, Springer, Lecture Notes in Computer Science, vol 3353, pp 355–364
Cormen T, Leiserson C, Rivest R (2001) Introduction to Algorithms, 2nd edn.MIT press
Czarn A, MacNish C, Vijayan K, Turlach B, Gupta R (2004) Statistical exploratory analysis of genetic algorithms. Evolutionary Computation, IEEE Transactions on 8(4):405–421
Fox J (2002) Linear mixed models. Appendix to An R and S-PLUS Companion to Applied Regression,URL http://cran.r-project.org/doc/ contrib/Fox-Companion/appendix-mixed-models.pdf
Glover F, Kochenberger G (eds) (2002) Handbook of Metaheuristics, International Series in Operations Research & Management Science, vol 57. Kluwer Academic, Norwell, MA
Hooker JN (1996) Testing heuristics: We have it all wrong. Journal of Heuristics 1:32–42
Johnson R, Wichern D (2007) Applied Multivariate Statistical Analysis, 6th edn.Prentice-Hall International
MKutner MH, Nachtsheim CJ, Neter J, Li W (2005) Applied Linear Statistical Models, 5th edn. McGraw-Hill
Lehmann E (2003) Theory of point estimation.Springer
Lehmann E, Romano J (2008) Testing statistical hypothesis.Springer
Lenth RV (2006) Java applets for power and sample size [computer software]. Retrieved 29 January 2009 from http://www.stat.uiowa.edu/˜rlenth/ Power
Lin BW, Rardin RL (1979) Controlled experimental design for statistical comparison of integer programming algorithms. Management Science 25(12):1258–1271
McGeoch CC (1996) Toward an experimental method for algorithm simulation. INFORMS Journal on Computing 8(1):1–15, this journal issue contains also commentaries by Pierre L’Ecuyer, James B. Orlin and Douglas R. Shier, and a rejoinder by C. McGeoch
Michiels W, Aarts E, Korst J (2007) Theoretical Aspects of Local Search. Monographs in Theoretical Computer Science, An EATCS Series, Springer
Molenberghs G, Verbeke G (eds) (1997)Linear Mixed Models in Practice - A SASOriented Approach. >Springer
Molenberghs G, Verbeke G (2005) Models for Discrete Longitudinal Data. Springer
Montgomery DC (2005) Design and Analysis of Experiments, 6th edn. Wiley
Pinheiro J, Bates D, DebRoy S, Sarkar D, the R Core team (2008) nlme: Linear and Nonlinear Mixed Effects Models. R package version 3.1–89
Pinheiro JC, Bates DM (2000) Mixed-Effects Models in S and S-Plus. Springer
R Development Core Team (2008) R: A Language and Environment for StatisticalComputing. R Foundation for Statistical Computing, Vienna, Austria, URL http://www.R-project.org
Rardin RL, Uzsoy R (2001) Experimental evaluation of heuristic optimization algorithms: A tutorial. Journal of Heuristics 7(3):261–304
Rousseeuw PJ (1984) Least median of squares regression. Journal of the American Statistical Association 79(388):871–880, URL http://www.jstor.org/ stable/2288718 SAS Institute Inc. (2007) SAS online documentation: Parameterization of mixed models. http://www.webcitation.org/5h0u00trT, retrieved on 2009-05-24
Stram D, Lee J (1994) Variance components testing in the longitudinal mixed effects model. Biometrics 50(4):1171–1177
Stram D, Lee J (1995) Correction to ’Variance components testing in the longitudinal mixed effects model’. Biometrics 51(3):1196
Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation 1(1):67–82
Zanakis SH (1977) Heuristic 0-1 linear programming: An experimental comparison of three methods. Management Science 24(1):91–104
Zemel E (1981) Measuring the quality of approximate solutions to zero-one programming problems. Mathematics of Operations Research 6(3):319–332
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Chiarandini, M., Goegebeur, Y. (2010). Mixed Models for the Analysis of Optimization Algorithms. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds) Experimental Methods for the Analysis of Optimization Algorithms. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02538-9_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-02538-9_10
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-02537-2
Online ISBN: 978-3-642-02538-9
eBook Packages: Computer ScienceComputer Science (R0)