poster

Bayesian inference for algorithm ranking analysis

Authors:

Borja Calvo,

Josu Ceberio,

Jose A. LozanoAuthors Info & Claims

GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Pages 324 - 325

https://doi.org/10.1145/3205651.3205658

Published: 06 July 2018 Publication History

Get Access

Abstract

The statistical assessment of the empirical comparison of algorithms is an essential step in heuristic optimization. Classically, researchers have relied on the use of statistical tests. However, recently, concerns about their use have arisen and, in many fields, other (Bayesian) alternatives are being considered. For a proper analysis, different aspects should be considered. In this work we focus on the question: what is the probability of a given algorithm being the best? To tackle this question, we propose a Bayesian analysis based on the Plackett-Luce model over rankings that allows several algorithms to be considered at the same time.

References

[1]

Alessio Benavoli, Giorgio Corani, Janez Demsar, and Marco Zaffalon. 2017. Time for a Change: a Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis. Journal of Machine Learning Research 18, 77 (2017), 1--36.

Digital Library

Google Scholar

[2]

Josu Ceberio, Ekhine Irurozki, Alexander Mendiburu, and Jose A Lozano. 2012. A review on estimation of distribution algorithms in permutation-based combinatorial optimization problems. Progress in Artificial Intelligence 1, 1 (2012), 103--117.

Crossref

Google Scholar

[3]

C.P. de Campos and A. Benavoli. 2016. Joint Analysis of Multiple Algorithms and Performance Measures. New Generation Computing 35, 1 (2016), 69--86.

Crossref

Google Scholar

[4]

Gerd Gigerenzer and Julian N. Marewski. 2015. Surrogate Science: The Idol of a Universal Method for Scientific Inference. Journal of Management 41, 2 (2015), 421--440.

Crossref

Google Scholar

[5]

Walter R Gilks, Sylvia Richardson, and David Spiegelhalter. 1995. Markov chain Monte Carlo in practice. CRC press.

Google Scholar

[6]

Greenland, Sander and Senn, Stephen J and Rothman, Kenneth J and Carlin, John B and Poole, Charles and Goodman, Steven N and Altaian, Douglas G. 2016. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European journal of epidemiology 31, 4 (2016), 337--350.

Google Scholar

[7]

Robin L. Plackett. 1975. The Analysis of Permutations. Journal of the Royal Statistical Society 24, 10 (1975), 193--202.

Google Scholar

[8]

Wasserstein, Ronald L and Lazar, Nicole A. 2016. The ASA's statement on p-values: context, process, and purpose. The American Statistician 70, 2 (2016), 129--133.

Crossref

Google Scholar

Cited By

View all

Martín-Santamaría RCavero SHerrán ADuarte AColmenar J(2024)A Practical Methodology for Reproducible Experimentation: An Application to the Double-Row Facility Layout ProblemEvolutionary Computation10.1162/evco_a_0031732:1(69-104)Online publication date: 1-Mar-2024
https://doi.org/10.1162/evco_a_00317
Rama-Maneiro EVidal JLama M(2024)Embedding Graph Convolutional Networks in Recurrent Neural Networks for Predictive MonitoringIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.328601736:1(137-151)Online publication date: Jan-2024
https://doi.org/10.1109/TKDE.2023.3286017
Heider MKrischan MSraj RHähner J(2024)Exploring Self-Adaptive Genetic Algorithms to Combine Compact Sets of Rules2024 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC60901.2024.10612101(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/CEC60901.2024.10612101
Show More Cited By

Index Terms

Bayesian inference for algorithm ranking analysis
1. Mathematics of computing
  1. Mathematical software
    1. Statistical software
  2. Probability and statistics
    1. Distribution functions
    2. Probabilistic inference problems
      1. Bayesian computation
      2. Hypothesis testing and confidence interval computation

Recommendations

Bayesian inference for Plackett-Luce ranking models
ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning

This paper gives an efficient Bayesian method for inferring the parameters of a Plackett-Luce ranking model. Such models are parameterised distributions over rankings of a finite set of objects, and have typically been studied and applied within the ...
Generalized Dirichlet distribution in Bayesian analysis

Generalized Dirichlet distribution has a more general covariance structure than Dirichlet distribution. This makes the generalized Dirichlet distribution to be more practical and useful. The concept of complete neutrality will be used to derive the ...
Bayesian analysis of semiparametric reproductive dispersion mixed-effects models

Semiparametric reproductive dispersion mixed-effects model (SPRDMM) is an extension of the reproductive dispersion model and the semiparametric mixed model, and it includes many commonly encountered models as its special cases. A Bayesian procedure is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference Companion

July 2018

1968 pages

ISBN:9781450357647

DOI:10.1145/3205651

Editor:
Hernan Aguirre
Shinshu University
,
General Chair:
Keiki Takadama
The University of Electro-Communications

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 July 2018

Check for updates

Author Tags

Qualifiers

Poster

Conference

GECCO '18

Sponsor:

SIGEVO

GECCO '18: Genetic and Evolutionary Computation Conference

July 15 - 19, 2018

Kyoto, Japan

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
212
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)2

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Martín-Santamaría RCavero SHerrán ADuarte AColmenar J(2024)A Practical Methodology for Reproducible Experimentation: An Application to the Double-Row Facility Layout ProblemEvolutionary Computation10.1162/evco_a_0031732:1(69-104)Online publication date: 1-Mar-2024
https://doi.org/10.1162/evco_a_00317
Rama-Maneiro EVidal JLama M(2024)Embedding Graph Convolutional Networks in Recurrent Neural Networks for Predictive MonitoringIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.328601736:1(137-151)Online publication date: Jan-2024
https://doi.org/10.1109/TKDE.2023.3286017
Heider MKrischan MSraj RHähner J(2024)Exploring Self-Adaptive Genetic Algorithms to Combine Compact Sets of Rules2024 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC60901.2024.10612101(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/CEC60901.2024.10612101
De La Cruz MGarnica OCervigon CVelasco JHidalgo J(2024)Explainable hypoglycemia prediction models through dynamic structured grammatical evolutionScientific Reports10.1038/s41598-024-63187-514:1Online publication date: 1-Jun-2024
https://doi.org/10.1038/s41598-024-63187-5
Rama-Maneiro EVidal JLama MMonteagudo-Lago P(2024)Exploiting recurrent graph neural networks for suffix prediction in predictive monitoringComputing10.1007/s00607-024-01315-9106:9(3085-3111)Online publication date: 9-Jul-2024
https://doi.org/10.1007/s00607-024-01315-9
Cui HHeider MHähner J(2024)Positional Bias Does Not Influence Cartesian Genetic Programming with CrossoverParallel Problem Solving from Nature – PPSN XVIII10.1007/978-3-031-70055-2_10(151-167)Online publication date: 14-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-70055-2_10
Rama-Maneiro EPatrizi FVidal JLama M(2024)Towards Learning the Optimal Sampling Strategy for Suffix Prediction in Predictive MonitoringAdvanced Information Systems Engineering10.1007/978-3-031-61057-8_13(215-230)Online publication date: 3-Jun-2024
https://doi.org/10.1007/978-3-031-61057-8_13
Cui HPätzel DMargraf AHähner J(2023)Weighted Mutation of Connections To Mitigate Search Space Limitations in Cartesian Genetic ProgrammingProceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms10.1145/3594805.3607130(50-60)Online publication date: 30-Aug-2023
https://dl.acm.org/doi/10.1145/3594805.3607130
Benavides XCeberio JHernando LLozano JSilva SPaquete L(2023)New Knowledge about the Elementary Landscape Decomposition for Solving the Quadratic Assignment ProblemProceedings of the Genetic and Evolutionary Computation Conference10.1145/3583131.3590369(239-247)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583131.3590369
Yan YLiu QLi Y(2023)Paradox-Free Analysis for Comparing the Performance of Optimization AlgorithmsIEEE Transactions on Evolutionary Computation10.1109/TEVC.2022.319964727:5(1275-1287)Online publication date: Oct-2023
https://doi.org/10.1109/TEVC.2022.3199647
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Bayesian inference for Plackett-Luce ranking models

Generalized Dirichlet distribution in Bayesian analysis

Bayesian analysis of semiparametric reproductive dispersion mixed-effects models