research-article

Performance measures for ranking and selection procedures

Authors:

Peter I. Frazier,

Shane G. HendersonAuthors Info & Claims

WSC '10: Proceedings of the Winter Simulation Conference

Pages 1235 - 1245

Published: 05 December 2010 Publication History

Abstract

To efficiently compare Ranking and Selection procedures, we present a three-layer performance evaluation process. The two most popular formulations, namely the Bayes and Indifference Zone formulations, have a common representation analogous to convex risk measures used in quantitative risk management. We study decision makers' acceptance sets via an axiomatic approach and introduce a new performance measure using computational cost.

References

[1]

Bechhofer, R., T. Santner, and D. Goldsman. 1995. Design and analysis of experiments for statistical selection, screening, and multiple comparisons. Wiley series in probability and statistics.

[2]

Berger, J. 1985. Statistical Decision Theory and Bayesian Analysis. Springer.

[3]

Boesel, J., B. Nelson, and S. Kim. 2003. Using Ranking and Selection to "Clean Up" after Simulation Optimization. Operations Research 51 (5): 814--825.

Digital Library

[4]

Branke, J., S. Chick, and C. Schmidt. 2007. Selecting a selection procedure. Management Science 53 (12): 1916--1932.

Digital Library

[5]

Chen, C. 1996. A lower bound for the correct subset-selection probability and its application to discrete-event system simulations. IEEE transactions on automatic control 41 (8): 1227--1231.

[6]

Chen, C., D. He, M. Fu, and L. Lee. 2008. Efficient simulation budget allocation for selecting an optimal subset. INFORMS Journal on Computing 20 (4): 579.

[7]

Chick, S. 1997. Selecting the best system: A decision-theoretic approach. In Proceedings of the 29th conference on Winter simulation, ed. S. Andradóttir, K. Healy, D. Withers, and B. Nelson, 326--333: IEEE Computer Society.

Digital Library

[8]

Chick, S. 2006. Subjective probability and Bayesian methodology. In Simulation, ed. S. G. Henderson and B. L. Nelson, Handbooks in Operations Research and Management Science. Elsevier.

[9]

Chick, S., and K. Inoue. 2001. New two-stage and sequential procedures for selecting the best simulated system. Operations Research 49 (5): 732--743.

Digital Library

[10]

DeGroot, M. 1970. Optimal Statistical Decision. McGraw-Hill, New York.

[11]

Föllmer, H., and A. Schied. 2004. Stochastic Finance, An Introduction in Discrete Time. Walter de Gruyter & Co., Berlin.

[12]

Kim, S., and B. Nelson. 2006. Selecting the best system. In Simulation, ed. S. G. Henderson and B. L. Nelson, Handbooks in Operations Research and Management Science. Elsevier.

[13]

Malone, G. 2004. Ranking and Selection Procedures for Bernoulli and Multinomial Data. Ph.D. thesis, Georgia Technical Institute, Atlanta GA.

[14]

McNeil, A., R. Frey, and P. Embrechts. 2005. Quantitative Risk Management: Concepts, Techniques, and Tools. Princeton university press, Princeton, NJ.

[15]

Rockafellar, T. 2007. Coherent approaches to risk in optimization under uncertainty. In Tutorials in Operations Research: OR Tools and Applications. Glimpses of Future Technologies. INFORMS.

[16]

Santner, T., and A. Tamhane. 1984. Design of experiments: ranking and selection. Marcel Dekker, inc.

[17]

Swisher, J., S. Jacobson, and E. Yücesan. 2003. Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey. ACM Transactions on Modeling and Computer Simulation (TOMACS) 13 (2): 134--154.

Digital Library

Cited By

Zhou EXie W(2015)Simulation optimization when facing input uncertaintyProceedings of the 2015 Winter Simulation Conference10.5555/2888619.2889116(3714-3724)Online publication date: 6-Dec-2015
https://dl.acm.org/doi/10.5555/2888619.2889116
Ryzhov IDefourny BPowell WRose OUhrmacher A(2012)Ranking and selection meets robust optimizationProceedings of the Winter Simulation Conference10.5555/2429759.2429822(1-11)Online publication date: 9-Dec-2012
https://dl.acm.org/doi/10.5555/2429759.2429822
Waeber RFrazier PHenderson S(2012)A Framework for Selecting a Selection ProcedureACM Transactions on Modeling and Computer Simulation10.1145/2331140.233114422:3(1-23)Online publication date: 1-Aug-2012
https://dl.acm.org/doi/10.1145/2331140.2331144

Index Terms

Performance measures for ranking and selection procedures
1. Computing methodologies
  1. Machine learning
    1. Machine learning algorithms
      1. Feature selection
  2. Modeling and simulation
    1. Simulation evaluation
    2. Simulation types and techniques

Recommendations

Do mean-based ranking and selection procedures consider systems' risk?
WSC '09: Winter Simulation Conference

The legacy simulation approach in ranking and selection procedures compares systems based on a mean performance metric. The best system is most often deemed as the one with the largest (or smallest) mean performance metric. In this paper, we discuss the ...
Consistency Measures for Hesitant Fuzzy Linguistic Preference Relations

Hesitant fuzzy linguistic term sets (HFLTSs) are used to deal with situations in which the decision makers (DMs) think of several possible linguistic values or richer expressions than a single term for an indicator, alternative, variable, etc. Compared ...
A comparative study of ranking methods, similarity measures and uncertainty measures for interval type-2 fuzzy sets

Ranking methods, similarity measures and uncertainty measures are very important concepts for interval type-2 fuzzy sets (IT2 FSs). So far, there is only one ranking method for such sets, whereas there are many similarity and uncertainty measures. A new ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WSC '10: Proceedings of the Winter Simulation Conference

December 2010

3519 pages

ISBN:9781424498642

Sponsors

IIE: Institute of Industrial Engineers
INFORMS-SIM: Institute for Operations Research and the Management Sciences: Simulation Society
SCS: Shanghai Computer Society
SIGSIM: ACM Special Interest Group on Simulation and Modeling

Publisher

Winter Simulation Conference

Publication History

Published: 05 December 2010

Check for updates

Qualifiers

Research-article

Conference

WSC10

Sponsor:

IIE
INFORMS-SIM
SCS
SIGSIM

WSC10: Winter Simulation Conference

December 5 - 8, 2010

Maryland, Baltimore

Acceptance Rates

WSC '10 Paper Acceptance Rate 184 of 281 submissions, 65%;

Overall Acceptance Rate 3,413 of 5,075 submissions, 67%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
25
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou EXie W(2015)Simulation optimization when facing input uncertaintyProceedings of the 2015 Winter Simulation Conference10.5555/2888619.2889116(3714-3724)Online publication date: 6-Dec-2015
https://dl.acm.org/doi/10.5555/2888619.2889116
Ryzhov IDefourny BPowell WRose OUhrmacher A(2012)Ranking and selection meets robust optimizationProceedings of the Winter Simulation Conference10.5555/2429759.2429822(1-11)Online publication date: 9-Dec-2012
https://dl.acm.org/doi/10.5555/2429759.2429822
Waeber RFrazier PHenderson S(2012)A Framework for Selecting a Selection ProcedureACM Transactions on Modeling and Computer Simulation10.1145/2331140.233114422:3(1-23)Online publication date: 1-Aug-2012
https://dl.acm.org/doi/10.1145/2331140.2331144

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten