Article

Free access

Output analysis: output analysis for simulations

Authors:

Christos Alexopoulos,

Andrew F. SeilaAuthors Info & Claims

WSC '00: Proceedings of the 32nd conference on Winter simulation

Pages 101 - 108

Published: 10 December 2000 Publication History

Abstract

This paper reviews statistical methods for analyzing output data from computer simulations of single systems. In particular, it focuses on the estimation of steady-state system parameters. The estimation techniques include the replication/deletion approach, the regenerative method, the batch means method, and the standardized time series method.

References

[1]

Alexopoulos, C., and A. F. Seila. 1998. Output data analysis. In Handbook of Simulation, ed. J. Banks, Chapter 7, New York: John Wiley & Sons.

[2]

Alexopoulos, C., A. F. Seila and G. S. Fishman. 1998. Computational experience with the batch means method. In Proceedings of the 1997 Winter Simulation Conference, 194-201. Institute of Electrical and Electronics Engineers, Piscataway, New Jersey.

Digital Library

[3]

Billingsley, P. 1968. Convergence of probability measures, Wiley, New York.

[4]

Bratley, P., B. L. Fox, and L. E. Schrage. 1987. A guide to simulation, 2d Ed. Springer-Verlag, New York, New York.

Digital Library

[5]

Chance, F., and L. W. Schruben. 1992. Establishing a truncation point in simulation output. Technical Report, School of Operations Research and Industrial Engineering, Cornell University, Ithaca, New York.

[6]

Charnes, J. M. 1989. Statistical analysis of multivariate discrete-event simulation output. Ph.D. Thesis, Department of Operations and Management Science, University of Minnesota, Minneapolis, Minnesota.

[7]

Charnes, J. M. 1990. Power comparisons for the multivariate batch-means method. In Proceedings of the 1990 Winter Simulation Conference, 281-287. IEEE, Piscataway, New Jersey.

Digital Library

[8]

Charnes, J. M. 1991. Multivariate simulation output analysis. In Proceedings of the 1991 Winter Simulation Conference, 187-193. IEEE, Piscataway, New Jersey.

Digital Library

[9]

Chen, R. D., and A. F. Seila. 1987. Multivariate inference in stationary simulation using batch means. In Proceedings of the 1987 Winter Simulation Conference, 302-304. IEEE, Piscataway, New Jersey.

Digital Library

[10]

Chien, C.-H. 1989. Small sample theory for steady state confidence intervals. Technical Report No. 37, Department of Operations Research, Stanford University, Palo Alto, California.

[11]

Chien, C., D. Goldsman, and B. Melamed. 1997. Large-sample results for batch means. Management Science 43:1288-1295.

Digital Library

[12]

Chow, Y. S., and H. Robbins. 1965. On the asymptotic theory of fixed-width sequential confidence intervals for the mean. Annals of Mathematical Statistics 36:457-462.

[13]

Conway, R. W. 1963. Some tactical problems in digital simulation. Management Science 10:47-61.

Digital Library

[14]

Crane, M. A., and D. L. Iglehart. 1974a. Simulating stable stochastic systems, I: General multiserver queues. Journal of the ACM 21:103-113.

Digital Library

[15]

Crane, M. A., and D. L. Iglehart. 1974b. Simulating stable stochastic systems, II: Markov chains. Journal of the ACM 21:114-123.

Digital Library

[16]

Crane, M. A., and D. L. Iglehart. 1975. Simulating stable stochastic systems III: Regenerative processes and discrete-event simulations. Operations Research 23:33-45.

Digital Library

[17]

Damerdji, H. 1994. Strong consistency of the variance estimator in steady-state simulation output analysis. Mathematics of Operations Research 19:494-512.

Digital Library

[18]

Fishman, G. S. 1973. Statistical analysis for queueing simulations. Management Science 20:363-369.

Digital Library

[19]

Fishman, G. S. 1974. Estimation of multiserver queueing simulations. Operations Research 22:72-78.

Digital Library

[20]

Fishman, G. S. 1978. Principles of discrete event simulation, New York: John Wiley & Sons.

Digital Library

[21]

Fishman, G. S. 1996. Monte Carlo: Concepts, algorithms, and applications. New York: Springer Verlag.

[22]

Fishman, G. S. 2000. Private communication.

[23]

Fishman, G. S., and L. S. Yarberry. 1997. An implementation of the batch means method. INFORMS Journal on Computing 9:296-310.

Digital Library

[24]

Gafarian, A. V., C. J. Ancker, and F. Morisaku. 1978. Evaluation of commonly used rules for detecting steady-state in computer simulation. Naval Research Logistics Quarterly 25:511-529.

[25]

Glynn, P. W., and D. L. Iglehart. 1990. Simulation analysis using standardized time series. Mathematics of Operations Research 15:1-16.

Digital Library

[26]

Goldsman, D., M. Meketon, and L. W. Schruben. 1990. Properties of standardized time series weighted area variance estimators. Management Science 36:602-612.

Digital Library

[27]

Goldsman, D., and L. W. Schruben. 1984. Asymptotic properties of some confidence interval estimators for simulation output. Management Science 30:1217-1225.

Digital Library

[28]

Goldsman, D., and L. W. Schruben. 1990. New confidence interval estimators using standardized time series. Management Science 36:393-397.

Digital Library

[29]

Goldsman, D., L. W. Schruben, and J. J. Swain. 1994. Tests for transient means in simulated time series. Naval Research Logistics 41:171-187.

[30]

Heidelberger, P., and P. A. W. Lewis. 1984. Quantile estimation in dependent sequences. Operations Research 32:185-209.

Digital Library

[31]

Heidelberger, P., and P. D. Welch. 1981a. A spectral method for confidence interval generation and run length control in simulations. Communications of the ACM 24:233-245.

Digital Library

[32]

Iglehart, D. L. 1975. Simulating stable stochastic systems, V: Comparison of ratio estimators. Naval Research Logistics Quarterly 22:553-565.

[33]

Iglehart, D. L. 1976. Simulating stable stochastic systems, VI: Quantile estimation. Journal of the ACM 23:347-360.

Digital Library

[34]

Iglehart, D. L. 1978. The regenerative method for simulation analysis. In Current Trends in Programming Methodology, Vol. III, eds. K. M. Chandy, and K. M. Yeh, 52-71. Prentice-Hall, Englewood Cliffs, New Jersey.

[35]

Kelton, W. D. 1989. Random initialization methods in simulation. IIE Transactions 21:355-367.

[36]

Law, A. M., and J. S. Carson. 1979. A sequential procedure for determining the length of a steady-state simulation. Operations Research 27:1011-1025.

Digital Library

[37]

Law, A. M., and W. D. Kelton. 2000. Simulation modeling and analysis, 3d Ed. McGraw-Hill, New York.

Digital Library

[38]

Mechanic, H., and W. McKay. 1966. Confidence intervals for averages of dependent data in simulations II. Technical Report ASDD 17-202, IBM Corporation, Yorktown Heights, New York.

[39]

Meketon, M. S., and B. W. Schmeiser. 1984. Overlapping batch means: Something for nothing? In Proceedings of the 1984 Winter Simulation Conference, 227-230. IEEE, Piscataway, New Jersey.

Digital Library

[40]

Moore, L. W. 1980. Quantile estimation in regenerative processes. Ph.D. Thesis, Curriculum in Operations Research and Systems Analysis, University of North Carolina, Chapel Hill, North Carolina.

[41]

Nadas, A. 1969. An extension of the theorem of Chow and Robbins on sequential confidence intervals for the mean. Annals of Mathematical Statistics 40:667-671.

[42]

Ockerman, D. H. 1995. Initialization bias tests for stationary stochastic processes based upon standardized time series techniques. Ph.D. Thesis, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia.

[43]

Resnick, S. I. 1994. Adventures in stochastic processes. Boston: Birkhaüser.

Digital Library

[44]

Sargent, R. G., K. Kang, and D. Goldsman. 1992. An investigation of finite-sample behavior of confidence interval estimators. Operations Research 40:898-913.

Digital Library

[45]

Schmeiser, B. W. 1982. Batch size effects in the analysis of simulation output. Operations Research 30:556-568.

Digital Library

[46]

Schriber, T. J., and R. W. Andrews. 1979. Interactive analysis of simulation output by the method of batch means. In Proceedings of the 1979 Winter Simulation Conference, 513-525. IEEE, Piscataway, New Jersey.

Digital Library

[47]

Schruben, L. W. 1982. Detecting initialization bias in simulation output. Operations Research 30:569-590.

Digital Library

[48]

Schruben, L. W. 1983. Confidence interval estimation using standardized time series. Operations Research 31:1090-1108.

Digital Library

[49]

Schruben, L. W., H. Singh, and L. Tierney. 1983. Optimal tests for initialization bias in simulation output. Operations Research 31:1167-1178.

Digital Library

[50]

Seila, A. F. 1982a. A batching approach to quantile estimation in regenerative simulations. Management Science 28:573-581.

Digital Library

[51]

Seila, A. F. 1982b. Percentile estimation in discrete event simulation. Simulation 39:193-200.

[52]

Song, W.-M. T., and B. W. Schmeiser. 1993. Optimal mean-squared-error batch sizes. Management Science 41:110-123.

Digital Library

[53]

von Neumann, J. 1941a. The mean square difference. Annals of Mathematical Statistics 12:153-162.

[54]

von Neumann, J. 1941b. Distribution of the ratio of the mean square successive difference and the variance. Annals of Mathematical Statistics 12:367-395.

[55]

Welch, P. D. 1983. The statistical analysis of simulation results. In The computer performance modeling handbook, ed. S. Lavenberg, 268-328. New York: Academic Press.

[56]

Welch, P. D. 1987. On the relationship between batch means, overlapping batch means and spectral estimation, In Proceedings of the 1987 Winter Simulation Conference, 320-323. IEEE, Piscataway, New Jersey.

Digital Library

[57]

Wilson, J. R., and A. A. B. Pritsker. 1978a. A survey of research on the simulation startup problem. Simulation 31:55-58.

[58]

Wilson, J. R., and A. A. B. Pritsker. 1978b. Evaluation of startup policies in simulation experiments. Simulation 31:79-89.

[59]

Yarberry, L. S. 1993. Incorporating a dynamic batch size selection mechanism in a fixed-sample-size batch means procedure. Ph.D. dissertation, Department of Operations Research, University of North Carolina, Chapel Hill, North Carolina.

[60]

Young, L. C. 1941. On randomness of order sequences. Annals of Mathematical Statistics 12:293-300.

Cited By

Law A(2004)Statistical analysis of simulation output dataProceedings of the 36th conference on Winter simulation10.5555/1161734.1161752(67-72)Online publication date: 5-Dec-2004
https://dl.acm.org/doi/10.5555/1161734.1161752
Naylor ARohrer MMedeiros DGrabau M(2001)Quantifying simulation output variability using confidence intervals and statistical process controlProceedings of the 33nd conference on Winter simulation10.5555/564124.564251(896-901)Online publication date: 9-Dec-2001
https://dl.acm.org/doi/10.5555/564124.564251
Goldsman DTokol GFishwick PKang KJoines JBarton R(2000)Output analysisProceedings of the 32nd conference on Winter simulation10.5555/510378.510388(39-45)Online publication date: 10-Dec-2000
https://dl.acm.org/doi/10.5555/510378.510388

Index Terms

Output analysis: output analysis for simulations
1. Mathematics of computing
  1. Probability and statistics

Recommendations

Simulation output analysis: a wavelet-based spectral method for steady-state simulation analysis
WSC '03: Proceedings of the 35th conference on Winter simulation: driving innovation

We develop an automated wavelet-based spectral method for constructing an approximate confidence interval on the steady-state mean of a simulation output process. This procedure, called WASSP, determines a batch size and a warm-up period beyond which ...
Output analysis: output data analysis for simulations
WSC '01: Proceedings of the 33nd conference on Winter simulation

This paper reviews statistical methods for analyzing output data from computer simulations of single systems. In particular, it focuses on the estimation of steady-state system parameters. The estimation techniques include the replication/deletion ...
Mean-Square Consistency of the Variance Estimator in Steady-State Simulation Output Analysis

In steady-state simulation output analysis, mean-square consistency of the process-variance estimator is important for a number of reasons. One way to construct an asymptotically valid confidence interval around a sample mean is via construction of a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WSC '00: Proceedings of the 32nd conference on Winter simulation

December 2000

2014 pages

ISBN:0780365828

Sponsors

IIE: Institute of Industrial Engineers
ASA: American Statistical Association
SIGSIM: ACM Special Interest Group on Simulation and Modeling
IEEE/CS: Institute of Electrical and Electronics Engineers/Computer Society
NIST: National Institute of Standards and Technology
INFORMS-CS: Institute for Operations Research and the Management Sciences-College on Simulation
IEEE/SMCS: Institute of Electrical and Electronics Engineers/Systems, Man, and Cybernetics Society
SCS: The Society for Computer Simulation International

Publisher

Society for Computer Simulation International

San Diego, CA, United States

Publication History

Published: 10 December 2000

Check for updates

Qualifiers

Article

Conference

WSC00

Sponsor:

IIE
ASA
SIGSIM
IEEE/CS
NIST
INFORMS-CS
IEEE/SMCS
SCS

WSC00: Winter Simulation Conference

December 10 - 13, 2000

Florida, Orlando

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
99
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)3

Reflects downloads up to 20 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Law A(2004)Statistical analysis of simulation output dataProceedings of the 36th conference on Winter simulation10.5555/1161734.1161752(67-72)Online publication date: 5-Dec-2004
https://dl.acm.org/doi/10.5555/1161734.1161752
Naylor ARohrer MMedeiros DGrabau M(2001)Quantifying simulation output variability using confidence intervals and statistical process controlProceedings of the 33nd conference on Winter simulation10.5555/564124.564251(896-901)Online publication date: 9-Dec-2001
https://dl.acm.org/doi/10.5555/564124.564251
Goldsman DTokol GFishwick PKang KJoines JBarton R(2000)Output analysisProceedings of the 32nd conference on Winter simulation10.5555/510378.510388(39-45)Online publication date: 10-Dec-2000
https://dl.acm.org/doi/10.5555/510378.510388

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents