Inferential statistics and simulation generated samples: a critical reflection
Abstract
A review of recently published papers demonstrates: simulation practitioners apply the standard methods of inferential and descriptive statistics for their reasoning with simulation generated samples without much critical reflection. Yet, simulation-generated samples differ in important aspects from empirical samples, for which the standard statistical methods have been developed. Simulation models do have inherent epistemic and computational limits for replication that do not exist with empirical data sets. Consequently, neither is simulation-based data generation the same as the collection of empirical data nor is the analysis of synthetic data equally beneficial as of empirical data. These differences are much more fundamental for computer simulation than the problems of specific techniques of inferential statistics which have been criticized recently. If simulation generated data is used for testing research hypotheses the core issue is not the method of statistical reasoning but the assurance of what might be called evidential content.
References
[1]
Arnold, E., and J. Kästner. 2013. "When can a Computer Simulation act as Substitute for an Experiment?". Preprint series, Stuttgart Research.
[2]
Bae, J. W., J. H. Kim, I.-C. Moon, and T. G. Kim. 2016. "Accelerated Simulation of Hierarchical Military Operations with Tabulation Technique". Journal of Simulation 10(1):36--49.
[3]
Bedoya-Valencia, L., and E. Kirac. 2016. "Evaluating Alternative Resource Allocation in an Emergency Department using Discrete Event Simulation". Simulation 92(12):1041--1051.
[4]
Bova, M. J., F. W. Ciarallo, and R. R. Hill. 2016. "Development of an Agent-Based Model for the Secondary Threat Resulting from a Ballistic Impact Event". Journal of Simulation 10(1):24--35.
[5]
Calle, M., P. L. Gonzlez-R, J. M. Leon, H. Pierreval, and D. Canca. 2016. "Integrated Management of Inventory and Production Systems Based on Floating Decoupling Point and Real-Time Information: A Simulation Based Analysis". International Journal of Production Economics 181:48--57.
[6]
Chen, X., J.-h. Li, and Q. Gao. 2015. "A Simple Process Simulation Model for Strategic Planning on the Airside of an Airport: a Case Study". Journal of Simulation 9(1):64--72.
[7]
Conrads, A., M. Scheffer, H. Mattern, M. König, and M. Thewes. 2017. "Assessing Maintenance Strategies for Cutting Tool Replacements in Mechanized Tunneling using Process Simulation". Journal of Simulation 11(1):51--61.
[8]
Cook, M. 2004. "Universality in Elementary Cellular Automata". Complex Systems 15:140.
[9]
Dung, L. T., T. D. Hieu, and S.-G. Choi. 2016. "Simulation Modeling and Analysis of the Hop Count Distribution in Cognitive Radio Ad-Hoc Networks with Shadow Fading". Simulation Modelling Practice and Theory 69:43--54.
[10]
Grafarend, E. W. 2006. Linear and Nonlinear Models: Fixed Effects, Random Effects, and Mixed Models. Berlin: Walter de Gruyter.
[11]
Haramoto, H. 2009. "Automation of Statistical Tests on Randomness to Obtain Clearer Conclusion". In Monte Carlo and Quasi-Monte Carlo Methods 2008, edited by P. L' Ecuyer and A. B. Owen, 411--421. Springer Berlin Heidelberg.
[12]
Hellekalek, P. 1998. "Good Random Number Generators are (not so) Easy to Find". Mathematics and Computers in Simulation 46:485--505.
[13]
Henchey, M. J., R. Batta, A. Blatt, M. Flanigan, and K. Majka. 2014, May. "A Simulation Approach to Study Emergency Response". Journal of Simulation 8(2):115--128.
[14]
Hoaglin, D. C., F. Mosteller, and J. W. Tukey. 2000. Understanding Robust and Exploratory Data Analysis. New York: Wiley.
[15]
Hofmann, M. 2013. "Simulation-Based Exploratory Data Generation and Analysis (Data Farming): a Critical Reflection on its Validity and Methodology". The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology 10(4):381--393.
[16]
Hofmann, M. 2015a. "Reasoning beyond Predictive Validity: The Role of Plausibility in Decision-Supporting Social Simulation". In Proceedings of the 2015 Winter Simulation Conference, edited by M. D. Rossetti et al. Piscataway, New Jersey: IEEE.
[17]
Hofmann, M. 2015b. "Searching for Effects in Big Data: Why p-Values are not advised and what to use instead". In Proceedings of the 2015 Winter Simulation Conference, edited by L. Yilmaz et al. Piscataway, New Jersey: IEEE.
[18]
Hofmann, M., and S. Meyer-Nieberg. 2018. "Time to Dispense with the p-Value in OR?". Central European Journal of Operations Research 26(1):193--214.
[19]
Horne, G. E., and T. E. Meyer. 2005. "Data Farming: Discovering Surprise". In Proceedings of the 2005 Winter Simulation Conference, edited by J. A. Joines et al. Piscataway, New Jersey: IEEE.
[20]
Imputato, P., and S. Avallone. 2018. "An Analysis of the Impact of Network Device Buffers on Packet Schedulers through Experiments and Simulations". Simulation Modelling Practice and Theory 80:1--18.
[21]
Kim, T. H., C. H. Kim, S. D. Shin, and S. Haam. 2016. "Influence of Personal Protective Equipment on the Performance of Life-Saving Interventions by Emergency Medical Service Personnel". Simulation 92(10):893--898.
[22]
Law, A. M. 2014. Simulation Modeling and Analysis (5th ed.). New York: Mcgraw-Hill, Inc.
[23]
L'Ecuyer, P. 2015. "Random Number Generators with Multiple Streams for Sequential and Parallel Computing". In Proceedings of the 2015 Winter Simulation Conference, edited by M. D. Rossetti et al. Piscataway, New Jersey: IEEE.
[24]
Lee, S., D. Min, J.-H. Ryu, and Y. Yih. 2013. "A Simulation Study of Appointment Scheduling in Outpatient Clinics: Open Access and Overbooking". Simulation 89(12):1459--1473.
[25]
Rahimikelarijani, B., A. Abedi, M. Hamidi, and J. Cho. 2018. "Simulation Modeling of Houston Ship Channel Vessel Traffic for Optimal Closure Scheduling". Simulation Modelling Practice and Theory 80:89--103.
[26]
Romano, S., and H. ElAarag. 2012. "A Quantitative Study of Web Cache Replacement Strategies using Simulation". Simulation 88(5):507--541.
[27]
Rosenthal, R., and R. L. Rosnow. 1991. Essentials of Behavioral Research: Methods and Data Analysis (2nd ed.). New York: McGraw-Hill, Inc.
[28]
Rukhin, A., J. Soto, J. Nechvatal, M. Smid, E. Barker, S. Leigh, M. Levenson, M. Vangel, D. Banks, A. Heckert, J. Dray, and S. Vo. 2010. A Statistical Test Suite for Random and Pseudorandom Number Generators for Cryptographic Applications. Number 800-22 in NIST Special Publication. National Institute of Standards and Technology.
[29]
Sedlmeier, P. 1996. "Jenseits des Signifikanztest-Rituals: Ergaenzungen und Alternativen". Methods of Psychological Research Online 1(4):41--63.
[30]
Soto, J. 1999. "Statistical Testing of Random Number Generators". In Proc. of the 22nd National Information Systems Security Conference, 1--12. NIST.
[31]
Troitzsch, K. 2014. "Analysing Simulation Results Statistically: Does Significance Matter?". In Interdisciplinary Applications of Agent-Based Social Simulation and Modeling, edited by H. Coelho et al., 88--105. PA, USA: Hershey.
[32]
Tukey, J. W. 1977. Exploratory Data Analysis. Reading, Mass.
[33]
Vile, J. L., E. Allkins, J. Frankish, S. Garland, P. Mizen, and J. E. Williams. 2017. "Modelling Patient Flow in an Emergency Department to better understand Demand Management Strategies". Journal of Simulation 11(2):115--127.
[34]
White, J., A. Rassweiler, J. Samhouri, A. Stier, and C. White. 2014. "Ecologists should not use Statistical Significance Tests to interpret Simulation Model Results". Oikos 123:385--388.
[35]
Wolfram, S. 2002. A New Kind of Science. Champaign, Ilinois, US, United States: Wolfram Media.
[36]
Yates, J., A. Ford, and J. Kuglics. 2014. "Identifying Key Parameters and Trends in Civil Violence: a Sub-Regional, Agent-Based Simulation Approach using GIS". Journal of Simulation 8(3):179--194.
- Inferential statistics and simulation generated samples: a critical reflection
Recommendations
Inferential Statistics in Computing Education Research: A Methodological Review
ICER '19: Proceedings of the 2019 ACM Conference on International Computing Education ResearchThe goal of most computing education research is to effect positive change in how computing is taught and learned. Statistical techniques are one important tool for achieving this goal. In this paper we report on an analysis of ICER papers that use ...
Multivariate U-statistics: a tutorial with applications
U-statistics represent an important class of statistics arising from modeling quantities of interest defined by multi-subject responses such as the classic Mann-Whitney-Wilcoxon rank tests. However, classic applications of U-statistics are largely ...
A Comparative Study of Ontology Matching Systems via Inferential Statistics
Ontology matching systems are typically compared by comparing their average performances over multiple datasets. However, this paper examines the alignment systems using statistical inference since averaging is statistically unsafe and inappropriate. ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Sponsors
Publisher
IEEE Press
Publication History
Published: 09 December 2018
Check for updates
Qualifiers
- Research-article
Conference
WSC '18
Sponsor:
Acceptance Rates
WSC '18 Paper Acceptance Rate 183 of 260 submissions, 70%;
Overall Acceptance Rate 3,413 of 5,075 submissions, 67%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 68Total Downloads
- Downloads (Last 12 months)3
- Downloads (Last 6 weeks)0
Reflects downloads up to 24 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in