Abstract
When using simulation modeling to explore pedagogical phenomena, there are several issues a designer/practitioner should consider. One of the most important decisions has to do with determining how many runs of a simulation to perform in order to be confident in the results produced by the simulation [1]. With a deterministic model, a single simulation run is adequate. This issue becomes more challenging when part of the simulation model is based on stochastic elements. One of the solutions that has been used to address this challenge in other research communities is the use of Monte Carlo simulation [2]. Within the AIED research community, however, this question of how many times should a pedagogical simulation model be run to produce predictions in which the designer can have confidence has received surprisingly little attention. The aim of this paper is to explore this issue using a pedagogical simulation model, SimDoc, designed to explore longer term mentoring issues [3]. In particular, we demonstrate how to run this simulation model over many iterations until the accumulated results of the iteration runs reach a statistically stable level that matches real world performance but also has appropriate variability among the runs. We believe this approach generalizes beyond our simulation environment and could be applied to other pedagogical simulations and would be especially useful for medium and high fidelity simulations where each run may take a long time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bogdoll, J., Hartmanns, A., Hermanns, H.: Simulation and statistical model checking for modestly nondeterministic models. In: Schmitt, J.B. (ed.) International GI/ITG Conference on Measurement, Modelling, and Evaluation of Computing Systems and Dependability and Fault Tolerance, pp. 249–252. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28540-0_20
Koehler, E., Brown, E., Haneuse, S.J.-P.A.: On the assessment of Monte Carlo error in simulation-based statistical analyses. Am. Stat. 63(2), 155–162 (2009)
Lelei, D.E.K., McCalla, G.: How to use simulation in the design and evaluation of learning environments with self-directed longer-term learners. In: Penstein Rosé, C., et al. (eds.) AIED 2018. LNCS (LNAI), vol. 10947, pp. 253–266. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93843-1_19
Lane, H.C., McCalla, G.I., Looi, C.-K., Bull, S.: The next 25 years: how advanced interactive learning technologies will change the world. Int. J. Artif. Intell. Educ. 26(1), 539–543 (2016)
Lelei, D.E.K., McCalla, G.I.: The role of simulation in the development of mentoring technology to support longer-term learning. In: The Proceedings of 3rd International Workshop on Intelligent Mentoring Systems Held in Conjunction with the 19th International Conference on Artificial Intelligence in Education (2018)
VanLehn, K., Ohlsson, S., Nason, R.: Applications of simulated students: an exploration. J. Artif. Intell. Educ. 5(2), 1–42 (1994)
Dorça, F.: Implementation and use of simulated students for test and validation of new adaptive educational systems: a practical insight. Int. J. Artif. Intell. Educ. 25(3), 319–345 (2015)
Laberge, S., Lin, F.: Simulated learners for testing agile teaming in social educational games. In: CEUR Workshop Proceedings, vol. 1432, pp. 65–77 (2015)
Frost, S., McCalla, G.: Exploring through simulation an instructional planner for dynamic open-ended learning environments. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.Felisa (eds.) AIED 2015. LNCS (LNAI), vol. 9112, pp. 578–581. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19773-9_66
Lelei, D.E.K., McCalla, G.I.: Exploring the issues in simulating a semi-structured learning environment: the SimGrad doctoral program design. In: The Proceedings of the 2nd Workshop on Simulated Learners at the 17th International Conference on Artificial Intelligence in Education, vol. 5, pp. 11–20 (2015)
Carlson, R., Keiser, V., Matsuda, N., Koedinger, K.R., Penstein Rosé, C.: Building a conversational SimStudent. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 563–569. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30950-2_73
Booth, J.G., Sarkar, S.: Monte Carlo approximation of bootstrap variances. Am. Stat. 52(4), 354–357 (1998)
Truong, L.T., Sarvi, M., Currie, G., Garoni, T.M.: How many simulation runs are required to achieve statistically confident results: a case study of simulation-based surrogate safety measures. In: IEEE 18th International Conference on Intelligent Transportation Systems, pp. 274–278 (2015)
Van Joolingen, W.: Design and implementation of simulation-based discovery environments: the SMISLE solution. J. Artif. Intell. Educ. 7(4), 253–276 (1996)
Rosenberg-Kima, R.B., Pardos, Z.A.: Is this model for real? Simulating data to reveal the proximity of a model to reality. In: The Proceedings of the 17th International Conference on Artificial Intelligence in Education, pp. 78–87 (2015)
Weber, G.: Individual selection of examples in an intelligent learning environment. Int. J. Artif. Intell. Educ. 7(1), 3–31 (2015)
Liu, C.: A simulation-based experience in learning structures of bayesian networks to represent how students learn composite concepts. Int. J. Artif. Intell. Educ. 18(3), 237–285 (2008)
Dzikovska, M., Steinhauser, N., Farrow, E., Moore, J., Campbell, G.: BEETLE II: deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics. Int. J. Artif. Intell. Educ. 24(3), 284–332 (2014)
Desmarais, M.C., Pu, X.: A Bayesian student model without hidden nodes and its comparison with item response theory. Int. J. Artif. Intell. Educ. 15(4), 291–323 (2005)
Yeh, A., Corp, M.: More accurate tests for the statistical significance of result differences. In: Proceedings of the 18th International Conference on Computational Linguistics, pp. 947–953 (2000)
Erickson, G., Frost, S., Bateman, S., McCalla, G.: Using the ecological approach to create simulations of learning environments. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 411–420. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39112-5_42
Riedesel, M.A., Zimmerman, N., Baker, R., Titchener, T., Cooper, J.: Using a model for learning and memory to simulate learner response in spaced practice. In: The Proceedings of the 18th International Conference on Artificial Intelligence in Education, pp. 644–649 (2017)
Heath, T.: A quantitative analysis of PhD students’ views of supervision. High. Educ. Res. Dev. 21, 37–41 (2002)
Gatfield, T.: An investigation into PhD supervisory management styles: development of a dynamic conceptual model and its managerial implications. J. High. Educ. Policy Manage. 27(3), 311–325 (2005)
Ritter, F.E., Schoelles, M.J., Quigley, K.S., Klein, L.C.: Determining the number of simulation runs treating simulations as theories by not sampling their behavior. In: Rothrock, L., Narayanan, S. (eds.) Human-in-the-Loop Simulations: Methods and Practice, pp. 97–116. Springer, London (2011). https://doi.org/10.1007/978-0-85729-883-6_5
Acknowledgements
We would like to acknowledge the financial support of the Natural Sciences and Engineering Research Council of Canada for this research. We also would like to thank the University of Saskatchewan for providing (anonymized) data on its Ph.D. programs that we could use to inform the SimDoc simulation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Lelei, D.E.K., McCalla, G. (2019). How Many Times Should a Pedagogical Agent Simulation Model Be Run?. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds) Artificial Intelligence in Education. AIED 2019. Lecture Notes in Computer Science(), vol 11625. Springer, Cham. https://doi.org/10.1007/978-3-030-23204-7_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-23204-7_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-23203-0
Online ISBN: 978-3-030-23204-7
eBook Packages: Computer ScienceComputer Science (R0)