research-article

Studying Retrieval Practice in an Intelligent Tutoring System

Authors:

Jeffrey Matayoshi,

Eric CosynAuthors Info & Claims

L@S '20: Proceedings of the Seventh ACM Conference on Learning @ Scale

Pages 51 - 62

https://doi.org/10.1145/3386527.3405927

Published: 12 August 2020 Publication History

Abstract

Retrieval practice (also known as testing effect or test-enhanced learning) is a well-studied and established technique for improving the retention of knowledge. Many previous works have confirmed the benefits of retrieval practice in laboratory experiments involving the memorization of words or facts. In this study, we build on these works and analyze retrieval practice in an intelligent tutoring system. Using a large data set composed of the actions of almost 4 million students studying math and chemistry, we look at the possible benefits of retrieval practice in the ALEKS adaptive learning and assessment system. We compare two different types of retrieval practice---one involving the assessment of learned material, and another involving the learning of closely related content that builds on the learned material---leveraging the scale of the available data to control for several confounding variables. Finally, we look at the timing of retrieval practice within the system and the possible effect it has on forgetting. The results indicate that a delay in retrieval practice is associated with better retention and that, while being assessed on learned material is beneficial, the learning of closely related content is associated with an even higher rate of retention.

References

[1]

Olusola O. Adesope, Dominic A. Trevisan, and Narayankripa Sundararajan. 2017. Rethinking the use of tests: A meta-analysis of practice testing. Review of Educational Research 87, 3 (2017), 659--701.

[2]

Pooja K. Agarwal, Patrice M. Bain, and Roger W. Chamberlain. 2012. The value of applied research: Retrieval practice improves classroom learning and recommendations from a teacher, a principal, and a scientist. Educational Psychology Review 24 (2012), 437--448.

[3]

Hirotugu Akaike. 1974. A new look at the statistical model identification. IEEE Trans. Automat. Control 19, 6 (1974), 716--723.

[4]

Lee Averell and Andrew Heathcote. 2011. The form of the forgetting curve and the fate of memories. Journal of Mathematical Psychology 55 (2011), 25--35.

[5]

Christine L. Bae, David J. Therriault, and Jenni L. Redifer. 2019. Investigating the testing effect: Retrieval as a characteristic of effective study strategies. Learning and Instruction 60 (2019), 206--214.

[6]

Robert L. Bangert-Drowns, Chen-Lin C. Kulik, James A. Kulik, and MaryTeresa Morgan. 1991. The instructional effect of feedback in test-like events. Review of Educational Research 61, 2 (1991), 213--238.

[7]

Katharina Barzagar Nazari and Mirjam Ebersbach. 2019. Distributing mathematical practice of third and seventh graders: Applicability of the spacing effect in the classroom. Applied Cognitive Psychology 33, 2 (2019), 288--298.

[8]

Robert A. Bjork. 1994. Memory and metamemory considerations in the training of human beings. In Metacognition: Knowing about knowing, Janet Metcalfe, Arthur P. Shimamura, et almbox. (Eds.). MIT press.

[9]

Robert A. Bjork. 1999. Assessing our own competence: Heuristics and illusions. In Attention and performance XVII: Cognitive regulation of performance: Interaction of theory and application, Daniel Gopher and Asher Koriat (Eds.). MIT Press.

[10]

Andrew C. Butler, Jeffrey D. Karpicke, and Henry L. Roediger III. 2008. Correcting a metacognitive error: feedback increases retention of low-confidence correct responses. Journal of Experimental Psychology: Learning, Memory, and Cognition 34, 4 (2008), 918.

[11]

Benôit Choffin, Fabrice Popineau, Yolaine Bourda, and Jill-Jênn Vie. 2019. DAS3H: Modeling Student Learning and Forgetting for Optimally Scheduling Distributed Practice of Skills. In Proceedings of the 12th International Conference on Educational Data Mining. 29--38.

[12]

Jean-Paul Doignon and Jean-Claude Falmagne. 1985. Spaces for the assessment of knowledge. International Journal of Man-Machine Studies 23 (1985), 175--196.

[13]

Hermann Ebbinghaus. 1885; translated by Henry A. Ruger and Clara E. Bussenius (1913). Memory: A Contribution to Experimental Psychology. Originally published by Teachers College, Columbia University, New York.

[14]

Jean-Claude Falmagne, Dietrich Albert, Christopher Doble, David Eppstein, and Xiangen Hu (Eds.). 2013. Knowledge Spaces: Applications in Education. Springer-Verlag, Heidelberg.

[15]

Jean-Claude Falmagne and Jean-Paul Doignon. 2011. Learning Spaces. Springer-Verlag, Heidelberg.

[16]

Nicole A.M.C. Goossens, Gino Camp, Peter P.J.L. Verkoeijen, Huib K. Tabbers, Samantha Bouwmeester, and Rolf A. Zwaan. 2016. Distributed Practice and Retrieval Practice in Primary School Vocabulary Learning: A Multi-classroom Study. Applied Cognitive Psychology 30, 5 (2016), 700--712.

[17]

Patricia Hanley-Dunn and John L. McIntosh. 1984. Meaningfulness and recall of names by young and old adults. Journal of Gerontology 39 (1984), 583--585.Issue 5.

[18]

James W. Hardin and Joseph M. Hilbe. 2012. Generalized Estimating Equations. Chapman and Hall/CRC.

[19]

Patrick J. Heagerty and Scott L. Zeger. 2000. Marginalized multilevel models and likelihood inference (with comments and a rejoinder by the authors). Statist. Sci. 15, 1 (2000), 1--26.

[20]

Jeffrey D. Karpicke and Althea Bauernschmidt. 2011. Spaced retrieval: absolute spacing enhances learning regardless of relative spacing. Journal of Experimental Psychology: Learning, Memory, and Cognition 37, 5 (2011), 1250.

[21]

Jeffrey D. Karpicke and Henry L. Roediger. 2008. The critical importance of retrieval for learning. Science 319, 5865 (2008), 966--968.

[22]

Jeffrey D. Karpicke and Henry L. Roediger III. 2007. Expanding retrieval practice promotes short-term retention, but equally spaced retrieval enhances long-term retention. Journal of Experimental Psychology: Learning, Memory, and Cognition 33, 4 (2007), 704.

[23]

Kung-Yee Liang and Scott L. Zeger. 1986. Longitudinal data analysis using generalized linear models. Biometrika 73, 1 (1986), 13--22.

[24]

Robert V. Lindsey, Jeffery D. Shroyer, Harold Pashler, and Michael C. Mozer. 2014. Improving students long-term knowledge retention through personalized review. Psychological science 25, 3 (2014), 639--647.

[25]

Jeffrey Matayoshi, Umberto Granziol, Christopher Doble, Hasan Uzun, and Eric Cosyn. 2018. Forgetting Curves and Testing Effect in an Adaptive Learning and Assessment System. In Proceedings of the 11th International Conference on Educational Data Mining. 607--612.

[26]

Jeffrey Matayoshi, Hasan Uzun, and Eric Cosyn. 2019. Deep (Un)Learning: Using Neural Networks to Model Retention and Forgetting in an Adaptive Learning System. In Artificial Intelligence in Education-20th International Conference, AIED 2019. 258--269.

Digital Library

[27]

Dawn M. McBride and Barbara Anne Dosher. 1997. A comparison of forgetting in an implicit and explicit memory task. Journal of Experimental Psychology: General 126 (1997), 371--392.Issue 4.

[28]

Mark A. McDaniel and Gilles O. Einstein. 2005. Material Appropriate Difficulty: A Framework for Determining When Difficulty Is Desirable for Improving Learning. In Decade of Behavior. Experimental Cognitive Psychology and its Applications, A. F. Healy (Ed.). American Psychological Association, 73--85.

[29]

McGraw-Hill Education/ALEKS Corporation. 2019. What is ALEKS? https://www.aleks.com/about_aleks. (2019).

[30]

Bruna Fernanda Tolentino Moreira, Tatiana Salazar Silva Pinto, Daniela Siqueira Veloso Starling, and Antônio Jaeger. 2019. Retrieval practice in classroom settings: A review of applied research. Frontiers in Education 4 (2019), 5.

[31]

Allan Paivio and Padric C. Smythe. 1971. Word imagery, frequency, and meaningfulness in short-term memory. Psychonomic Science 22 (1971), 333--335.Issue 6.

[32]

Steven C. Pan and Timothy C. Rickard. 2018. Transfer of test-enhanced learning: Meta-analytic review and synthesis. Psychological Bulletin 144, 7 (2018), 710.

[33]

Wei Pan. 2001. Akaike's information criterion in generalized estimating equations. Biometrics 57, 1 (2001), 120--125.

[34]

Harold Pashler, Nicholas J. Cepeda, John T. Wixted, and Doug Rohrer. 2005. When does feedback facilitate learning of words? Journal of Experimental Psychology: Learning, Memory, and Cognition 31, 1 (2005), 3.

[35]

Philip I. Pavlik and John R. Anderson. 2008. Using a model to compute the optimal schedule of practice. Journal of Experimental Psychology: Applied 14, 2 (2008), 101.

[36]

Mary A. Pyc and Katherine A. Rawson. 2009. Testing the retrieval effort hypothesis: Does greater difficulty correctly recalling information lead to higher levels of memory? Journal of Memory and Language 60, 4 (2009), 437--447.

[37]

Yumeng Qiu, Yingmei Qi, Hanyuan Lu, Zachary A. Pardos, and Neil T. Heffernan. 2011. Does Time Matter? Modeling the Effect of Time with Bayesian Knowledge Tracing. In Proceedings of the 4th International Conference on Educational Data Mining. 139--148.

[38]

Katherine A. Rawson and John Dunlosky. 2011. Optimizing schedules of retrieval practice for durable and efficient learning: How much is enough? Journal of Experimental Psychology: General 140, 3 (2011), 283.

[39]

Katherine A. Rawson, Kalif E. Vaughn, and Shana K. Carpenter. 2015. Does the benefit of testing depend on lag, and if so, why? Evaluating the elaborative retrieval hypothesis. Memory & Cognition 43, 4 (2015), 619--633.

[40]

Henry L. Roediger III and Andrew C. Butler. 2011. The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences 15 (2011), 20--27. Issue 1.

[41]

Henry L. Roediger III and Jeffrey D. Karpicke. 2006a. The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science 1, 3 (2006), 181--210.

[42]

Henry L. Roediger III and Jeffrey D. Karpicke. 2006b. Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science 17, 3 (2006), 249--255.

[43]

Henry L. Roediger III, Adam L. Putnam, and Megan A. Smith. 2011. Ten benefits of testing and their applications to educational practice. In Psychology of Learning and Motivation. Vol. 55. Elsevier, 1--36.

[44]

Skipper Seabold and Josef Perktold. 2010. Statsmodels: Econometric and statistical modeling with Python. In 9th Python in Science Conference.

[45]

Burr Settles and Brendan Meeder. 2016. A trainable spaced repetition model for language learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1848--1858.

[46]

Steven M. Smith. 1979. Remembering in and out of context. Journal of Experimental Psychology: Human Learning and Memory 4 (1979), 460--471.Issue 5.

[47]

Camille Szmaragd, Paul Clarke, and Fiona Steele. 2013. Subject specific and population average models for binary longitudinal data: a tutorial. Longitudinal and Life Course Studies 4, 2 (2013), 147--165.

[48]

Behzad Tabibian, Utkarsh Upadhyay, Abir De, Ali Zarezade, Bernhard Schölkopf, and Manuel Gomez-Rodriguez. 2019. Enhancing human learning via spaced repetition optimization. Proceedings of the National Academy of Sciences 116, 10 (2019), 3988--3993.

[49]

Yutao Wang and Joseph E. Beck. 2012. Incorporating Factors Influencing Knowledge Retention into a Student Model. In Proceedings of the 5th International Conference on Educational Data Mining.

[50]

Yutao Wang and Neil T. Heffernan. 2011. Towards Modeling Forgetting and Relearning in ITS: Preliminary Analysis of ARRS Data. In Proceedings of the 4th International Conference on Educational Data Mining. 351--352.

[51]

Xiaolu Xiong and Joseph E. Beck. 2014. A study of exploring different schedules of spacing and retrieval interval on mathematics skills in ITS environment. In International Conference on Intelligent Tutoring Systems. Springer, 504--509.

[52]

Xiaolu Xiong, Shoujing Li, and Joseph E. Beck. 2013. Will you get it right next week: Predict delayed performance in enhanced ITS mastery cycle. In The Twenty-Sixth International FLAIRS Conference.

[53]

Xiaolu Xiong, Yan Wang, and Joseph Barbosa Beck. 2015. Improving Students' Long-term Retention Performance: A Study on Personalized Retention Schedules. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge. ACM, 325--329.

Digital Library

Cited By

Matayoshi JUzun H(2022)Learning, forgetting, and the correlation of knowledge in knowledge space theoryJournal of Mathematical Psychology10.1016/j.jmp.2022.102674109(102674)Online publication date: Aug-2022
https://doi.org/10.1016/j.jmp.2022.102674
Chine DBrentley CThomas-Browne CRichey JGul ACarvalho PBranstetter LKoedinger K(2022)Educational Equity Through Combined Human-AI Personalization: A Propensity Matching EvaluationArtificial Intelligence in Education10.1007/978-3-031-11644-5_30(366-377)Online publication date: 27-Jul-2022
https://doi.org/10.1007/978-3-031-11644-5_30
Wenzel KReinhard M(2021)Learning With a Double-Edged Sword? Beneficial and Detrimental Effects of Learning Tests—Taking a First Look at Linkages Among Tests, Later Learning Outcomes, Stress Perceptions, and IntelligenceFrontiers in Psychology10.3389/fpsyg.2021.69358512Online publication date: 31-Aug-2021
https://doi.org/10.3389/fpsyg.2021.693585
Show More Cited By

Index Terms

Studying Retrieval Practice in an Intelligent Tutoring System
1. Applied computing
  1. Education
    1. E-learning
    2. Interactive learning environments
  2. Law, social and behavioral sciences
    1. Psychology

Recommendations

Deep (Un)Learning: Using Neural Networks to Model Retention and Forgetting in an Adaptive Learning System
Artificial Intelligence in Education
Abstract
ALEKS, which stands for “Assessment and LEarning in Knowledge Spaces”, is a web-based, artificially intelligent, adaptive learning and assessment system. Previous work has shown that student knowledge retention within the ALEKS system exhibits the ...
Towards an Intelligent Tutoring System Architecturethat Supports Remedial Tutoring

For successful teaching to take place an intelligent tutoring system has to be able to cope with any student errors that may occur during a tutoring interaction. Remedial tutoring is increasingly viewed as a central part of the overall tutoring process, ...
The effectiveness of an intelligent tutoring system on the attitude and achievement of developmental mathematics students in a community college

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

L@S '20: Proceedings of the Seventh ACM Conference on Learning @ Scale

August 2020

442 pages

ISBN:9781450379519

DOI:10.1145/3386527

General Chair:
David Joyner
Georgia Tech
,
Program Chairs:
Rene Kizilcec
Cornell University
,
Susan Singer
Rollins College

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

L@S '20

L@S '20: Seventh (2020) ACM Conference on Learning @ Scale

August 12 - 14, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
285
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)1

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Matayoshi JUzun H(2022)Learning, forgetting, and the correlation of knowledge in knowledge space theoryJournal of Mathematical Psychology10.1016/j.jmp.2022.102674109(102674)Online publication date: Aug-2022
https://doi.org/10.1016/j.jmp.2022.102674
Chine DBrentley CThomas-Browne CRichey JGul ACarvalho PBranstetter LKoedinger K(2022)Educational Equity Through Combined Human-AI Personalization: A Propensity Matching EvaluationArtificial Intelligence in Education10.1007/978-3-031-11644-5_30(366-377)Online publication date: 27-Jul-2022
https://doi.org/10.1007/978-3-031-11644-5_30
Wenzel KReinhard M(2021)Learning With a Double-Edged Sword? Beneficial and Detrimental Effects of Learning Tests—Taking a First Look at Linkages Among Tests, Later Learning Outcomes, Stress Perceptions, and IntelligenceFrontiers in Psychology10.3389/fpsyg.2021.69358512Online publication date: 31-Aug-2021
https://doi.org/10.3389/fpsyg.2021.693585
Matayoshi JCosyn EUzun H(2021)Evaluating the Impact of Research-Based Updates to an Adaptive Learning SystemArtificial Intelligence in Education10.1007/978-3-030-78270-2_80(451-456)Online publication date: 12-Jun-2021
https://doi.org/10.1007/978-3-030-78270-2_80

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents