research-article

Open access

OATutor: An Open-source Adaptive Tutoring System and Curated Content Library for Learning Sciences Research

Authors:

Zachary A. Pardos,

Ioannis Anastasopoulos,

Shreya K. Sheel,

Ethan ZhangAuthors Info & Claims

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Article No.: 416, Pages 1 - 17

https://doi.org/10.1145/3544548.3581574

Published: 19 April 2023 Publication History

All formats PDF Full text

Abstract

Despite decades long establishment of effective tutoring principles, no adaptive tutoring system has been developed and open-sourced to the research community. The absence of such a system inhibits researchers from replicating adaptive learning studies and extending and experimenting with various tutoring system design directions. For this reason, adaptive learning research is primarily conducted on a small number of proprietary platforms. In this work, we aim to democratize adaptive learning research with the introduction of the first open-source adaptive tutoring system based on Intelligent Tutoring System principles. The system, we call Open Adaptive Tutor (OATutor), has been iteratively developed over three years with field trials in classrooms drawing feedback from students, teachers, and researchers. The MIT-licensed source code includes three creative commons (CC BY) textbooks worth of algebra problems, with tutoring supports authored by the OATutor project. Knowledge Tracing, an A/B testing framework, and LTI support are included.

Supplementary Material

Supplemental Materials (3544548.3581574-supplemental-materials.zip)

Download
116.94 MB

MP4 File (3544548.3581574-video-figure.mp4)

Video Figure

Download
68.77 MB

MP4 File (3544548.3581574-video-preview.mp4)

Video Preview

Download
8.92 MB

References

[1]

Vincent Aleven and Kenneth R Koedinger. 2013. Knowledge component (KC) approaches to learner modeling. Design Recommendations for Intelligent Tutoring Systems 1 (2013), 165–182.

[2]

Vincent Aleven, Bruce M McLaren, Jonathan Sewall, and Kenneth R Koedinger. 2006. The cognitive tutor authoring tools (CTAT): Preliminary evaluation of efficiency gains. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems. Springer, 61–70.

Digital Library

[3]

Vincent Aleven, Ido Roll, Bruce M McLaren, and Kenneth R Koedinger. 2016. Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education 26, 1(2016), 205–223.

[4]

John R Anderson, Albert T Corbett, Kenneth R Koedinger, and Ray Pelletier. 1995. Cognitive tutors: Lessons learned. The journal of the learning sciences 4, 2 (1995), 167–207.

[5]

Anirudhan Badrinath, Frederic Wang, and Zachary Pardos. 2021. pyBKT: An Accessible Python Library of Bayesian Knowledge Tracing Models. In Proceedings of the 14th International Conference on Educational Data Mining. 468–474.

[6]

Meltem Huri Baturay. 2015. An overview of the world of MOOCs. Procedia-Social and Behavioral Sciences 174 (2015), 427–433.

[7]

Sameer Bhatnagar, Amal Zouaq, Michel C Desmarais, and Elizabeth Charles. 2020. Learnersourcing Quality Assessment of Explanations for Peer Instruction. In European Conference on Technology Enhanced Learning. Springer, 144–157.

[8]

Paul De Bra, David Smits, Kees van der Sluijs, Alexandra I Cristea, Jonathan Foss, Christian Glahn, and Christina M Steiner. 2013. GRAPPLE: Learning management systems meet adaptive learning environments. In Intelligent and adaptive educational-learning systems. Springer, 133–160.

[9]

Hao Cen, Kenneth R Koedinger, and Brian Junker. 2007. Is Over Practice Necessary?-Improving Learning Efficiency with the Cognitive Tutor through Educational Data Mining.Frontiers in artificial intelligence and applications 158 (2007), 511.

[10]

Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Jineon Baek, Chan Bae, Byungsoo Kim, and Jaewe Heo. 2020. Ednet: A large-scale hierarchical dataset in education. In Proceedings of the 21st International Conference on Artificial Intelligence in Education. Springer, 69–73.

Digital Library

[11]

Albert Corbett and Holly Trask. 2000. Instructional Interventions in Computer-Based Tutoring: Differential Impact on Learning Time and Accuracy. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands) (CHI ’00). ACM, 97–104. https://doi.org/10.1145/332040.332412

Digital Library

[12]

Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 4 (1994), 253–278.

[13]

Albert T. Corbett and John R. Anderson. 2001. Locus of Feedback Control in Computer-Based Tutoring: Impact on Learning Rate, Achievement and Attitudes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI ’01). ACM, 245–252. https://doi.org/10.1145/365024.365111

Digital Library

[14]

Gilan M El Saadawi, Roger Azevedo, Melissa Castine, Velma Payne, Olga Medvedeva, Eugene Tseytlin, Elizabeth Legowski, Drazen Jukic, and Rebecca S Crowley. 2010. Factors affecting feeling-of-knowing in a medical intelligent tutoring system: the role of immediate feedback as a metacognitive scaffold. Advances in Health Sciences Education 15, 1 (2010), 9–30.

[15]

Enrique Estellés-Arolas and Fernando González-Ladrón de Guevara. 2012. Towards an integrated crowdsourcing definition. Journal of Information Science 38, 2 (2012), 189–200. https://doi.org/10.1177/0165551512437638

Digital Library

[16]

Mingyu Feng, Neil Heffernan, and Joseph E Beck. 2009. Using learning decomposition to analyze instructional effectiveness in the ASSISTment system. In Proceedings of the 14th International Conference Artificial Intelligence in Education. IOS Press, 523–530.

[17]

Christian Fischer, Zachary A Pardos, Ryan Shaun Baker, Joseph Jay Williams, Padhraic Smyth, Renzhe Yu, Stefan Slater, Rachel Baker, and Mark Warschauer. 2020. Mining big data in education: Affordances and challenges. Review of Research in Education 44, 1 (2020), 130–160. https://doi.org/10.3102/0091732X20903304

[18]

Ilya M Goldin, Kenneth R Koedinger, and Vincent Aleven. 2012. Learner Differences in Hint Processing. In Proceedings of the 5th International Conference on Educational Data Mining. ERIC, 73–80.

[19]

Cristina Heffernan, Neil Heffernan, Ashish Maharjan, Leena Razzaq, Prawal Shrestha, and Xing Wei. 2009. Are Worked Examples an Effective Feedback Mechanism During Problem Solving?. In Proceedings of the 31st Annual Meeting of the Cognitive Science Society. 1294–1299.

[20]

Neil T Heffernan and Cristina Lindquist Heffernan. 2014. The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24, 4(2014), 470–497.

[21]

Bryan Hobbs. 2013. Improving Educational Content–A Web-based Intelligent Tutoring System with Support for Teacher Collaboration. Ph. D. Dissertation. Worcester Polytechnic Institute.

[22]

Scott E. Hudson and Jennifer Mankoff. 2014. Concepts, Values, and Methods for Technical Human–Computer Interaction Research. Springer New York, 69–93. https://doi.org/10.1007/978-1-4939-0378-8_4

[23]

Sally Johnstone. 2005. Forum on the Impact of Open Courseware for Higher Education in Developing Countries–Final Report. Education Quarterly 3(2005), 15–18.

[24]

David Kabugo. 2020. Utilizing Open Education Resources to Enhance Students’ Learning Outcomes during the COVID-19 Schools Lockdown: A Case of Using Kolibri by Selected Government Schools in Uganda.Journal of Learning for Development 7, 3 (2020), 447–458.

[25]

Juho Kim. 2015. Learnersourcing: improving learning with collective learner activity. Ph. D. Dissertation. Massachusetts Institute of Technology.

[26]

Ryung S Kim, Rob Weitz, Neil T Heffernan, and Nathan Krach. 2009. Tutored problem solving vs.“pure” worked examples. In Proceedings of the 31st Annual Meeting of the Cognitive Science Society. 3121–3126.

[27]

Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing User Studies with Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). ACM, 453–456. https://doi.org/10.1145/1357054.1357127

Digital Library

[28]

René F. Kizilcec. 2016. How Much Information? Effects of Transparency on Trust in an Algorithmic Interface. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). ACM, 2390–2395. https://doi.org/10.1145/2858036.2858402

Digital Library

[29]

Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science 36, 5 (2012), 757–798.

[30]

Kenneth R Koedinger, Ryan SJ d Baker, Kyle Cunningham, Alida Skogsholm, Brett Leber, and John Stamper. 2010. A Data Repository for the EDM Community: The PSLC DataShop. In Handbook of Educational Data Mining. CRC Press, 65–78.

[31]

Timothy Shin Heng Mak and Albert Lam. 2022. Two-Stage Auction Mechanism for Long-Term Participation in Crowdsourcing. arXiv preprint arXiv:2202.10064(2022).

[32]

John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn Eastmond. 2010. The Scratch Programming Language and Environment. ACM Trans. Comput. Educ. 10, 4, Article 16 (nov 2010), 15 pages. https://doi.org/10.1145/1868358.1868363

Digital Library

[33]

Antonija Mitrovic, Brent Martin, Pramuditha Suraweera, Konstantin Pavlovich Zakharov, Nancy Milik, Jay Holland, and Nicholas McGuigan. 2009. ASPIRE: An Authoring System and Deployment Environment for Constraint-Based Tutors. International Journal of Artificial Intelligence in Education 19 (2009), 155–188.

Digital Library

[34]

Andrew McGregor Olney. 2009. GnuTutor: An open source intelligent tutoring system based on AutoTutor. In 2009 AAAI Fall Symposium Series.

[35]

Andrew M Olney, Sidney D’Mello, Natalie Person, Whitney Cade, Patrick Hays, Claire Williams, Blair Lehman, and Arthur Graesser. 2012. Guru: A Computer Tutor That Models Expert Human Tutors. In Proceedings of the 11th International Intelligent Tutoring Systems Conference. Springer, 256–261.

Digital Library

[36]

Eleanor O’Rourke, Erik Andersen, Sumit Gulwani, and Zoran Popović. 2015. A Framework for Automatically Generating Interactive Instructional Scaffolding. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). ACM, 1545–1554. https://doi.org/10.1145/2702123.2702580

Digital Library

[37]

John F Pane, Beth Ann Griffin, Daniel F McCaffrey, and Rita Karam. 2014. Effectiveness of cognitive tutor algebra I at scale. Educational Evaluation and Policy Analysis 36, 2 (2014), 127–144.

[38]

Zachary A Pardos, Anthony Whyte, and Kevin Kao. 2016. moocRP: Enabling open learning analytics with an open source platform for data distribution, analysis, and visualization. Technology, Knowledge and Learning 21, 1 (2016), 75–98. https://doi.org/10.1007/s10758-015-9268-2

[39]

Thanaporn Patikorn and Neil T. Heffernan. 2020. Effectiveness of Crowd-Sourcing On-Demand Assistance from Teachers in Online Learning Platforms. In Proceedings of the Seventh ACM Conference on Learning @ Scale (Virtual Event, USA) (L@S ’20). ACM, New York, NY, USA, 115–124. https://doi.org/10.1145/3386527.3405912

Digital Library

[40]

Rebecca Pitt. 2015. Mainstreaming open textbooks: Educator perspectives on the impact of OpenStax college open textbooks. International Review of Research in Open and Distributed Learning 16, 4(2015).

[41]

Darrell Porcello and Sherry Hsi. 2013. Crowdsourcing and curating online education resources. Science 341, 6143 (2013), 240–241.

[42]

Ethan Prihar, Thanaporn Patikorn, Anthony Botelho, Adam Sales, and Neil Heffernan. 2021. Toward Personalizing Students’ Education with Crowdsourced Tutoring. ACM, New York, NY, USA, 37–45. https://doi.org/10.1145/3430895.3460130

Digital Library

[43]

Leena Razzaq and Neil T Heffernan. 2006. Scaffolding vs. hints in the Assistment System. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems. Springer, 635–644.

Digital Library

[44]

Leena Razzaq, Jozsef Patvarczki, Shane F. Almeida, Manasi Vartak, Mingyu Feng, Neil T. Heffernan, and Kenneth R. Koedinger. 2009. The ASSISTment Builder: Supporting the Life Cycle of Tutoring System Content Creation. IEEE Transactions on Learning Technologies 2, 2 (2009), 157–166. https://doi.org/10.1109/TLT.2009.23

Digital Library

[45]

Steven Ritter, Thomas K Harris, Tristan Nixon, Daniel Dickison, R Charles Murray, and Brendon Towle. 2009. Reducing the Knowledge Tracing Space. In Proceedings of the 2nd International Conference on Educational Data Mining. 151–160.

[46]

Yigal Rosen, Ilia Rushkin, Rob Rubin, Liberty Munson, Andrew Ang, Gregory Weber, Glenn Lopez, and Dustin Tingley. 2018. The Effects of Adaptive Learning in a Massive Open Online Course on Learners’ Skill Development. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (London, United Kingdom) (L@S ’18). ACM, Article 6, 8 pages. https://doi.org/10.1145/3231644.3231651

Digital Library

[47]

Anjali Singh, Christopher Brooks, and Shayan Doroudi. 2022. Learnersourcing in Theory and Practice: Synthesizing the Literature and Charting the Future. In Proceedings of the Ninth ACM Conference on Learning @ Scale (New York City, NY, USA) (L@S ’22). Association for Computing Machinery, New York, NY, USA, 234–245. https://doi.org/10.1145/3491140.3528277

Digital Library

[48]

John Stamper and Zachary A Pardos. 2016. The 2010 KDD Cup Competition Dataset: Engaging the machine learning community in predictive learning analytics. Journal of Learning Analytics 3, 2 (2016), 312–316. https://doi.org/10.18608/jla.2016.32.16

[49]

Robert G. Taylor, A. Smith, Samuel P. Leeman-Munk, Bradford W. Mott, and James C. Lester. 2014. Towards ITS Authoring Tools for Domain Experts. In IntelliMedia Group.

[50]

Ilkka Tuomi. 2013. Open educational resources and the transformation of education. European Journal of Education 48, 1 (2013), 58–78.

[51]

Terrence E Turner, Michael A Macasek, Goss Nuzzo-Jones, Neil T Heffernan, and Kenneth R Koedinger. 2005. The Assistment Builder: A Rapid Development Tool for ITS. In Proceedings of the 12th International Conference on Artificial Intelligence in Education. 929–931.

[52]

Daniel Weitekamp, Erik Harpstead, and Ken R. Koedinger. 2020. An Interaction Design for Machine Teaching to Develop AI Tutors. ACM, 1–11. https://doi.org/10.1145/3313831.3376226

Digital Library

[53]

John Zimmerman, Jodi Forlizzi, and Shelley Evenson. 2007. Research through Design as a Method for Interaction Design Research in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’07). ACM, 493–502. https://doi.org/10.1145/1240624.1240704

Digital Library

Cited By

Ksemidov BAbgaryan K(2024)Adaptive Intelligent Tutoring SystemModelling and Data AnalysisМоделирование и анализ данных10.17759/mda.202414021114:2(152-165)Online publication date: 1-Jul-2024
https://doi.org/10.17759/mda.2024140211
Pardos ZBhandari S(2024)ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skillsPLOS ONE10.1371/journal.pone.030401319:5(e0304013)Online publication date: 24-May-2024
https://doi.org/10.1371/journal.pone.0304013
Gautam SZhang XRosson MFarzan RLópez CCardoso Llach DQuercia DMustafa MNiu SWong-Villacrés M(2024)Towards Dynamic Learning: A Framework for Simulating Adaptive Learning SystemsCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681913(603-608)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3678884.3681913
Show More Cited By

Index Terms

OATutor: An Open-source Adaptive Tutoring System and Curated Content Library for Learning Sciences Research
1. Applied computing
  1. Education
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools

Recommendations

Introducing an Open-source Adaptive Tutoring System to Accelerate Learning Sciences Experimentation
L@S '23: Proceedings of the Tenth ACM Conference on Learning @ Scale

Learning @ Scale has embraced movements that spread access to education through open and free platforms of learning. In this tutorial, we introduce OATutor (recently published at CHI'23), the field's first free and open-source adaptive tutoring system ...
Towards an Intelligent Tutoring System Architecturethat Supports Remedial Tutoring

For successful teaching to take place an intelligent tutoring system has to be able to cope with any student errors that may occur during a tutoring interaction. Remedial tutoring is increasingly viewed as a central part of the overall tutoring process, ...
An analysis of the research on adaptive learning: the next generation of e-learning

This study examines the evidence for the effectiveness of adaptive learning and the satisfaction level of students when using this type of learning. It first analyses the different classifications of adaptive learning systems existing in the literature, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

April 2023

14911 pages

ISBN:9781450394215

DOI:10.1145/3544548

Editors:
Albrecht Schmidt
LMU Munich, Germany60028717
,
Kaisa Väänänen
Tampere University, Finland60011170
,
Tesh Goyal
Google Research, USA60006191
,
Per Ola Kristensson
University of Cambridge, UK60031101
,
Anicia Peters
University of Namibia, Namibia60072704
,
Stefanie Mueller
Massachusetts Institute of Technology, USA60022195
,
Julie R. Williamson
University of Glasgow, UK60001490
,
Max L. Wilson
University of Nottingham, UK60015138

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '23

Sponsor:

SIGCHI

CHI '23: CHI Conference on Human Factors in Computing Systems

April 23 - 28, 2023

Hamburg, Germany

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
6,189
Total Downloads

Downloads (Last 12 months)4,391
Downloads (Last 6 weeks)394

Reflects downloads up to 30 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ksemidov BAbgaryan K(2024)Adaptive Intelligent Tutoring SystemModelling and Data AnalysisМоделирование и анализ данных10.17759/mda.202414021114:2(152-165)Online publication date: 1-Jul-2024
https://doi.org/10.17759/mda.2024140211
Pardos ZBhandari S(2024)ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skillsPLOS ONE10.1371/journal.pone.030401319:5(e0304013)Online publication date: 24-May-2024
https://doi.org/10.1371/journal.pone.0304013
Gautam SZhang XRosson MFarzan RLópez CCardoso Llach DQuercia DMustafa MNiu SWong-Villacrés M(2024)Towards Dynamic Learning: A Framework for Simulating Adaptive Learning SystemsCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681913(603-608)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3678884.3681913
Bradford ALi WGerard LLinn MJoyner DKim MWang XXia M(2024)Comparing Expert and ChatGPT-authored Guidance PromptsProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664669(388-392)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664669
Sheel SAnastasopoulos IPardos Z(2024)Comparing Authoring Experiences with Spreadsheet Interfaces vs GUIsProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636919(598-607)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636919
Filinger WMullen JCohen J(2024)Developing HPC Learning Pathways: Challenges and RecommendationsPractice and Experience in Advanced Research Computing 2024: Human Powered Computing10.1145/3626203.3670575(1-4)Online publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1145/3626203.3670575
Zheng CYuan KGuo BHadi Mogavi RPeng ZMa SMa X(2024)Charting the Future of AI in Project-Based Learning: A Co-Design Exploration with StudentsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642807(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642807
Kwak YPardos Z(2024)Bridging large language model disparities: Skill tagging of multilingual educational contentBritish Journal of Educational Technology10.1111/bjet.1346555:5(2039-2057)Online publication date: 8-May-2024
https://doi.org/10.1111/bjet.13465
Viberg OMutimukwe CHrastinski SCerratto‐Pargman TLilliesköld J(2024)Exploring teachers' (future) digital assessment practices in higher education: Instrument and model developmentBritish Journal of Educational Technology10.1111/bjet.1346255:6(2597-2616)Online publication date: 30-Mar-2024
https://doi.org/10.1111/bjet.13462
R SN GS SR SB SB V(2024)Intelligent Tutoring Systems using Long Short-Term Memory Networks and Bayesian Knowledge Tracing2024 5th International Conference on Mobile Computing and Sustainable Informatics (ICMCSI)10.1109/ICMCSI61536.2024.00010(24-29)Online publication date: 18-Jan-2024
https://doi.org/10.1109/ICMCSI61536.2024.00010
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View full text|Download PDF

View Table of Conten