research-article

Open access

Empowering Teacher Learning with AI: Automated Evaluation of Teacher Attention to Student Ideas during Argumentation-focused Discussion

Authors:

Tanya Nazaretsky,

Jamie N. Mikeska,

Beata Beigman KlebanovAuthors Info & Claims

LAK2023: LAK23: 13th International Learning Analytics and Knowledge Conference

Pages 122 - 132

https://doi.org/10.1145/3576050.3576067

Published: 13 March 2023 Publication History

All formats PDF

Abstract

Engaging students in argument from evidence is an essential goal of science education. This is a complex skill to develop; recent research in science education proposed the use of simulated classrooms to facilitate the practice of the skill. We use data from one such simulated environment to explore whether automated analysis of the transcripts of the teacher’s interaction with the simulated students using Natural Language Processing techniques could yield an accurate evaluation of the teacher’s performance. We are especially interested in explainable models that could also support formative feedback. The results are encouraging: Not only can the models score the transcript as well as humans can, but they can also provide justifications for the scores comparable to those provided by human raters.

References

[1]

Sterling Alic, Dorottya Demszky, Zid Mancenido, Jing Liu, Heather Hill, and Dan Jurafsky. 2022. Computationally Identifying Funneling and Focusing Questions in Classroom Discourse. In 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022). 224–233.

[2]

Meghan Bathgate, Amanda Crowell, Christian Schunn, Mac Cannady, and Rena Dorph. 2015. The learning benefits of being willing and able to engage in scientific argumentation. International Journal of Science Education 37, 10 (2015), 1590–1612.

[3]

Amanda Benedict-Chambers and Roberta Aram. 2017. Tools for teacher noticing: Helping preservice teachers notice and analyze student thinking and scientific practice use. Journal of Science Teacher Education 28, 3 (2017), 294–318.

[4]

Amanda Benedict-Chambers, Sarah J Fick, and Anna Maria Arias. 2020. Preservice Teachers’ Noticing of Instances for Revision during Rehearsals: A Comparison across Three University Contexts. Journal of Science Teacher Education 31, 4 (2020), 435–459.

[5]

Leema K Berland and David Hammer. 2012. Framing for scientific argumentation. Journal of Research in Science Teaching 49, 1 (2012), 68–94.

[6]

Leema K Berland and Katherine L McNeill. 2010. A learning progression for scientific argumentation: Understanding student work and designing supportive instructional contexts. Science Education 94, 5 (2010), 765–793.

[7]

Julie Cohen, Vivian Wong, Anandita Krishnamachari, and Rebekah Berlin. 2020. Teacher coaching in a simulated environment. Educational Evaluation and Policy Analysis 42, 2 (2020), 208–231.

[8]

Christine P Dancey and John Reidy. 2017. Statistics without maths for psychology. Pearson London.

[9]

Elizabeth A Davis, Matthew Kloser, Andrea Wells, Mark Windschitl, Janet Carlson, and John-Carlos Marino. 2017. Teaching the practice of leading sense-making discussions in science: Science teacher educators using rehearsals. Journal of Science Teacher Education 28, 3 (2017), 275–293.

[10]

Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. 2021. Measuring conversational uptake: A case study on student-teacher interactions. arXiv preprint arXiv:2106.03873(2021).

[11]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018).

[12]

Lisa A Dieker, Charles E Hughes, Michael C Hynes, and Carrie Straub. 2017. Using simulated virtual environments to improve teacher performance. School-University Partnerships 10, 3 (2017), 62–81.

[13]

Patrick J Donnelly, Nathan Blanchard, Borhan Samei, Andrew M Olney, Xiaoyi Sun, Brooke Ward, Sean Kelly, Martin Nystran, and Sidney K D’Mello. 2016. Automatic teacher modeling from live classroom audio. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization. 45–53.

Digital Library

[14]

Fayyaz Ahmad Faize, Waqar Husain, and Farhat Nisar. 2017. A critical review of scientific argumentation in science education. Eurasia Journal of Mathematics, Science and Technology Education 14, 1(2017), 475–483.

[15]

Evan J Fishman, Hilda Borko, Jonathan Osborne, Florencia Gomez, Stephanie Rafanelli, Emily Reigh, Anita Tseng, Susan Million, and Eric Berson. 2017. A practice-based professional development program to support scientific argumentation from evidence in the elementary classroom. Journal of Science Teacher Education 28, 3 (2017), 222–249.

[16]

Anthony Tuf Francis, Mark Olson, Paul J Weinberg, and Amanda Stearns-Pfeiffer. 2018. Not just for novices: The programmatic impact of practice-based teacher education. Action in Teacher Education 40, 2 (2018), 119–132.

[17]

Vetti Giri and MU Paily. 2020. Effect of scientific argumentation on the development of critical thinking. Science & Education 29, 3 (2020), 673–690.

[18]

J Bryan Henderson, Katherine L McNeill, María González-Howard, Kevin Close, and Mat Evans. 2018. Key challenges and future directions for educational research on scientific argumentation. Journal of Research in Science Teaching 55, 1 (2018), 5–18.

[19]

Matthew Huggins, Sharifa Alghowinem, Sooyeon Jeong, Pedro Colon-Hernandez, Cynthia Breazeal, and Hae Won Park. 2021. Practical guidelines for intent recognition: Bert with minimal training data evaluated in real-world hri application. In Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction. 341–350.

Digital Library

[20]

Emily Jensen, Samuel L. Pugh, and Sidney K. D’Mello. 2021. A deep transfer learning approach to modeling teacher discourse in the classroom. In LAK21: 11th International Learning Analytics and Knowledge Conference. 302–312.

[21]

David Kaufman and Alice Ireland. 2016. Enhancing teacher education with simulations. TechTrends 60, 3 (2016), 260–267.

[22]

Sean Kelly, Andrew M Olney, Patrick Donnelly, Martin Nystrand, and Sidney K D’Mello. 2018. Automatically measuring question authenticity in real-world classrooms. Educational Researcher 47, 7 (2018), 451–464.

[23]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014).

[24]

René F Kizilcec. 2016. How much information? Effects of transparency on trust in an algorithmic interface. In Proceedings of the 2016 CHI conference on human factors in computing systems. 2390–2395.

Digital Library

[25]

Anastassia Loukina, Klaus Zechner, James Bruno, and Beata Beigman Klebanov. 2018. Using exemplar responses for training and evaluating automated speech scoring systems. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications. 1–12.

[26]

Katherine L McNeill and Amanda M Knight. 2013. Teachers’ pedagogical content knowledge of scientific argumentation: The impact of professional development on K–12 teachers. Science Education 97, 6 (2013), 936–972.

[27]

Jamie N. Mikeska and Heather Howell. 2020. Simulations as practice-based spaces to support elementary teachers in learning how to facilitate argumentation-focused science discussions. Journal of Research in Science Teaching 57, 9 (2020), 1356–1399.

[28]

Jamie N. Mikeska and Heather Howell. 2021. Authenticity perceptions in virtual environments. Information and Learning Sciences 122, 7/8 (2021), 480–502.

[29]

Jamie N. Mikeska, Heather Howell, Joseph Ciofalo, Adam Devitt, Elizabeth Orlandi, Kenneth King, Michelle Lipari, and Glenn Simonelli. 2021. Conceptualization and development of a performance task for assessing and building elementary preservice teachers’ ability to facilitate argumentation-focused discussions in mathematics: The mystery powder task (Research Memorandum No. RM-21-06), ETS.

[30]

Jamie N. Mikeska, Heather Howell, Lisa Dieker, and Michael Hynes. 2021. Understanding the role of simulations in K-12 mathematics and science teacher education: Outcomes from a teacher education simulation conference. Contemporary Issues in Technology and Teacher Education 21, 3 (2021), 781–812.

[31]

Jamie N. Mikeska, Heather Howell, and Devon Kinsey. In press. Do simulated teaching experiences impact elementary preservice teachers’ ability to facilitate argumentation-focused discussions in mathematics and science?Journal of Teacher Education(In press).

[32]

Jamie N. Mikeska, Heather Howell, and Carrie Straub. 2019. Using Performance Tasks within Simulated Environments to Assess Teachers’ Ability to Engage in Coordinated, Accumulated, and Dynamic (CAD) Competencies. International Journal of Testing 19, 2 (2019), 128–147.

[33]

Jamie N. Mikeska, Calli Shekell, Adam V. Maltese, Justin Reich, Meredith Thompson, Heather Howell, Pamela S. Lottero-Perdue, and Meredith Park Rogers. 2022. Exploring the Potential of an Online Suite of Practice-Based Activities for Supporting Preservice Elementary Teachers in Learning How to Facilitate Argumentation-Focused Discussions in Mathematics and Science. In Proceedings of Society for Information Technology & Teacher Education International Conference 2022, Elizabeth Langran (Ed.).

[34]

Tanya Nazaretsky, Moriah Ariely, Mutlu Cukurova, and Giora Alexandron. 2022. Teachers’ trust in AI-powered educational technology and a professional development program to improve it. British Journal of Educational Technology 53, 4 (2022), 914–931.

[35]

Tanya Nazaretsky, Mutlu Cukurova, and Giora Alexandron. 2022. An Instrument for Measuring Teachers’ Trust in AI-Based Educational Technology. In LAK22: 12th international learning analytics and knowledge conference. 56–66.

[36]

Ani Nenkova, Rebecca Passonneau, and Kathleen McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Language Processing (TSLP) 4, 2(2007), 4–es.

Digital Library

[37]

NGSS Lead States. 2013. Next generation science standards: For states, by states (Vol 1) Washington.

[38]

Amy Ogan. 2019. Reframing classroom sensing: Promise and peril. Interactions 26, 6 (2019), 26–32.

Digital Library

[39]

Jonathan F Osborne, Hilda Borko, Evan Fishman, Florencia Gomez Zaccarelli, Eric Berson, KC Busch, Emily Reigh, and Anita Tseng. 2019. Impacts of a practice-based professional development program on elementary teachers’ facilitation of and student engagement with scientific argumentation. American Educational Research Journal 56, 4 (2019), 1067–1112.

[40]

Joe Oyler. 2019. Exploring teacher contributions to student argumentation quality. Studia Paedagogica 24, 4 (2019), 173–198.

[41]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 8024–8035.

Digital Library

[42]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.

Digital Library

[43]

Luis P Prieto, Kshitij Sharma, Pierre Dillenbourg, and María Jesús. 2016. Teaching analytics: towards automatic extraction of orchestration graphs using wearable sensors. In LAK16: 6th International Learning Analytics and Knowledge Conference. 148–157.

Digital Library

[44]

Luis Pablo Prieto, Kshitij Sharma, Łukasz Kidzinski, María Jesús Rodríguez-Triana, and Pierre Dillenbourg. 2018. Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning 34, 2 (2018), 193–203.

[45]

GO Discuss Project. 2021. Scoring. https://doi.org/10.5064/F6NJU10I

[46]

Victor Sampson and Margaret R Blanchard. 2012. Science teachers and scientific argumentation: Trends in views and practice. Journal of Research in Science Teaching 49, 9 (2012), 1122–1148.

[47]

Victor Sampson, Jonathon Grooms, and Joi Phelps Walker. 2011. Argument-Driven Inquiry as a way to help students learn how to participate in scientific argumentation and craft written arguments: An exploratory study. Science Education 95, 2 (2011), 217–257.

[48]

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108(2019).

[49]

Abhijit Suresh, Jennifer Jacobs, Vivian Lai, Chenhao Tan, Wayne Ward, James H Martin, and Tamara Sumner. 2021. Using transformers to provide teachers with personalized feedback on their classroom discourse: The TalkMoves application. arXiv preprint arXiv:2105.07949(2021).

[50]

Abhijit Suresh, Tamara Sumner, Jennifer Jacobs, Bill Foland, and Wayne Ward. 2019. Automating analysis and feedback to improve mathematics teachers’ classroom discourse. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 9721–9728.

Digital Library

[51]

Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys (csur) 53, 3 (2020), 1–34.

Digital Library

[52]

Zuowei Wang, Kevin Miller, and Kai Cortina. 2013. Using the LENA in Teacher Training: Promoting Student Involement through automated feedback. 4 (2013), 290–305.

Cited By

Booth BJacobs JBush JMilne BFischaber TDMello S(2024)Human-tutor Coaching Technology (HTCT): Automated Discourse Analytics in a Coached Tutoring ModelProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636937(725-735)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636937
Cosentino GAnton JSharma KGelsomini MGiannakos MAbrahamson D(2024)Hybrid teaching intelligence: Lessons learned from an embodied mathematics learning experienceBritish Journal of Educational Technology10.1111/bjet.13525Online publication date: 19-Oct-2024
https://doi.org/10.1111/bjet.13525
Cánovas Reverte ÓGonzález Férez PGarcía Clemente FPardo García F(2024)Analyzing Wooclap’s Competition Mode With AI Through Classroom RecordingsIEEE Revista Iberoamericana de Tecnologias del Aprendizaje10.1109/RITA.2024.345886519(220-229)Online publication date: 2024
https://doi.org/10.1109/RITA.2024.3458865
Show More Cited By

Index Terms

Empowering Teacher Learning with AI: Automated Evaluation of Teacher Attention to Student Ideas during Argumentation-focused Discussion

Recommendations

Teacher and student perspectives on a blended learning intensive english program writing course
A review of empirical research on blended learning in teacher education programs

Although blended learning has been considered as an important alternative approach that can overcome various limitations related to both face-to-face and online learning, there is relatively limited empirical studies on blended learning approach in ...
Elementary Teacher Perceptions of Blended Learning in the Digital Classroom

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK2023: LAK23: 13th International Learning Analytics and Knowledge Conference

March 2023

692 pages

ISBN:9781450398657

DOI:10.1145/3576050

Program Chairs:
Isabel Hilliger
Pontificia Universidad Católica de Chile, Chile
,
Hassan Khosravi
University of Queensland, Australia
,
Bart Rienties
Open University, United Kingdom
,
Shane Dawson
University of South Australia, Australia

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 March 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

LAK 2023

LAK 2023: 13th International Learning Analytics and Knowledge Conference

March 13 - 17, 2023

TX, Arlington, USA

Acceptance Rates

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
936
Total Downloads

Downloads (Last 12 months)625
Downloads (Last 6 weeks)97

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Booth BJacobs JBush JMilne BFischaber TDMello S(2024)Human-tutor Coaching Technology (HTCT): Automated Discourse Analytics in a Coached Tutoring ModelProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636937(725-735)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636937
Cosentino GAnton JSharma KGelsomini MGiannakos MAbrahamson D(2024)Hybrid teaching intelligence: Lessons learned from an embodied mathematics learning experienceBritish Journal of Educational Technology10.1111/bjet.13525Online publication date: 19-Oct-2024
https://doi.org/10.1111/bjet.13525
Cánovas Reverte ÓGonzález Férez PGarcía Clemente FPardo García F(2024)Analyzing Wooclap’s Competition Mode With AI Through Classroom RecordingsIEEE Revista Iberoamericana de Tecnologias del Aprendizaje10.1109/RITA.2024.345886519(220-229)Online publication date: 2024
https://doi.org/10.1109/RITA.2024.3458865
García FCánovas ÓClemente F(2024)Exploring AI Techniques for Generalizable Teaching Practice IdentificationIEEE Access10.1109/ACCESS.2024.345691512(134702-134713)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3456915
Canovas OGarcia-Clemente FPardo F(2023)AI-driven Teacher Analytics: Informative Insights on Classroom Activities2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE)10.1109/TALE56641.2023.10398309(1-8)Online publication date: 28-Nov-2023
https://doi.org/10.1109/TALE56641.2023.10398309
Amarasinghe IMarques FOrtiz-Beltrán AHernández-Leo D(2023)Generative Pre-trained Transformers for Coding Text Data? An Analysis with Classroom Orchestration DataResponsive and Sustainable Educational Futures10.1007/978-3-031-42682-7_3(32-43)Online publication date: 4-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-42682-7_3

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents