research-article

Teaching analytics: towards automatic extraction of orchestration graphs using wearable sensors

Authors:

Luis P. Prieto,

Kshitij Sharma,

Pierre Dillenbourg,

María JesúsAuthors Info & Claims

LAK '16: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge

Pages 148 - 157

https://doi.org/10.1145/2883851.2883927

Published: 25 April 2016 Publication History

Abstract

'Teaching analytics' is the application of learning analytics techniques to understand teaching and learning processes, and eventually enable supportive interventions. However, in the case of (often, half-improvised) teaching in face-to-face classrooms, such interventions would require first an understanding of what the teacher actually did, as the starting point for teacher reflection and inquiry. Currently, such teacher enactment characterization requires costly manual coding by researchers. This paper presents a case study exploring the potential of machine learning techniques to automatically extract teaching actions during classroom enactment, from five data sources collected using wearable sensors (eye-tracking, EEG, accelerometer, audio and video). Our results highlight the feasibility of this approach, with high levels of accuracy in determining the social plane of interaction (90%, κ=0.8). The reliable detection of concrete teaching activity (e.g., explanation vs. questioning) accurately still remains challenging (67%, κ=0.56), a fact that will prompt further research on multimodal features and models for teaching activity extraction, as well as the collection of a larger multimodal dataset to improve the accuracy and generalizability of these methods.

References

[1]

M. Bienkowski, M. Feng, and B. Means. Enhancing teaching and learning through educational data mining and learning analytics: An issue brief. Technical report, U.S. Department of Education, Office of Educational Technology, 2012.

[2]

N. Blanchard, S. D'Mello, M. Nystrand, and A. M. Olney. Automatic classification of question & answer discourse segments from teacher's speech in classrooms. In Proceedings of the 8th International Conference on Educational Data Mining (EDM 2015), International Educational Data Mining Society, 2015.

[3]

P. Blikstein. Multimodal learning analytics. In Proceedings of the third international conference on learning analytics and knowledge, pages 102--106. ACM, 2013.

Digital Library

[4]

L. Breiman. Random forests. Machine learning, 45(1):5--32, 2001.

Digital Library

[5]

D. Clow. The learning analytics cycle: closing the loop effectively. In Proceedings of the 2nd international conference on learning analytics and knowledge, pages 134--138. ACM, 2012.

Digital Library

[6]

J. Cohen. Statistical power analysis for the behavioral sciences (rev. Lawrence Erlbaum Associates, Inc., 1977.

[7]

P. Dillenbourg. Orchestration Graphs: Modeling Scalable Education. EPFL Press, Lausanne, Switzerland, 1 edition, 2015.

[8]

P. Dillenbourg, S. Järvelä, and F. Fischer. The evolution of research on computer-supported collaborative learning. In Technology-enhanced learning, pages 3--19. Springer, 2009.

[9]

P. Dillenbourg and P. Tchounikine. Flexibility in macro-scripts for computer-supported collaborative learning. Journal of computer assisted learning, 23(1):1--13, 2007.

[10]

S. K. D'Mello, A. M. Olney, N. Blanchard, B. Samei, X. Sun, B. Ward, and S. Kelly. Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pages 557--566. ACM, 2015.

Digital Library

[11]

F. Domínguez, K. Chiluiza, V. Echeverria, and X. Ochoa. Multimodal selfies: Designing a multimodal recording device for students in traditional classrooms. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pages 567--574. ACM, 2015.

Digital Library

[12]

E. Durall and T. Leinonen. Feeler: supporting awareness and reflection about learning through EEG data. In Proceedings of the 5th Workshop on Awareness and Reflection in Technology Enhanced Learning In conjunction with the 10th European Conference on Technology Enhanced Learning, pages 67--73, 2015.

[13]

A. L. Dyckhoff, D. Zielke, M. Bültmann, M. A. Chatti, and U. Schroeder. Design and implementation of a learning analytics toolkit for teachers. Journal of Educational Technology & Society, 15(3):58--76, 2012.

[14]

T. Elias. Learning analytics: The definitions, the processes, and the potential. 2011.

[15]

V. Emin-Martinez, C. Hansen, M. J. Rodríguez-Triana, B. Wasson, Y. Mor, R. Ferguson, and J.-P. Pernin. Towards teacher-led design inquiry of learning. E-learning papers. Special issue on Learning Analytics and Assessment, 36:3--14, 2014.

[16]

R. Ferguson. Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5-6):304--317, 2012.

Digital Library

[17]

D. H. Freedman. Why scientific studies are so often wrong: The streetlight effect. Discover Magazine, 26, 2010.

[18]

J. H. Friedman. Stochastic gradient boosting. Computational Statistics & Data Analysis, 38(4):367--378, 2002.

Digital Library

[19]

G. Gauthier. Using teaching analytics to inform assessment practices in technology mediated problem solving tasks. In Vatrapu et al. {48}.

[20]

T. Giannakopoulos and A. Pikrakis. Introduction to Audio Analysis: A MATLAB® Approach. Academic Press, 2014.

Digital Library

[21]

Y. Hayashi and R. Mizoguchi. Articulation of scenario construction of lessons based on ontological engineering. In Vatrapu et al. {47}.

[22]

D. Hernández-Leo, J. I. Asensio-Pérez, M. Derntl, L. P. Prieto, and J. Chacón. Ilde: Community environment for conceptualizing, authoring and deploying learning activities. In C. Rensing, S. de Freitas, T. Ley, and P. Muñoz-Merino, editors, Open Learning and Teaching in Educational Communities, volume 8719 of Lecture Notes in Computer Science, pages 490--493. Springer International Publishing, 2014.

Digital Library

[23]

J. Jovanović, D. Gašević, C. Brooks, V. Devedžić, and M. Hatala. Loco-analyst: A tool for raising teachers' awareness in online learning environments. In Creating New Learning Experiences on a Global Scale, pages 112--126. Springer, 2007.

Digital Library

[24]

B.-K. Kim, H. Lee, J. Roh, and S.-Y. Lee. Hierarchical committee of deep cnns with exponentially-weighted decision fusion for static facial expression recognition. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI '15, pages 427--434, New York, NY, USA, 2015. ACM.

Digital Library

[25]

D. Laurillard. Teaching as a design science: Building pedagogical patterns for learning and technology. Routledge, 2012.

[26]

Y. Mor, R. Ferguson, and B. Wasson. Editorial: Learning design, teacher inquiry into student learning and learning analytics: A call for action. British Journal of Educational Technology, 46(2):221--229, 2015.

[27]

L.-P. Morency, S. Oviatt, S. Scherer, N. Weibel, and M. Worsley. Icmi 2013 grand challenge workshop on multimodal learning analytics. In Proceedings of the 15th ACM on International conference on multimodal interaction, pages 373--378. ACM, 2013.

Digital Library

[28]

X. Ochoa, K. Chiluiza, G. Méndez, G. Luzardo, B. Guamán, and J. Castells. Expertise estimation based on simple multimodal features. In Proceedings of the 15th ACM on International conference on multimodal interaction, pages 583--590. ACM, 2013.

Digital Library

[29]

X. Ochoa, M. Worsley, K. Chiluiza, and S. Luz. Mla'14: Third multimodal learning analytics workshop and grand challenges. In Proceedings of the 16th International Conference on Multimodal Interaction, pages 531--532. ACM, 2014.

Digital Library

[30]

M. Oliver and K. Trigwell. Can 'blended learning'be redeemed? E-learning and Digital Media, 2(1):17--26, 2005.

[31]

D. Persico and F. Pozzi. Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2):230--248, 2015.

[32]

S. J. Preece, J. Y. Goulermas, L. P. Kenney, D. Howard, K. Meijer, and R. Crompton. Activity identification using body-mounted sensors---a review of classification techniques. Physiological measurement, 30(4):R1, 2009.

[33]

L. P. Prieto, K. Sharma, and P. Dillenbourg. Studying teacher orchestration load in technology-enhanced classrooms. In Design for Teaching and Learning in a Networked World, pages 268--281. Springer, 2015.

Digital Library

[34]

L. P. Prieto, S. Villagrá-Sobrino, I. M. Jorrín-Abellán, A. Martínez-Monés, and Y. Dimitriadis. Recurrent routines: Analyzing and supporting orchestration in technology-enhanced primary classrooms. Computers & Education, 57(1):1214--1227, 2011.

Digital Library

[35]

M. Raca and P. Dillenbourg. System for assessing classroom attention. In Proceedings of the Third International Conference on Learning Analytics and Knowledge, pages 265--269. ACM, 2013.

Digital Library

[36]

S. Rebholz, P. Libbrecht, and W. Müller. Learning analytics as an investigation tool for teaching practitioners. In Vatrapu et al. {47}.

[37]

V. Rivera-Pelayo, E. Lacić, V. Zacharias, and R. Studer. LIM App: Reflecting on Audience Feedback for Improving Presentation Skills. In D. Hernández-Leo, T. Ley, R. Klamma, and A. Harrer, editors, Scaling up Learning for Sustained Impact, number 8095 in Lecture Notes in Computer Science, pages 514--519. Springer Berlin Heidelberg, Sept. 2013.

Digital Library

[38]

M. J. Rodríguez-Triana, A. Martínez-Monés, J. I. Asensio-Pérez, and Y. Dimitriadis. Script-aware monitoring model: Using teachers' pedagogical intentions to guide learning analytics. In Vatrapu et al. {47}.

[39]

R. K. Sawyer. Creative teaching: Collaborative discussion as disciplined improvisation. Educational researcher, 33(2):12--20, 2004.

[40]

G. Siemens. Sensemaking: Beyond analytics as a technical activity. Presentation at the EDUCAUSE ELI 2012 Online Spring Focus Session.

[41]

G. Siemens and P. Long. Penetrating the fog: Analytics in learning and education. EDUCAUSE review, 46(5):30, 2011.

[42]

K. Thompson, R. Martinez-Maldonado, D. Wardak, P. Goodyear, and L. Carvalho. Analysing F2F Collaborative Design and Learning: Experiences in a Design Studio A Place for Learning. In L. P. Prieto, Y. Dimitriadis, A. Harrer, M. Milrad, and J. D. Slotta, editors, Proceedings of the Orchestrated Collaborative Classroom Workshop 2015 co-located with 11th International Conference on Computer Supported Collaborative Learning, pages 25--29, Gothenburg, Sweden, 2015.

[43]

M. van Harmelen and D. Workman. Analytics for learning and teaching. JISC CETIS Analytics Series, 1(3):1--41, 2012.

[44]

R. Vatrapu, C. Teplovs, N. Fujita, and S. Bull. Towards visual analytics for teachers' dynamic diagnostic pedagogical decision-making. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge, pages 93--98. ACM, 2011.

Digital Library

[45]

R. K. Vatrapu. Towards semiology of Teaching Analytics. In Workshop Towards Theory and Practice of Teaching Analytics, at the European Conference on Technology Enhanced Learning, TAPTA'12, Saarbrücken, Germany, 2012.

[46]

R. K. Vatrapu, K. Kocherla, and K. Pantazos. iklassroom: Real-time, real-place teaching analytics. In Vatrapu et al. {48}.

[47]

R. K. Vatrapu, P. Reimann, W. Halb, and S. Bull, editors. Proceedings of the Workshop Towards Theory and Practice of Teaching Analytics (TaPTA 2012), number 894 in CEUR Workshop Proceedings, Aachen, 2012.

[48]

R. K. Vatrapu, P. Reimann, W. Halb, and S. Bull, editors. Proceedings of the 2nd International Workshop on Teaching Analytics (IWTA 2013), number 985 in CEUR Workshop Proceedings, Aachen, 2013.

[49]

M. Worsley. Multimodal learning analytics: enabling the future of learning through multimodal data analysis and interfaces. In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 353--356. ACM, 2012.

Digital Library

[50]

M. Worsley and P. Blikstein. Leveraging multimodal learning analytics to differentiate student learning strategies. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, pages 360--367. ACM, 2015.

Digital Library

[51]

B. Xu and M. Recker. Teaching Analytics: A clustering and triangulation study of digital library user data. Educational Technology & Society, 15(3):103--115, 2012.

Cited By

Schneegass CWilson MShaban JNiess JChiossi FMitrevska TWoźniak P(2024)Broadening the mind: how emerging neurotechnology is reshaping HCI and interactive system designi-com10.1515/icom-2024-000723:2(165-177)Online publication date: 23-May-2024
https://doi.org/10.1515/icom-2024-0007
Bayrak Karsli MCilligol Karabey SKaba EGuler MAydemir Arslan MKursun E(2024)Research Trends in Teaching Analytics: Bibliometric Mapping and Content AnalysisTechnology, Knowledge and Learning10.1007/s10758-024-09773-yOnline publication date: 8-Aug-2024
https://doi.org/10.1007/s10758-024-09773-y
Tulodziecki G(2024)Digitalisierung als Forschungsgegenstand der SchulpädagogikDigitale Medien und die Produktion von Wissenschaft10.1007/978-3-658-42542-5_5(63-83)Online publication date: 2-Mar-2024
https://doi.org/10.1007/978-3-658-42542-5_5
Show More Cited By

Index Terms

Teaching analytics: towards automatic extraction of orchestration graphs using wearable sensors
1. Applied computing
  1. Education
    1. Collaborative learning
  2. Enterprise computing

Recommendations

Multimodal Teaching and Learning Analytics for Classroom and Online Educational Settings
ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

Automatic analysis of teacher student interactions is an interesting research problem in social computing. Such interactions happen in both online and class room settings. While teaching effectiveness is the goal in both settings, the mechanism to ...
Towards teaching analytics: communication and negotiation tool (CoNeTo)
NordiCHI '12: Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design

In this paper, we present a short description of CoNeTo, a web-based collaborative application that facilitates the communication and negotiation of students' learning activities and inferences about their knowledge states, skill levels, and aptitudes ...
Towards visual analytics for teachers' dynamic diagnostic pedagogical decision-making
LAK '11: Proceedings of the 1st International Conference on Learning Analytics and Knowledge

The focus of this paper is to delineate and discuss design considerations for supporting teachers' dynamic diagnostic decision-making in classrooms of the 21st century. Based on the Next Generation Teaching Education and Learning for Life (NEXT-TELL) ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK '16: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge

April 2016

567 pages

ISBN:9781450341905

DOI:10.1145/2883851

General Chairs:
Dragan Gašević
University of Edinburgh, United Kingdom
,
Grace Lynch
Society for Learning Analytics Research
,
Program Chairs:
Shane Dawson
University of South Australia
,
Hendrik Drachsler
University of the Netherlands
,
Carolyn Penstein Rosé
Carnegie Mellon University

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Seventh Framework Programme

Conference

LAK '16

LAK '16: 6th International Conference on Learning Analytics and Knowledge

April 25 - 29, 2016

Edinburgh, United Kingdom

Acceptance Rates

LAK '16 Paper Acceptance Rate 36 of 116 submissions, 31%;

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

72
Total Citations
View Citations
1,111
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)14

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schneegass CWilson MShaban JNiess JChiossi FMitrevska TWoźniak P(2024)Broadening the mind: how emerging neurotechnology is reshaping HCI and interactive system designi-com10.1515/icom-2024-000723:2(165-177)Online publication date: 23-May-2024
https://doi.org/10.1515/icom-2024-0007
Bayrak Karsli MCilligol Karabey SKaba EGuler MAydemir Arslan MKursun E(2024)Research Trends in Teaching Analytics: Bibliometric Mapping and Content AnalysisTechnology, Knowledge and Learning10.1007/s10758-024-09773-yOnline publication date: 8-Aug-2024
https://doi.org/10.1007/s10758-024-09773-y
Tulodziecki G(2024)Digitalisierung als Forschungsgegenstand der SchulpädagogikDigitale Medien und die Produktion von Wissenschaft10.1007/978-3-658-42542-5_5(63-83)Online publication date: 2-Mar-2024
https://doi.org/10.1007/978-3-658-42542-5_5
Michel CPierrot LOru FVigneau O(2024)The Challenge of Modeling the Complexity of Use for the Measurement of Digital Maturity in EducationTechnology Enhanced Learning for Inclusive and Equitable Quality Education10.1007/978-3-031-72315-5_19(268-283)Online publication date: 13-Sep-2024
https://doi.org/10.1007/978-3-031-72315-5_19
Alhilo TAl-Sakaa A(2024)Eye Tracking Review: Importance, Tools, and ApplicationsEmerging Trends and Applications in Artificial Intelligence10.1007/978-3-031-56728-5_32(383-394)Online publication date: 30-Apr-2024
https://doi.org/10.1007/978-3-031-56728-5_32
Dhamija ADhamija DHöl D(2023)Integrated Model for Asynchronous Learning and Predictive Analytics for Enhanced Learner ExperienceEmergent Practices of Learning Analytics in K-12 Classrooms10.4018/979-8-3693-0066-4.ch003(26-56)Online publication date: 29-Dec-2023
https://doi.org/10.4018/979-8-3693-0066-4.ch003
Kędras MSobecki J(2023)What Is Hidden in Clear Sight and How to Find It—A Survey of the Integration of Artificial Intelligence and Eye TrackingInformation10.3390/info1411062414:11(624)Online publication date: 20-Nov-2023
https://doi.org/10.3390/info14110624
Chu SGarcia BRani N(2023)Research on wearable technologies for learning: a systematic reviewFrontiers in Education10.3389/feduc.2023.12703898Online publication date: 9-Nov-2023
https://doi.org/10.3389/feduc.2023.1270389
Zhu CPeng CWu Y(2023)A study on analyzing the interactive behavior of elementary school mathematics classroom teaching in a smart classroom environmentProceedings of the 2023 International Conference on Information Education and Artificial Intelligence10.1145/3660043.3660068(138-143)Online publication date: 22-Dec-2023
https://dl.acm.org/doi/10.1145/3660043.3660068
Emerson AMin WRowe JAzevedo RLester JHilliger IKhosravi HRienties BDawson S(2023)Multimodal Predictive Student Modeling with Multi-Task Transfer LearningLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576101(333-344)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3576050.3576101
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents