research-article

Written and multimodal representations as predictors of expertise and problem-solving success in mathematics

Authors:

Adrienne CohenAuthors Info & Claims

ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

Pages 599 - 606

https://doi.org/10.1145/2522848.2533793

Published: 09 December 2013 Publication History

Abstract

One aim of multimodal learning analytics is to analyze rich natural communication modalities to identify domain expertise and learning rapidly and reliably. In this research, written and multimodal representations are analyzed from the Math Data Corpus, which involves multimodal data (digital pen, speech, images) on collaborating students as they solve math problems. Findings reveal that in 96-97% of cases the correctness of a group's solution was predictable in advance based on students' written work content. In addition, a linear regression revealed that 65% of the variance in individual students' domain expertise rankings could be accounted for based on their written work content. A multimodal content analysis based on both written and spoken input correctly predicted the dominant domain expert in a group 100% of the time, exceeding unimodal prediction rates. Further analysis revealed a reversal between experts and non-experts in the percentage of time that a match versus mismatch was present between their oral and written answer contributions, with non-experts demonstrating higher mismatches. Implications are discussed for developing reliable multimodal learning analytics systems that incorporate digital pen input to automatically track consolidation of domain expertise.

References

[1]

Alibali, M. & Goldin-Meadow, S. (1993) Gesture-speech mismatch and mechanisms of learning: What the hand reveals about a child's state of mind, Cognitive Psychology, 25, 468--523.

[2]

Arthur, A., Lunsford, R., Wesson, M. & Oviatt, S.L. (2006) Prototyping novel collaborative multimodal systems: Simulation, data collection and analysis tools for the next decade, Eighth International Conference on Multimodal Interfaces (ICMI'06), ACM: New York, 209--226.

Digital Library

[3]

Baker, R. & Yacef, K. (2009) The state of educational data mining in 2009: A review and future visions, Journal of Educational Data Mining (JEDM), 1:3--17.

[4]

Cheng, P. C. H., & Rojas-Anaya, H. (2007). Measuring mathematical formula writing competence: An application of graphical protocol analysis, in D. S. McNamara & J. G. Trafton (Eds.), Proceedings of the Twenty Ninth Annual Conference of the Cognitive Science Society, Austin, TX: Cognitive Science Society, 869--874.

[5]

Dral, J., Heylen, D. & op den Akker, R. (2008) Detecting uncertainty in spoken dialogues: An explorative research to the automatic detection of a speakers' uncertainty by using prosodic markers, Sentiment analysis: emotion, metaphor, ontology and terminology, Marrakech, Morocco, 72--78.

[6]

Liscombe, J., Hirschberg, J. & Vendetti, J. (2005) Detecting certainness in spoken tutorial dialogs, Proc. of Interspeech 2005-Europeech, Lisbon, Portugal.

[7]

Lunsford, R., Oviatt, S. L. & Arthur, A. (2006) Toward open-microphone engagement for multiparty interactions, Eighth International Conference on Multimodal Interfaces (ICMI'06), ACM: New York, 273--280.

Digital Library

[8]

Oviatt, S.L. (2012) Multimodal interfaces, The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications (revised 3rd edition), (ed. by J. Jacko), Lawrence Erlbaum Assoc: Mahwah, New Jersey, chap. 18, 405--430.

[9]

Oviatt, S.L. (2013a) Problem solving, domain expertise and learning: Ground-truth performance results for math data corpus, Second Intl. Grand Challenge Workshop on Multimodal Learning Analytics, Sydney Australia, December 2013.

Digital Library

[10]

Oviatt, S.L. (2013b) The Design of Future Educational Interfaces, Routledge Press.

[11]

Oviatt, S. L., Arthur, A., Brock, Y. & Cohen, J. (2007) Expressive pen-based interfaces for math education, in C. Chinn, G. Erkens & S. Puntambekar (Eds.) Proceedings of Computer Supported Collaborative Learning, ISLS, 8(2), 569--578.

Digital Library

[12]

Oviatt, S., Cohen, A. & Weibel, N. (2013) Multimodal learning analytics: Description of math data corpus for ICMI grand challenge workshop, Second Intl. Workshop on Multimodal Learning Analytics, Sydney Australia (for version with full appendices, see: http://mla.ucsd.edu/data/MMLA_Math_Data_Corpus.pdf

Digital Library

[13]

Purandare, A. & Litman, D. (2008) Content-learning correlations in spoken tutoring dialogs at word, turn, and discourse levels, Proc. of the 21st Intl. FLAIRS Conference, Coconut Grove, FL.

[14]

Scherer, S., Weibel, N., Oviatt, S. & Morency, L.P. Multimodal prediction of expertise and leadership in learning groups, Proc. of the First International Workshop on Multimodal Learning Analytics, 2012, ACM: N.Y.

Digital Library

[15]

Wigboldus, D., Sherman, J., Franzese, H. & Knippenberg, A. (2004) Capacity and comprehension: Spontaneous stereotyping under cognitive load, Social Cognition, 22 (3), 292--309.

[16]

Worsley, M. & Blikstein, P. (2010) Toward the development of learning analytics: Student speech as an automatic and natural form of assessment, AERA Conference.

Cited By

Alwahaby HCukurova MPapamitsiou ZGiannakos M(2022)The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic Literature ReviewThe Multimodal Learning Analytics Handbook10.1007/978-3-031-08076-0_12(289-325)Online publication date: 9-Oct-2022
https://doi.org/10.1007/978-3-031-08076-0_12
Song MQian KChen BOkabayashi KParada-cabaleiro EYang ZLiu STogami KHidaka IWang YSchuller BYamamoto Y(2021)Predicting Group Work Performance from Physical Handwriting Features in a Smart English ClassroomProceedings of the 2021 5th International Conference on Digital Signal Processing10.1145/3458380.3458404(140-145)Online publication date: 26-Feb-2021
https://dl.acm.org/doi/10.1145/3458380.3458404
Stewart AKeirn ZD’Mello S(2021)Multimodal modeling of collaborative problem-solving facets in triadsUser Modeling and User-Adapted Interaction10.1007/s11257-021-09290-yOnline publication date: 2-Feb-2021
https://doi.org/10.1007/s11257-021-09290-y
Show More Cited By

Index Terms

Written and multimodal representations as predictors of expertise and problem-solving success in mathematics
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Spoken Interruptions Signal Productive Problem Solving and Domain Expertise in Mathematics
ICMI '15: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction

Prevailing social norms prohibit interrupting another person when they are speaking. In this research, simultaneous speech was investigated in groups of students as they jointly solved math problems and peer tutored one another. Analyses were based on ...
Problem solving, domain expertise and learning: ground-truth performance results for math data corpus
ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

Problem solving, domain expertise, and learning are analyzed for the Math Data Corpus, which involves multimodal data on collaborating student groups as they solve math problems together across sessions. Compared with non-expert students, domain experts ...
Expertise estimation based on simple multimodal features
ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

Multimodal Learning Analytics is a field that studies how to process learning data from dissimilar sources in order to automatically find useful information to give feedback to the learning process. This work processes video, audio and pen strokes ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

December 2013

630 pages

ISBN:9781450321297

DOI:10.1145/2522848

General Chairs:
Julien Epps
The University of New South Wales, Australia
,
Fang Chen
National ICT Australia, Australia
,
Sharon Oviatt
Incaa Designs, USA
,
Kenji Mase
Nagoya University, Japan
,
Program Chairs:
Andrew Sears
Rochester Institute of Technology, USA
,
Kristiina Jokinen
University of Helsinki, Finland
,
Björn Schuller
Technische Universität München, Germany

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 December 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '13

Sponsor:

SIGCHI

ICMI '13: 2013 International Conference on Multimodal Interaction

December 9 - 13, 2013

Sydney, Australia

Acceptance Rates

ICMI '13 Paper Acceptance Rate 49 of 133 submissions, 37%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
276
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)1

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alwahaby HCukurova MPapamitsiou ZGiannakos M(2022)The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic Literature ReviewThe Multimodal Learning Analytics Handbook10.1007/978-3-031-08076-0_12(289-325)Online publication date: 9-Oct-2022
https://doi.org/10.1007/978-3-031-08076-0_12
Song MQian KChen BOkabayashi KParada-cabaleiro EYang ZLiu STogami KHidaka IWang YSchuller BYamamoto Y(2021)Predicting Group Work Performance from Physical Handwriting Features in a Smart English ClassroomProceedings of the 2021 5th International Conference on Digital Signal Processing10.1145/3458380.3458404(140-145)Online publication date: 26-Feb-2021
https://dl.acm.org/doi/10.1145/3458380.3458404
Stewart AKeirn ZD’Mello S(2021)Multimodal modeling of collaborative problem-solving facets in triadsUser Modeling and User-Adapted Interaction10.1007/s11257-021-09290-yOnline publication date: 2-Feb-2021
https://doi.org/10.1007/s11257-021-09290-y
Kasparova ACeliktutan OCukurova MTruong KHeylen DCzerwinski MBerthouze NChetouani MNakano M(2020)Inferring Student Engagement in Collaborative Problem Solving from Visual CuesCompanion Publication of the 2020 International Conference on Multimodal Interaction10.1145/3395035.3425961(177-181)Online publication date: 25-Oct-2020
https://dl.acm.org/doi/10.1145/3395035.3425961
Cukurova MZhou QSpikol DLandolfi LRensing CDrachsler HKovanović VPinkwart NScheffel MVerbert K(2020)Modelling collaborative problem-solving competence with transparent learning analyticsProceedings of the Tenth International Conference on Learning Analytics & Knowledge10.1145/3375462.3375484(270-275)Online publication date: 23-Mar-2020
https://dl.acm.org/doi/10.1145/3375462.3375484
Roque FCechinel CWeber TLemos RVillarroel RMiranda DMunoz R(2019)Using Depth Cameras to Detect Patterns in Oral Presentations: A Case Study Comparing Two Generations of Computer Engineering StudentsSensors10.3390/s1916349319:16(3493)Online publication date: 9-Aug-2019
https://doi.org/10.3390/s19163493
Chua YDauwels JTan SHsiao SCunningham JMcCarthy KLynch GBrooks CFerguson RHoppe U(2019)Technologies for automated analysis of co-located, real-life, physical learning spacesProceedings of the 9th International Conference on Learning Analytics & Knowledge10.1145/3303772.3303811(11-20)Online publication date: 4-Mar-2019
https://dl.acm.org/doi/10.1145/3303772.3303811
Echeverria VMartinez-Maldonado RBuckingham Shum SBrewster SFitzpatrick GCox AKostakos V(2019)Towards Collaboration TranslucenceProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300269(1-16)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300269
Oviatt SD'Mello SGeorgiou PScherer SProvost ESoleymani MWorsley M(2018)Ten Opportunities and Challenges for Advancing Student-Centered Multimodal Learning AnalyticsProceedings of the 20th ACM International Conference on Multimodal Interaction10.1145/3242969.3243010(87-94)Online publication date: 2-Oct-2018
https://dl.acm.org/doi/10.1145/3242969.3243010
Oviatt SHang KZhou JYu KChen F(2018)Dynamic Handwriting Signal Features Predict Domain ExpertiseACM Transactions on Interactive Intelligent Systems10.1145/32133098:3(1-21)Online publication date: 24-Jul-2018
https://dl.acm.org/doi/10.1145/3213309
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents