Abstract
Recent years have seen a growing recognition of the central role of affect and motivation in learning. In particular, nonverbal behaviors such as posture and gesture provide key channels signaling affective and motivational states. Developing a clear understanding of these mechanisms will inform the development of personalized learning environments that promote successful affective and motivational outcomes. This paper investigates posture and gesture in computer-mediated tutorial dialogue using automated techniques to track posture and hand-to-face gestures. Annotated dialogue transcripts were analyzed to identify the relationships between student posture, student gesture, and tutor and student dialogue. The results indicate that posture and hand-to-face gestures are significantly associated with particular tutorial dialogue moves. Additionally, two-hands-to-face gestures occurred significantly more frequently among students with low self-efficacy. The results shed light on the cognitive-affective mechanisms that underlie these nonverbal behaviors. Collectively, the findings provide insight into the interdependencies among tutorial dialogue, posture, and gesture, revealing a new avenue for automated tracking of embodied affect during learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 39–58 (2009)
Kleinsmith, A., Bianchi-Berthouze, N.: Affective Body Expression Perception and Recognition: A Survey. IEEE Transactions on Affective Computing (2012)
Calvo, R.A., D’Mello, S.K.: Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. IEEE Transactions on Affective Computing 1, 18–37 (2010)
El Kaliouby, R., Robinson, P.: The Emotional Hearing Aid: an Assistive Tool for Children with Asperger Syndrome. Universal Access in the Information Society 4, 121–134 (2005)
Mahmoud, M., Robinson, P.: Interpreting Hand-Over-Face Gestures. In: D’Mello, S., Graesser, A., Schuller, B., Martin, J.-C. (eds.) ACII 2011, Part II. LNCS, vol. 6975, pp. 248–255. Springer, Heidelberg (2011)
Rodrigo, M.M.T., Baker, R.S.J.d.: Comparing Learners’ Affect while using an Intelligent Tutor and an Educational Game. Research and Practice in Technology Enhanced Learning 6, 43–66 (2011)
Woolf, B.P., Burleson, W., Arroyo, I., Dragon, T., Cooper, D.G., Picard, R.W.: Affect-Aware Tutors: Recognising and Responding to Student Affect. International Journal of Learning Technology 4, 129–164 (2009)
Forbes-Riley, K., Litman, D.: Benefits and Challenges of Real-Time Uncertainty Detection and Adaptation in a Spoken Dialogue Computer Tutor. Speech Communication 53, 1115–1136 (2011)
D’Mello, S., Dale, R., Graesser, A.: Disequilibrium in the Mind, Disharmony in the Body. Cognition & Emotion 26, 362–374 (2012)
Baltrusaitis, T., McDuff, D., Banda, N., Mahmoud, M., El Kaliouby, R., Robinson, P., Picard, R.: Real-Time Inference of Mental States from Facial Expressions and Upper Body Gestures. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, pp. 909–914 (2011)
Brawner, K.W., Goldberg, B.S.: Real-Time Monitoring of ECG and GSR Signals during Computer-Based Training. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 72–77. Springer, Heidelberg (2012)
Baker, R.S.J.d., Gowda, S.M., Wixon, M., Kalka, J., Wagner, A.Z., Salvi, A., Aleven, V., Kusbit, G.W., Ocumpaugh, J., Rossi, L.: Towards Sensor-Free Affect Detection in Cognitive Tutor Algebra. In: Proceedings of the 5th International Conference on Educational Data Mining, pp. 126–133 (2012)
Kapoor, A., Burleson, W., Picard, R.W.: Automatic Prediction of Frustration. International Journal of Human-Computer Studies 65, 724–736 (2007)
Sanghvi, J., Castellano, G., Leite, I., Pereira, A., McOwan, P.W., Paiva, A.: Automatic Analysis of Affective Postures and Body Motion to Detect Engagement with a Game Companion. In: Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, pp. 305–311 (2011)
Grafsgaard, J.F., Fulton, R.M., Boyer, K.E., Wiebe, E.N., Lester, J.C.: Multimodal Analysis of the Implicit Affective Channel in Computer-Mediated Textual Communication. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction, pp. 145–152 (2012)
McNeill, D.: Gesture & Thought. The University of Chicago Press, Chicago (2005)
Grafsgaard, J.F., Boyer, K.E., Phillips, R., Lester, J.C.: Modeling Confusion: Facial Expression, Task, and Discourse in Task-Oriented Tutorial Dialogue. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) AIED 2011. LNCS, vol. 6738, pp. 98–105. Springer, Heidelberg (2011)
Bandura, A.: Guide for Constructing Self-Efficacy Scales. In: Pajares, F., Urdan, T. (eds.) Self-Efficacy Beliefs of Adolescents, pp. 307–337. Information Age Publishing, Greenwich (2006)
Boyer, K.E., Phillips, R., Ingram, A., Ha, E.Y., Wallis, M., Vouk, M., Lester, J.: Characterizing the Effectiveness of Tutorial Dialogue with Hidden Markov Models. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010, Part I. LNCS, vol. 6094, pp. 55–64. Springer, Heidelberg (2010)
Ha, E.Y., Grafsgaard, J.F., Mitchell, C.M., Boyer, K.E., Lester, J.C.: Combining Verbal and Nonverbal Features to Overcome the “Information Gap” in Task-Oriented Dialogue. In: Proceedings of the Thirteenth Annual SIGDIAL Meeting on Discourse and Dialogue, pp. 247–256 (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Grafsgaard, J.F., Wiggins, J.B., Boyer, K.E., Wiebe, E.N., Lester, J.C. (2013). Embodied Affect in Tutorial Dialogue: Student Gesture and Posture. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds) Artificial Intelligence in Education. AIED 2013. Lecture Notes in Computer Science(), vol 7926. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39112-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-642-39112-5_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39111-8
Online ISBN: 978-3-642-39112-5
eBook Packages: Computer ScienceComputer Science (R0)