research-article

Do speech features for detecting cognitive load depend on specific languages?

Authors:

Ningjiu TangAuthors Info & Claims

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

Pages 76 - 83

https://doi.org/10.1145/2993148.2993149

Published: 31 October 2016 Publication History

Abstract

Speech-based cognitive load modeling recently proposed in English have enabled objective, quantitative and unobtrusive evaluation of cognitive load without extra equipment. However, no evidence indicates that these techniques could be applied to speech data in other languages without modification. In this study, a modified Stroop Test and a Reading Span Task were conducted to collect speech data in English and Chinese respectively, from which twenty non-linguistic features were extracted to investigate whether they were language dependent. Some discriminating speech features were observed language dependent, which serves as an evidence that there is a necessity to adapt speech-based cognitive load detection techniques to diverse language contexts for a higher performance.

References

[1]

Berthold, A. and Jameson, A. 1999. Interpreting symptoms of cognitive load in speech input. In UM99 User Modeling. J. Kay, ed. Springer Vienna. 235–244.

Digital Library

[2]

Boril, H., Omid Sadjadi, S., Kleinschmidt, T. and Hansen, J.H. 2010. Analysis and detection of cognitive load and frustration in drivers’ speech. In Proceedings of INTERSPEECH 2010 (2010), 502–505.

[3]

Campione, E. and Véronis, J. 2002. A large-scale multilingual study of silent pause duration. In Proceedings of the Speech Prosody 2002 Conference (Aix-en-Provence, 2002), 199–202.

[4]

Chandler, P. and Sweller, J. 1991. Cognitive load theory and the format of instruction. Cognition and Instruction. 8, 4 (Dec. 1991), 293–332.

[5]

Gorovoy, K., Tung, J. and Poupart, P. 2010. Automatic speech feature extraction for cognitive load classification. In Conference of the Canadian Medical and Biological Engineering Society (CMBEC) (2010).

[6]

Huttunen, K.H., Keränen, H.I., Pääkkönen, R.J., Päivikki Eskelinen-Rönkä, R. and Leino, T.K. 2011. Effect of cognitive load on articulation rate and formant frequencies during simulator flights. The Journal of the Acoustical Society of America. 129, 3 (Mar. 2011), 1580–1593.

[7]

Ikehara, C.S. and Crosby, M.E. 2005. Assessing cognitive load with physiological sensors. In Proceedings of the 38th Annual Hawaii International Conference on System Sciences (Jan. 2005), 295a–295a.

Digital Library

[8]

Jameson, A., Kiefer, J., Müller, C., Großmann-Hutter, B., Wittig, F. and Rummer, R. 2010. Assessment of a user’s time pressure and cognitive load on the basis of features of speech. In Resource-Adaptive Cognitive Processes. M.W. Crocker and J. Siekmann, eds. Springer Berlin Heidelberg. 171–204.

[9]

Khawaja, M.A., Chen, F. and Marcus, N. 2014. Measuring cognitive load using linguistic features: implications for usability evaluation and adaptive interaction design. International Journal of Human-Computer Interaction. 30, 5 (May 2014), 343–368.

[10]

Khawaja, M.A., Chen, F. and Marcus, N. 2010. Using language complexity to measure cognitive load for adaptive interaction design. In Proceedings of the 15th International Conference on Intelligent User Interfaces (New York, NY, USA, 2010), 333–336.

Digital Library

[11]

Khawaja, M.A., Chen, F., Owen, C. and Hickey, G. 2009. Cognitive load measurement from user’s linguistic speech features for adaptive interaction design. In Human-Computer Interaction – INTERACT 2009. T. Gross, J. Gulliksen, P. Kotzé, L. Oestreicher, P. Palanque, R.O. Prates, and M. Winckler, eds. Springer Berlin Heidelberg. 485–489.

Digital Library

[12]

Khawaja, M.A., Ruiz, N. and Chen, F. 2007. Potential speech features for cognitive load measurement. In Proceedings of the 19th Australasian Conference on Computer-Human Interaction: Entertaining User Interfaces (New York, NY, USA, 2007), 57–60.

Digital Library

[13]

Le, P.N., Ambikairajah, E., Epps, J., Sethu, V. and Choi, E.H. 2011. Investigation of spectral centroid features for cognitive load classification. Speech Communication. 53, 4 (2011), 540–551.

Digital Library

[14]

Lin, T., Xie, T., Chen, Y. and Tang, N. 2013. Automatic cognitive load evaluation using writing features: an exploratory study. International Journal of Industrial Ergonomics. 43, 3 (May 2013), 210–217.

[15]

Müller, C., Großmann-Hutter, B., Jameson, A., Rummer, R. and Wittig, F. 2001. Recognizing time pressure and cognitive load on the basis of speech: an experimental study. In User Modeling 2001. M. Bauer, P.J. Gmytrasiewicz, and J. Vassileva, eds. Springer Berlin Heidelberg. 24–33.

Digital Library

[16]

Paas, F., Tuovinen, J.E., Tabbers, H. and Gerven, P.W.M.V. 2003. Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist. 38, 1 (Mar. 2003), 63–71.

[17]

Pedersen, C., Togelius, J. and Yannakakis, G.N. 2010. Modeling player experience for content creation. IEEE Transactions on Computational Intelligence and AI in Games. 2, 1 (Mar. 2010), 54–67.

[18]

Rothkrantz, L.J.M., Wiggers, P., Van Wees, J.-W.A. and Van Vark, R.J. 2004. Voice stress analysis. In Text, Speech and Dialogue. P. Sojka, I. Kopeček, and K. Pala, eds. Springer Berlin Heidelberg. 449–456.

[19]

Ruiz, N., Feng, Q.Q., Taib, R., Handke, T. and Chen, F. 2010. Cognitive skills learning: pen input patterns in computer-based athlete training. In International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction (New York, NY, USA, 2010), 41:1–41:4.

Digital Library

[20]

Shahnaz, C., Zhu, W. p and Ahmad, M.O. 2006. A new technique for the estimation of jitter and shimmer of voiced speech signal. In Canadian Conference on Electrical and Computer Engineering (May 2006), 2112–2115.

[21]

Slyh, R.E., Nelson, W.T. and Hansen, E.G. 1999. Analysis of mrate, shimmer, jitter, and F 0 contour features across stress and speaking style in the SUSAS database. In Proceedings of the 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing (Mar. 1999), 2091–2094.

Digital Library

[22]

Unsworth, N., Heitz, R.P., Schrock, J.C. and Engle, R.W. 2005. An automated version of the operation span task. Behavior Research Methods. 37, 3 (Aug. 2005), 498–505.

[23]

Vizer, L.M., Zhou, L. and Sears, A. 2009. Automated stress detection using keystroke and linguistic features: an exploratory study. International Journal of Human-Computer Studies. 67, 10 (Oct. 2009), 870–886.

Digital Library

[24]

Yap, T.F. 2012. Speech production under cognitive load: effects and classification. Doctoral Thesis. University of New South Wales.

[25]

Yap, T.F., Epps, J., Ambikairajah, E. and Choi, E.H.C. 2011. Formant frequencies under cognitive load: effects and classification. EURASIP J. Adv. Signal Process. 2011, (Jan. 2011), 1:1–1:11.

Digital Library

[26]

Yap, T.F., Epps, J., Choi, E.H.C. and Ambikairajah, E. 2010. Glottal features for speech-based cognitive load classification. In IEEE International Conference on Acoustics, Speech and Signal Processing (Mar. 2010), 5234– 5237.

[27]

Yin, B. and Chen, F. 2007. Towards automatic cognitive load measurement from speech analysis. In Human-Computer Interaction. Interaction Design and Usability. J.A. Jacko, ed. Springer Berlin Heidelberg. 1011–1020.

Digital Library

[28]

Yin, B., Chen, F., Ruiz, N. and Ambikairajah, E. 2008. Speech-based cognitive load monitoring system. In IEEE International Conference on Acoustics, Speech and Signal Processing (Mar. 2008), 2041–2044.

[29]

Yin, B., Ruiz, N., Chen, F. and Ambikairajah, E. 2008. Investigating speech features and automatic measurement of cognitive load. In IEEE 10th Workshop on Multimedia Signal Processing (Oct. 2008), 988–993.

[30]

Yin, B., Ruiz, N., Chen, F. and Khawaja, M.A. 2007. Automatic cognitive load detection from speech features. In Proceedings of the 19th Australasian Conference on Computer-Human Interaction: Entertaining User Interfaces (New York, NY, USA, 2007), 249–255.

Digital Library

[31]

Zahorian, S.A. and Hu, H. 2008. A spectral/temporal method for robust fundamental frequency tracking. The Journal of the Acoustical Society of America. 123, 6 (Jun. 2008), 4559– 4571.

Cited By

Schneegass CWilson MShaban JNiess JChiossi FMitrevska TWoźniak P(2024)Broadening the mind: how emerging neurotechnology is reshaping HCI and interactive system designi-com10.1515/icom-2024-000723:2(165-177)Online publication date: 23-May-2024
https://doi.org/10.1515/icom-2024-0007
Alt FHassib MDistler V(2023)Human-centered Behavioral and Physiological SecurityProceedings of the 2023 New Security Paradigms Workshop10.1145/3633500.3633504(48-61)Online publication date: 18-Sep-2023
https://dl.acm.org/doi/10.1145/3633500.3633504
Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Show More Cited By

Index Terms

Do speech features for detecting cognitive load depend on specific languages?
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI

Recommendations

Think before you talk: an empirical study of relationship between speech pauses and cognitive load
OZCHI '08: Proceedings of the 20th Australasian Conference on Computer-Human Interaction: Designing for Habitus and Habitat

Measuring a user's level of cognitive load while they are interacting with the system could offer another dimension to the development of adaptable user interfaces. High levels of cognitive load affect performance and efficiency. However, current ...
Automatic cognitive load detection from speech features
OZCHI '07: Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces

Cognitive load variations have been found to impact multimodal behaviour, in particular, features of spoken input. In this paper, we present a design and implementation of a user study aimed at soliciting natural speech at three different levels of ...
Potential speech features for cognitive load measurement
OZCHI '07: Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces

Intelligent user interfaces with an awareness of a user's experienced level of cognitive load have the potential to change the way output strategies are implemented and executed. However, current methods of measuring cognitive load are intrusive and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

October 2016

605 pages

ISBN:9781450345569

DOI:10.1145/2993148

General Chairs:
Yukiko I. Nakano
Seikei University, Japan
,
Elisabeth André
Augsburg University, Germany
,
Toyoaki Nishida
Kyoto University, Japan
,
Program Chairs:
Louis-Philippe Morency
Carnegie Mellon University, USA
,
Carlos Busso
University of Texas at Dallas, USA
,
Catherine Pelachaud
ISIR, France / University of Paris6, France

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Science and Technology Supporting Program, Sichuan Province

Conference

ICMI '16

Sponsor:

SIGCHI

ICMI '16: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 12 - 16, 2016

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
192
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schneegass CWilson MShaban JNiess JChiossi FMitrevska TWoźniak P(2024)Broadening the mind: how emerging neurotechnology is reshaping HCI and interactive system designi-com10.1515/icom-2024-000723:2(165-177)Online publication date: 23-May-2024
https://doi.org/10.1515/icom-2024-0007
Alt FHassib MDistler V(2023)Human-centered Behavioral and Physiological SecurityProceedings of the 2023 New Security Paradigms Workshop10.1145/3633500.3633504(48-61)Online publication date: 18-Sep-2023
https://dl.acm.org/doi/10.1145/3633500.3633504
Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Pyfrom MLister JAnand S(2023)Influence of Cognitive Load on Voice Production: A Scoping ReviewJournal of Voice10.1016/j.jvoice.2023.08.024Online publication date: Sep-2023
https://doi.org/10.1016/j.jvoice.2023.08.024
Hecker PKappattanavar ASchmitt MMoontaha SWagner JEyben FSchuller BArnrich B(2022)Quantifying Cognitive Load from Voice using Transformer-Based Models and a Cross-Dataset Evaluation2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA)10.1109/ICMLA55696.2022.00055(337-344)Online publication date: Dec-2022
https://doi.org/10.1109/ICMLA55696.2022.00055

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents