Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

A module-based framework to emotion recognition by speech: a case study in clinical simulation

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Current research in human–computer interaction reveals the importance of taking into account emotional aspects in the interaction with computer systems. The main objective of promoting this emotional recognition is to contribute to the enhanced coherence, consistency and credibility of the computer reactions and responses during human interaction through the human–computer interface. In that context, computer systems can be explored that can identify and classify the user’s emotional condition. In this study, computer techniques are used to identify and classify emotional aspects based on the users’ discourse, aiming to assess the emotional behavior in the daily reality of critical care professionals. Hence, in the study of computer techniques and psychological theories, both areas can be related to classify emotion through a framework for the acquisition and classification of users’ discourse. The system developed was applied as an additional evaluation method in the development of simulated scenarios in the context of Clinical Simulation and demonstrated its efficiency as a new approach in health assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • AbdulGhaffar A, Mostafa SM, Alsaleh A, Sheltami T, Shakshuki EM (2019) Internet of things based multiple disease monitoring and health improvement system. J Ambient Intell Hum Comput. https://doi.org/10.1007/s12652-019-01204-6

    Article  Google Scholar 

  • Ahmed W, Van der Werf G, Kuyper H, Minnaert A (2013) Emotions, self-regulated learning, and achievement in mathematics: a growth curve analysis. J Educ Psychol 105(1):150

    Article  Google Scholar 

  • Almeida RGdS, Mazzo A, Martins JCA, Baptista RCN, Girão FB, Mendes IAC (2015) Validation to portuguese of the scale of student satisfaction and self-confidence in learning. Revista latino-americana de enfermagem 23(6):1007–1013

    Article  PubMed  PubMed Central  Google Scholar 

  • Arigbabu OA, Mahmood S, Ahmad SMS, Arigbabu AA (2016) Smile detection using hybrid face representation. J Ambient Intell Hum Comput 7(3):415–426

    Article  Google Scholar 

  • Bailenson JN, Pontikakis ED, Mauss IB, Gross JJ, Jabon ME, Hutcherson CA, Nass C, John O (2008) Real-time classification of evoked emotions using facial feature tracking and physiological responses. Int J Hum Comput Stud 66(5):303–317

    Article  Google Scholar 

  • Baptista RCN, Martins JCA, Pereira MFCR, Mazzo A (2014) Students’ satisfaction with simulated clinical experiences: validation of an assessment scale. Revista latino-americana de enfermagem 22(5):709–715

    Article  PubMed  PubMed Central  Google Scholar 

  • Becker ES, Keller MM, Goetz T, Frenzel AC, Taxer JL (2015) Antecedents of teachers’ emotions in the classroom: an intraindividual approach. Front Psychol 6:635

    Article  PubMed  PubMed Central  Google Scholar 

  • Brazeal KR, Brown TL, Couch BA (2016) Characterizing student perceptions of and buy-in toward common formative assessment techniques. CBE Life Sci Educ 15(4):ar73

    Article  PubMed  PubMed Central  Google Scholar 

  • Carneiro D, Pinheiro AP, Novais P (2017) Context acquisition in auditory emotional recognition studies. J Ambient Intell Hum Comput 8(2):191–203

    Article  Google Scholar 

  • Chang J, Scherer S (2017) Learning representations of emotional speech with deep convolutional generative adversarial networks. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 2746–2750. https://doi.org/10.1109/ICASSP.2017.7952656

  • Chouchani N, Abed M (2019) Enhance sentiment analysis on social networks with social influence analytics. J Ambient Intell Hum Comput. https://doi.org/10.1007/s12652-019-01234-0

    Article  Google Scholar 

  • Ekman P, Friesen WV (1981) The repertoire of nonverbal behavior: categories, origins, usage, and coding. Nonverbal Commun Interact Gesture 1:57–106

    Article  Google Scholar 

  • Ellsworth PC, Scherer KR (2003) Appraisal processes in emotion. Handb Affect Sci 572:V595

    Google Scholar 

  • Eyben F, Wöllmer M, Schuller B (2009) Openear—introducing the munich open-source emotion and affect recognition toolkit. In: Affective computing and intelligent interaction and workshops, 2009. ACII 2009. 3rd international conference on, IEEE, pp 1–6

  • Fast LA, Funder DC (2008) Personality as manifest in word use: correlations with self-report, acquaintance report, and behavior. J Personal Soc Psychol 94(2):334

    Article  Google Scholar 

  • Fontaine JR, Poortinga YH, Setiadi B, Markam SS (2002) Cognitive structure of emotion terms in Indonesia and The Netherlands. Cognit. Emot. 16(1):61–86

    Article  Google Scholar 

  • Frijda NH (1986) The emotions: Studies in emotion and social interaction. Paris: Maison de Sciences de l’Homme

  • Fuentes C, Herskovic V, Rodríguez I, Gerea C, Marques M, Rossel PO (2017) A systematic literature review about technologies for self-reporting emotional information. J Ambient Intell Hum Comput 8(4):593–606

    Article  Google Scholar 

  • Gill R (2009) Breaking the silence: the hidden injuries of neo-liberal academia. Secr Silenc Res Process Fem Reflect 21:228–244

    Google Scholar 

  • Golbeck J, Robles C, Turner K (2011) Predicting personality with social media. In: CHI’11 extended abstracts on human factors in computing systems, ACM, pp 253–262

  • Goy H, Pichora-Fuller MK, Singh G, Russo FA (2018) Hearing aids benefit recognition of words in emotional speech but not emotion identification. Trends Hearing 22:2331216518801736. https://doi.org/10.1177/2331216518801736

    Article  Google Scholar 

  • Halstead J, Green P, Speziale H et al (2005) Core competencies of nurse educators with task statements. National League for Nursing Publications, New York

    Google Scholar 

  • Harati S, Crowell A, Mayberg H, Nemati S (2018) Depression severity classification from speech emotion. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, pp 5763–5766. https://doi.org/10.1109/EMBC.2018.8513610

  • Hirsh JB, Peterson JB (2009) Personality and language use in self-narratives. J Res Pers 43(3):524–527

    Article  Google Scholar 

  • Huang KY, Wu CH, Su MH, Kuo YT (2018) Detecting unipolar and bipolar depressive disorders from elicited speech responses using latent affective structure model. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2018.2803178

    Article  Google Scholar 

  • IBM (2017) Cloud infrastructure, storage, security&, more - IBM bluemix. https://www.ibm.com/cloud-computing/bluemix/node/4471. Accessed 13 Sep 2017

  • IBM (2017) Documentation, IBM Watson developer cloud. https://www.ibm.com/watson/developercloud/doc/index.html. Accessed 13 Sep 2017

  • IBM (2017) Language translator, IBM Watson developer cloud. https://www.ibm.com/watson/developercloud/doc/language-translator/index.html. Accessed 13 Sept 2017

  • IBM (2017) Speech to text, about speech to text, ibm watson developer cloud. https://www.ibm.com/watson/developercloud/doc/speech-to-text/index.html. Accessed 13 Sep 2017

  • IBM (2017) Tone analyzer, about tone analyzer, IBM Watson developer cloud. https://www.ibm.com/watson/developercloud/doc/tone-analyzer/index.html. Accessed 13 Sep 2017

  • Lee CH, Kim K, Seo YS, Chung CK (2007) The relations between personality and language use. J Gen Psychol 134(4):405–413

    Article  CAS  PubMed  Google Scholar 

  • Lichtenstein A, Oehme A, Kupschick S, Jürgensohn T (2008) Comparing two emotion models for deriving affective states from physiological data. In: Affect and emotion in human–computer interaction, Springer Berlin Heidelberg, Berlin, Heidelberg, pp 35–50. https://doi.org/10.1007/978-3-540-85099-1_4

  • Lugović S, Dunđer I, Horvat M (2016) Techniques and applications of emotion recognition in speech. In: 2016 39th international convention on information and communication technology, electronics and microelectronics (MIPRO). IEEE, pp 1278–1283. https://doi.org/10.1109/MIPRO.2016.7522336

  • Mano LY (2018) Emotional condition in the health smart homes environment: emotion recognition using ensemble of classifiers. In: 2018 Innovations in intelligent systems and applications (INISTA), IEEE, pp 1–8

  • Mano LY, Faiçal BS, Nakamura LH, Gomes PH, Libralon GL, Meneguete RI, Filho GP, Giancristofaro GT, Pessin G, Krishnamachari B, Ueyama J (2016) Exploiting iot technologies for enhancing health smart homes through patient identification and emotion recognition. Comput Commun 89–90:178–190. https://doi.org/10.1016/j.comcom.2016.03.010

    Article  Google Scholar 

  • Mano LY, Giancristofaro GT, Faiçal BS, Libralon GL, Pessin G, Gomes PH, Ueyama J (2015) Exploiting the use of ensemble classifiers to enhance the precision of user’s emotion classification. In: Proceedings of the 16th international conference on engineering applications of neural networks (INNS), ACM, p 5

  • Mano LY, Vasconcelos E, Ueyama J (2016) Identifying emotions in speech patterns: adopted approach and obtained results. IEEE Latin Am Trans 14(12):4775–4780

    Article  Google Scholar 

  • Mehl MR, Gosling SD, Pennebaker JW (2006) Personality in its natural habitat: manifestations and implicit folk theories of personality in daily life. J Pers Soc Psychol 90(5):862

    Article  PubMed  Google Scholar 

  • Nardelli M, Valenza G, Greco A, Lanata A, Scilingo EP (2015) Recognizing emotions induced by affective sounds through heart rate variability. IEEE Trans Affect Comput 6(4):385–394

    Article  Google Scholar 

  • Neto JRT, Filho GP, Mano LY, Ueyama J (2018) Verbo: voice emotion recognition database in Portuguese language. J Comput Sci 14(11):1420–1430. https://doi.org/10.3844/jcssp.2018.1420.1430

    Article  Google Scholar 

  • Oinas-Kukkonen H (2013) A foundation for the study of behavior change support systems. Pers Ubiquitous Comput 17(6):1223–1235

    Article  Google Scholar 

  • Parkinson B (1995) Ideas and realities of emotion. Psychology Press, London

    Google Scholar 

  • Patwardhan A, Knapp G (2016) Augmenting supervised emotion recognition with rule-based decision model. arXiv:1607.02660

  • Pennebaker JW, Francis ME, Booth RJ (2001) Linguistic inquiry and word count: LIWC 2001. Lawrence Erlbaum Associates, Mahwah, NJ

    Google Scholar 

  • Pennebaker JW, King LA (1999) Linguistic styles: language use as an individual difference. J Pers Soc Psychol 77(6):1296

    Article  CAS  PubMed  Google Scholar 

  • Peter C, Urban B (2012) Emotion in human–computer interaction. In: Expanding the frontiers of visual analytics and visualization, Springer London, pp 239–262, https://doi.org/10.1007/978-1-4471-2804-5_14

  • Picard RW (2003) What does it mean for a computer to “have” emotions? In: Emotions in humans and artifacts, MIT Press, chap 7

  • Scherer KR (2001) Appraisal considered as a process of multilevel sequential checking. Apprais Process Emot Theory Methods Res 92:120

    Google Scholar 

  • Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inf 44(4):695–729

    Article  Google Scholar 

  • Stemmler G (2003) Methodological considerations in the psychophysiological study of emotion. Handb Affect Sci 37:225–255

    Google Scholar 

  • Tausczik YR, Pennebaker JW (2010) The psychological meaning of words: Liwc and computerized text analysis methods. J lang Soc Psychol 29(1):24–54

    Article  Google Scholar 

  • Verma P, Sood SK, Kalra S (2018) Cloud-centric iot based student healthcare monitoring framework. J Ambient Intell Hum Comput 9(5):1293–1309. https://doi.org/10.1007/s12652-017-0520-6

    Article  Google Scholar 

  • Xia R, Liu Y (2017) A multi-task learning framework for emotion recognition using 2d continuous space. IEEE Trans Affect Comput 8(1):3–14. https://doi.org/10.1109/TAFFC.2015.2512598

    Article  CAS  Google Scholar 

  • Yarkoni T (2010) Personality in 100,000 words: a large-scale analysis of personality and word use among bloggers. J Res Pers 44(3):363–373

    Article  PubMed  PubMed Central  Google Scholar 

  • Yogesh C, Hariharan M, Ngadiran R, Adom AH, Yaacob S, Berkai C, Polat K (2017) A new hybrid pso assisted biogeography-based optimization for emotion and stress recognition from speech signal. Expert Syst Appl 69:149–158. https://doi.org/10.1016/j.eswa.2016.10.035

    Article  Google Scholar 

  • Zhou F, Qu X, Helander MG, Jiao JR (2011) Affect prediction from physiological measures via visual stimuli. Int J Hum Comput Stud 69(12):801–819

    Article  Google Scholar 

Download references

Acknowledgements

Author would like to thank the support from Foundation for Research Support of the State of Sao Paulo - FAPESP (Grant Numbers 2016/14267-7, 2016/25865-2, 2017/21054-2 and 2017/23655-3) for funding the bulk of this research project. In addition, we would like to thank the University of Groningen and the University of Ottawa.

Author information

Authors and Affiliations

  1. Institute of Mathematical and Computer Sciences, University of São Paulo, São Carlos, São Paulo, Brazil

    Luana Okino Sawada, Leandro Yukio Mano, José Rodrigues Torres Neto & Jó Ueyama

  2. Johann Bernoulli Institute for Mathematics and Computer Science, University of Groningen, Groningen, The Netherlands

    Leandro Yukio Mano

  3. School of Eletrical Engineering and Computer Science, University of Ottawa, Ottawa, ON, Canada

    José Rodrigues Torres Neto

Authors
  1. Luana Okino Sawada
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leandro Yukio Mano
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. José Rodrigues Torres Neto
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jó Ueyama
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luana Okino Sawada.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sawada, L.O., Mano, L.Y., Torres Neto, J.R. et al. A module-based framework to emotion recognition by speech: a case study in clinical simulation. J Ambient Intell Human Comput 14, 15513–15522 (2023). https://doi.org/10.1007/s12652-019-01280-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-019-01280-8

Keywords

Search

Navigation