survey

Driver Emotion Recognition for Intelligent Vehicles: A Survey

Authors:

Sebastian Zepf,

Javier Hernandez,

Alexander Schmitt,

Wolfgang Minker,

Rosalind W. PicardAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 53, Issue 3

Article No.: 64, Pages 1 - 30

https://doi.org/10.1145/3388790

Published: 04 July 2020 Publication History

Abstract

Driving can occupy a large portion of daily life and often can elicit negative emotional states like anger or stress, which can significantly impact road safety and long-term human health. In recent decades, the arrival of new tools to help recognize human affect has inspired increasing interest in how to develop emotion-aware systems for cars. To help researchers make needed advances in this area, this article provides a comprehensive literature survey of work addressing the problem of human emotion recognition in an automotive context. We systematically review the literature back to 2002 and identify 63 peer-review published articles on this topic. We overview each study’s methodology to measure and recognize emotions in the context of driving. Across the literature, we find a strong preference toward studying emotional states associated with high arousal and negative valence, monitoring the different states with cardiac, electrodermal activity, and speech signals, and using supervised machine learning to automatically infer the underlying human affective states. This article summarizes the existing work together with publicly available resources (e.g., datasets and tools) to help new researchers get started in this field. We also identify new research opportunities to help advance progress for improving driver emotion recognition.

References

[1]

Fadel Adib, Hongzi Mao, Zachary Kabelac, Dina Katabi, and Robert C. Miller. 2015. Smart homes that monitor breathing and heart rate. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, NY, 837--846.

[2]

Urvashi Agrawal, Shubhangi Giripunje, and Preeti Bajaj. 2013. Emotion and gesture recognition with soft computing tool for drivers assistance system in human centered transportation. In Proceedingsof the 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC’13). 4612--4616.

Digital Library

[3]

Ahmet Akbas. 2011. Evaluation of the physiological data indicating the dynamic stress level of drivers. Scientific Research and Essays 6, 2 (2011), 430--439.

[4]

Ignacio Alvarez, Karmele Lopez de Ipiña, Shaundra B. Daily, and Juan E. Gilbert. 2012. Emotional adaptive vehicle user interfaces: Moderating negative effects of failed technology interactions while driving. In Adjunct Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. 57--60.

[5]

Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. 2016. OpenFace: A General-Purpose Face Recognition Library with Mobile Applications. CMU School of Computer Science, Pittsburgh, PA.

[6]

Pongtep Angkititrakul, John H. L. Hansen, Sangjo Choi, Tyler Creek, Jeremy Hayes, Jeonghee Kim, Donggu Kwak, Levi T. Noecker, and Anhphuc Phan. 2009. UTDrive: The smart vehicle project. In In-Vehicle Corpus and Signal Processing for Driver Behavior, K. Takeda, J. H. L. Hangen, H. Erdogan, and H. Abut (Eds.). Springer, 55--67.

[7]

Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M. Martinez, and Seth D. Pollak. 2019. Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest 20, 1 (2019), 1--68.

[8]

Ahahina Begum, Mobyen Uddin Ahmed, Ahmed Mobyen Uddin, Peter Funk, and Reno Filla. 2012. Mental state monitoring system for the professional drivers based on heart rate variability analysis and case-based reasoning. In Proceedings of the Federal Conference on Computer Science and Information Systems (FedSIS’12). 35--42. http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=8arnumber=6354476.

[9]

Paul Boersma. 2014. The use of Praat in corpus research. In The Oxford Handbook of Corpus Phonology, J. Durand, U. Gut, and G. Kristoffersen (Eds.). Oxford Handbooks Online, 342--360.

[10]

Gianluca Borghini, Laura Astolfi, Giovanni Vecchiato, Donatella Mattia, and Fabio Babiloni. 2014. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience and Biobehavioral Reviews 44 (2014), 58--75.

[11]

Hynek Boril, Pinar Boyraz, and John H. L. Hansen. 2012. Towards multimodal driver’s stress detection. In Digital Signal Processing for In-Vehicle Systems and Safety, J. H. L. Hansen, P. Boyraz, K. Takeda, and H. Abut (Eds.). Springer, 3--19.

[12]

Hynek Boril, Tristan Kleinschmidt, Pinar Boyraz, and John H. L. Hansen. 2010. Impact of cognitive load and frustration on drivers’ speech.Journal of the Acoustical Society of America 127 (Sept. 2010), 1996. arxiv:33168

[13]

H. Boril, S. O. Sadjadi, and J. H. L. Hansen. 2011. UTDrive: Emotion and cognitive load classification for in-vehicle scenarios. In Proceedings of the 5th Biennial Workshop on Digital Signal Processing for In-Vehicle Systems (DSP’11). http://www.utd.edu/∼hynek/pdfs/BorilSadjadiHansen_DSP11.pdf.

[14]

Wolfram Boucsein. 2012. Electrodermal Activity. Springer Science 8 Business Media.

[15]

Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25, 1 (1994), 49--59. arxiv:0005-7916(93)E0016-Z

[16]

Johnell O. Brooks, Richard R. Goodenough, Matthew C. Crisler, Nathan D. Klein, Rebecca L. Alley, Beatrice L. Koon, William C. Logan, Jennifer H. Ogle, Richard A. Tyrrell, and Rebekkah F. Wills. 2010. Simulator sickness during driving simulation studies. Accident Analysis and Prevention 42, 3 (2010), 788--796.

[17]

John T. Cacioppo and Louis G. Tassinary. 1990. Inferring psychological significance from physiological signals. American Psychologist 45, 1 (1990), 16--28.

[18]

Hua Cai and Yingzi Lin. 2011. Modeling of operators emotion and task performance in a virtual driving environment. International Journal of Human Computer Studies 69, 9 (2011), 571--586.

Digital Library

[19]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2D pose estimation using part affinity fields. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). arxiv:1611.08050

[20]

Sailesh Conjeti, Rajiv Ranjan Singh, and Rahul Banerjee. 2012. Bio-inspired wearable computing architecture and physiological signal processing for on-road stress monitoring. Biomedical and Health Informatics 1, 0 (2012), 1--7.

[21]

Albert C. Cruz and Alex Rinaldi. 2017. Video summarization for expression analysis of motor vehicle operators. In Proceedings of the International Conference on Universal Access in Human-Computer Interaction. 313--323.

[22]

Yong Deng, Zhonghai Wu, Chao Hsien Chu, Qixun Zhang, and D. Frank Hsu. 2013. Sensor feature selection and combination for stress identification using combinatorial fusion. International Journal of Advanced Robotic Systems 10, 8 (2013), 306.

[23]

Ding Ding, Klaus Gebel, Philayrath Phongsavan, Adrian E. Bauman, and Dafna Merom. 2014. Driving: A road to unhealthy lifestyles and poor health outcomes. PloS One 9, 6 (June 2014), 1--5.

[24]

Monique Dittrich and Sebastian Zepf. 2019. Exploring the validity of methods to track emotions behind the wheel. In Proceedings of the International Conference on Persuasive Technology. 115--127.

[25]

Yanchao Dong, Zhencheng Hu, Keiichi Uchimura, and Nobuki Murayama. 2011. Driver inattention monitoring system for intelligent vehicles: A review. IEEE Transactions on Intelligent Transportation Systems 12, 2 (2011), 596--614.

Digital Library

[26]

Paul Ekman, Richard Davidson, Phoebe Ellsworth, Wallace V. Friesen, Robert Levenson, Harriet Oster, and Erika Rosenberg. 1992. Are there basic emotions? Psychological Review 99, 3 (1992), 550--553. arxiv:arXiv:1011.1669v3

[27]

Paul Ekman and Wallace V. Friesen. 1978. Facial Action Coding System: Investigator’s Guide. Consulting Psychologists Press.

[28]

Neska El Haouij, Jean Michel Poggi, Raja Ghozi, Sylvie Sevestre-Ghalila, and Mériem Jaïdane. 2018. Random forest-based approach for physiological functional variable selection for driver’s stress level classification. Statistical Methods 8 Applications 28 (2018), 157--185.

[29]

Luis Eudave and Miguel Valencia. 2017. Physiological response while driving in an immersive virtual environment. In Proceedings of the 2017 IEEE 14th International Conference on Wearable and Implantable Body Sensor Networks (BSN’17). 145--148.

[30]

Florian Eyben, Martin Wöllmer, Tony Poitschke, Björn Schuller, Christoph Blaschke, Berthold Färber, and Nhu Nguyen-Thien. 2010. Emotion on the road—Necessity, acceptance, and feasibility of affective computing in the car. Advances in Human-Computer Interaction 2010 (2010), Article 5.

[31]

Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. OpenSMILE: The Munich versatile and fast open-source audio feature extractor. In Proceedings of ACM Multimedia. 1459--1462.

Digital Library

[32]

Stephen H. Fairclough, Andrew J. Tattersall, and Kim Houston. 2006. Anxiety and performance in the British driving test. Transportation Research Part F: Traffic Psychology and Behaviour 9, 1 (2006), 43--52.

[33]

Raul Fernandez and Rosalind W. Picard. 2003. Modeling driver’s speech under stress. Speech Communication 40 (2003), 145--149.

Digital Library

[34]

H. Guo, A, Yüce, and J.-P. Thiran. 2014. Detecting emotional stress from facial expressions for driving safety. In Proceedings of the IEEE International Conference on Image Processing (ICIP’14), Vol. 1. 5961--5965.

[35]

Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. 2014. Physiobank, PhysioToolkit, and PhysioNet. Retrieved May 12, 2020 from https://www.ahajournals.org/doi/full/10.1161/01.cir.101.23.e215.

[36]

GPSBabel. 2019. Home Page. Retrieved May 12, 2020 from https://www.gpsbabel.org/.

[37]

Michael Grimm, Kristian Kroschel, Helen Harris, Clifford Nass, Bjorn Björn Schuller, Gerhard Rigoll, and Tobias Moosmayr. 2007. On the necessity and feasibility of detecting a driver’s emotional state while driving. Affective Computing and Intelligent Interaction 4738 (2007), 126--138.

Digital Library

[38]

Markus Groth, Thorsten Hennig-Thurau, and Gianfranco Walsh. 2009. Customer reactions to emotional labor: The roles of employee acting strategies and customer detection accuracy. Academy of Management Journal 52, 5 (2009), 958--974.

[39]

Oleg Gusikhin, Erica Klampfl, Dimitar Filev, and Yifan Chen. 2011. Emotive driver advisor system (EDAS). In Informatics in Control, Automation and Robotics. Lecture Notes in Electrical Engineering, Vol. Springer, 21--36.

[40]

Markus Gutmann, Patrik Grausberg, and Kyandoghere Kyamakya. 2015. Detecting human driver’s physiological stress and emotions using sophisticated one-person cockpit vehicle simulator. In Proceedings of the 2015 Information Technologies in Innovation Business Conference (ITIB’15). 15--18.

[41]

Mark A. Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations 11, 1 (2009), 10--18. arxiv:arXiv:1011.1669v3

Digital Library

[42]

Helen Harris and Clifford Nass. 2011. Emotion regulation for frustrating driving contexts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 749--752.

Digital Library

[43]

Jennifer A. Healey and Rosalind W. Picard. 2005. Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems 6, 2 (2005), 156--166.

Digital Library

[44]

Javier Hernandez, Zicheng Liu, Geoff Hulten, Dave DeBarr, Kyle Krum, and Zhengyou Zhang. 2013. Measuring the engagement level of TV viewers. In Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG’13). IEEE, Los Alamitos, CA, 1--7.

[45]

Javier Hernandez, Daniel McDuff, Xavier Benavides, Judith Amores, Pattie Maes, and Rosalind Picard. 2014. AutoEmotive: Bringing empathy to the driving experience to manage stress. In Proceedings of the 2014 Companion Publication on Designing Interactive Systems. 53--56.

Digital Library

[46]

Javier Hernandez, Rob R. Morris, and Rosalind W. Picard. 2011. Call center stress recognition with person-specific models. In Affective Computing and Intelligent Interaction. Lecture Notes in Computer Science, Vol. 6974, Springer, 125--134.

[47]

Stefan Hoch, Frank Althoff, G. McGlaun, and G. Rigoll. 2005. Bimodal fusion of emotional data in an automotive environment. In Proceedings of the IEEE International Conference on Acoustic, Speech, and Signal Processing. 1085--1088.

[48]

Klas Ihme, Christina Dömeland, Maria Freese, and Meike Jipp. 2018. Frustration in the Face of the Driver: A Simulator Study on Facial Muscle Activity During Frustrated Driving. Interaction Studies. John Benjamins Publishing Company.

[49]

Myounghoon Jeon. 2016. Don’t cry while you’re driving: Sad driving is as bad as angry driving. International Journal of Human-Computer Interaction 32, 10 (2016), 777--790.

[50]

Myounghoon Jeon, Jason Roberts, Parameshwaran Raman, Jung-Bin Yim, and Bruce N. Walker. 2011. Participatory design process for an in-vehicle affect detection and regulation system for various drivers. In Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility. 271--272.

[51]

In Cheol Jeong, Dong Hee Lee, Shin Woo Park, Jae Il Ko, and Hyung Ro Yoon. 2007. Automobile driver’s stress index provision system that utilizes electrocardiogram. In Proceedings of the 2007 IEEE Intelligent Vehicles Symposium. 652--656.

[52]

Christian Jones and Ing Marie Jonsson. 2008. Using paralinguistic cues in speech to recognise emotions in older car drivers. In Affect and Emotion in Human-Computer Interaction. Lecture Notes in Computer Science, Vol. 4868. Springer, 229--240.

[53]

Christian Martyn Jones and Ing-Marie Jonsson. 2005. Automatic recognition of affective cues in the speech of car drivers to allow appropriate responses. In Proceedings of the 17th Australia Conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future. 1--10.

Digital Library

[54]

Christian Martyn Jones and Ing Marie Jonsson. 2007. Performance analysis of acoustic emotion recognition for in-car conversational interfaces. In Universal Access in Human-Computer Interaction: Ambient Interaction. Lecture Notes in Computer Science, Vol. 4555. Springer, 411--420.

[55]

Ing-Marie Jonsson, Clifford Nass, Helen Harris, and Leila Takayama. 2005. Matching in-car voice with driver state: Impact on attitude and driving performance. In Proceedings of the 3rd International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design. 173--180.

[56]

O. Karaduman, H. Eren, H. Kurum, and M. Celenk. 2013. An effective variable selection algorithm for aggressive/calm driving detection via CAN bus. In Proceedings of the 2013 International Conference on Connected Vehicles and Expo (ICCVE’13). IEEE, Los Alamitos, CA, 586--591.

[57]

Salman Karimi and Mohammad Hossein Sedaaghi. 2013. Robust emotional speech classification in the presence of babble noise. International Journal of Speech Technology 16, 2 (2013), 215--227.

Digital Library

[58]

Tomokazu Kato, Haruki Kawanaka, Md. Shoaib Bhuiyan, and Koji Oguri. 2011. Classification of positive and negative emotion evoked by traffic jam based on electrocardiogram (ECG) and pulse wave. In Proceedings of the 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC’11). IEEE, Los Alamitos, CA, 1217--1222.

[59]

Christos D. Katsis, N. Katertsidis, George Ganiatsas, and Dimitrios I. Fotiadis. 2008. Toward emotion recognition in car racing drivers: A biosignal processing approach. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans 38, 3 (2008), 502--512.

[60]

Nobuo Kawaguchi and Shigeki Matsubara. 2001. Multimedia data collection of in-car speech communication. In Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech’01). 3--6.

[61]

N. Keshan, P. V. Parimi, and I. Bichindaritz. 2015. Machine learning for stress detection from ECG signals in automobile drivers. In Proceedings of the 2015 IEEE International Conference on Big Data (IEEE Big Data’15). 2661--2669.

Digital Library

[62]

Abhiram Kolli, Alireza Fasih, Fadi Al Machot, and Kyandoghere Kyamakya. 2011. Non-intrusive car driver’s emotion recognition using thermal camera. In Proceedings of the 3rd International Workshop on Nonlinear Dynamics and Synchronization (INDS’11) and the 16th International Symposium on Theoretical Electrical Engineering (ISTET’11). IEEE, Los Alamitos, CA, 1--5.

[63]

Arun Sai Krishnan, Xiping Hu, Jun-Qi Deng, Li Zhou, Edith C.-H. Ngai, Xitong Li, Victor C. M. Leung, and Yu-kwong Kwok. 2015. Towards in time music mood-mapping for drivers: A novel approach. In Proceedings of the 5th ACM Symposium on Development and Analysis of Intelligent Vehicular Networks and Applications. 59--66.

Digital Library

[64]

H. Leng, Y. Lin, and L. A. Zanzi. 2007. An experimental study on physiological parameters toward driver emotion recognition. Ergonomics and Health Aspects of Work with Computers 4566 (2007), 237--246.

[65]

Linda J. Levine. 1997. Reconstructing memory for emotions. Journal of Experimental Psychology: General 126, 2 (1997), 165.

[66]

Y. Lin, H. Leng, G. Yang, and H. Cai. 2007. An intelligent noninvasive sensor for driver pulse wave measurement. IEEE Sensors Journal 7, 5 (2007), 790--799.

[67]

C. Lisetti and F. Nasoz. 2005. Affective intelligent car interfaces with emotion recognition. Proceedings of 11th International Conference on Human Computer Interaction. 1--10. https://www.eurecom.fr/fr/publication/1797/download/mm-lisech-050722.pdf.

[68]

Andreas Löcken, Klas Ihme, and Anirudh Unni. 2017. Towards designing affect-aware systems for mitigating the effects of in-vehicle frustration. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications Adjunct (AutomotiveUI’17). 88--93.

Digital Library

[69]

Zhiyi Ma, Marwa Mahmoud, Peter Robinson, Eduardo Dias, and Lee Skrypchuk. 2017. Automatic detection of a driver’s complex mental states. In Proceedings of the International Conference on Computational Science and Its Applications. 678--691.

[70]

Diana MacLean, Asta Roseway, and Mary Czerwinski. 2013. MoodWings. In Proceedings of the 6th International Conference on Pervasive Technologies Related to Assistive Environments (PETRA’13).

[71]

Marek Malik, J. Thomas Bigger, A. John Camm, Robert E. Kleiger, Alberto Malliani, Arthur J. Moss, and Peter J. Schwartz. 1996. Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. European Heart Journal 17, 3 (1996), 354--381.

[72]

L. Malta, P. Angkititrakul, C. Miyajima, and K. Takeda. 2008. Multi-modal real-world driving data collection, transcription, and integration using Bayesian network. In Proceedings of the 2008 IEEE Intelligent Vehicles Symposium. 150--155.

[73]

Lucas Malta, Chiyomi Miyajima, Norihide Kitaoka, and Kazuya Takeda. 2011. Analysis of real-world driver’s frustration. IEEE Transactions on Intelligent Transportation Systems 12, 1 (2011), 109--118.

Digital Library

[74]

D. McNair, M. Lorr, and L. F. Droppleman. 1991. POMS: Profile of Mood States. Educational and Industrial Testing Service, San Diego, CA.

[75]

Jolieke Mesken, Marjan P. Hagenzieker, Talib Rothengatter, and Dick de Waard. 2007. Frequency, determinants, and consequences of different drivers’ emotions: An on-the-road study using self-reports, (observed) behaviour, and physiology. Transportation Research Part F: Traffic Psychology and Behaviour 10, 6 (2007), 458--475.

[76]

Tsuyoshi Moriyama. 2012. Face analysis of aggressive moods in automobile driving using mutual subspace method. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR’12). 2898--2901.

[77]

Nermine Munla, Mohamad Khalil, Ahmad Shahin, and Azzam Mourad. 2015. Driver stress level detection using HRV analysis. In Proceedings of the 2015 International Conference on Advances in Biomedical Engineering (ICABME’15). 61--64.

[78]

Kevin P. Murphy. 2009. The Bayes Net Toolbox for Matlab. Retrieved May 12, 2020 from https://www.cs.utah.edu/∼tch/notes/matlab/bnt/docs/bnt_pre_sf.html

[79]

Fatma Nasoz, Christine L. Lisetti, and Athanasios V. Vasilakos. 2010. Affectively intelligent and adaptive car interfaces. Information Sciences 180, 20 (2010), 3817--3836.

Digital Library

[80]

Fatma Nasoz, Onur Ozyer, Christine L. Lisetti, and Neal Finkelstein. 2002. Multimodal affective driver interfaces for future cars. In Proceedings of the 10th ACM International Conference on Multimedia. 319--322.

Digital Library

[81]

Clifford Nass, Ing-Marie Jonsson, Helen Harris, Ben Reaves, Jack Endo, Scott Brave, and Leila Takayama. 2005. Improving automotive safety by pairing driver emotion and car voice emotion. In Proceedings of CHI’05 Extended Abstracts on Human Factors in Computing Systems (CHI EA’05). 1973.

Digital Library

[82]

Khairun Nisa’Minhad, Sawal Hamid Md. Ali, Jonathan Ooi Shi Khai, and Siti Anom Ahmad. 2016. Human emotion classifications for automotive driver using skin conductance response signal. In Proceedings of the 2016 International Conference on Advances in Electrical, Electronic, and Systems Engineering (ICAEES’16). IEEE, Los Alamitos, CA, 371--375.

[83]

M. Oehl, F. W. Siebert, T.-K. Tews, R. Höger, and H.-R. Pfister. 2011. Improving human-machine interaction: A non invasive approach to detect emotions in car drivers. In Human-Computer Interaction: Towards Mobile and Intelligent Interaction Environments. Lecture Notes in Computer Science, Vol. 6763. Springer. 577--585.

[84]

Jonathan Shi Khai Ooi and Siti Anom Ahmad. 2016. Driver emotion recognition framework based on electrodermal activity measurements during simulated driving conditions. In Proceedings of the Conference on Biomedical Engineering and Sciences.

[85]

Pablo Enrique Paredes, Nur Al Huda Hamdan, Dav Clark, Carrie Cai, Wendy Ju, and James A. Landay. 2017. Evaluating in-car movements in the design of mindful commute interventions: Exploratory study. Journal of Medical Internet Research 19, 12 (2017), e372.

[86]

Pablo E. Paredes, Francisco Ordonez, Wendy Ju, and James A. Landay. 2018. Fast and furious: Detecting stress with a car steering wheel. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--12.

[87]

Thomas D. Parsons and Christopher G. Courtney. 2016. Interactions between threat and executive control in a virtual reality Stroop task. IEEE Transactions on Affective Computing 9, 1 (2016), 66--75.

Digital Library

[88]

M. Paschero, G. Del Vescovo, L. Benucci, A. Rizzi, M. Santello, G. Fabbri, and F. M. Frattale Mascioli. 2012. A real time classifier for emotion and stress recognition in a vehicle driver. In Proceedings of the IEEE International Symposium on Industrial Electronics. 1690--1695.

[89]

Rosalind W. Picard. 1997. Affective Computing. MIT Press, Cambridge, MA.

[90]

Ming Zher Poh, Daniel J. McDuff, and Rosalind W. Picard. 2011. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering 58, 1 (2011), 7--11.

[91]

Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. 2017. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion 37 (2017), 98--125.

Digital Library

[92]

Hamidur Rahman, Shaibal Barua, and Begum Shahina. 2015. Intelligent driver monitoring based on physiological sensor signals: Application using camera. In Proceedings of the IEEE Conference on Intelligent Transportation Systems (ITSC’15). 2637--2642.

Digital Library

[93]

Genaro Rebolledo-Mendez, Angelica Reyes, Sebastian Paszkowicz, Mari Carmen Domingo, and Lee Skrypchuk. 2014. Developing a body sensor network to detect emotions during driving. IEEE Transactions on Intelligent Transportation Systems 15, 4 (2014), 1850--1854.

[94]

Andreas Riener, Alois Ferscha, and Mohamed Aly. 2009. Heart on the road: HRV analysis for monitoring a driver’s affective state. In Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI’09). 99--106.

Digital Library

[95]

George Rigas, Yorgos Goletsis, and Dimitrios I. Fotiadis. 2012. Real-time driver’s stress event detection. IEEE Transactions on Intelligent Transportation Systems 13, 1 (2012), 221--234.

Digital Library

[96]

Ognjen Rudovic, Jaeryoung Lee, Miles Dai, Björn Schuller, and Rosalind W. Picard. 2018. Personalized machine learning for robot perception of affect and engagement in autism therapy. Science Robotics 3, 19 (2018), eaao6760.

[97]

Daniele Ruscio, Luca Bascetta, Alessandro Gabrielli, Matteo Matteucci, and Lorenzo Mussone. 2017. Collection and comparison of driver/passenger physiologic and behavioural data in simulation and on-road driving. In Proceedings of the IEEE International Conference on Models and Technologies for Intelligent Transportation Systems. 403--408.

[98]

James A. Russell. 1980. A circumplex model of affect. Personality and Social Psychology 39 (1980), 1161--1178.

[99]

Aaqib Saeed and Stojan Trajanovski. 2017. Personalized driver stress detection with multi-task neural networks using physiological signals. In Proceedings of the Conference on Neural Information Processing Systems. http://arxiv.org/abs/1711.06116

[100]

Arun Sahayadhas, Kenneth Sundaraj, and Murugappan Murugappan. 2012. Detecting driver drowsiness based on sensors: A review. Sensors (Switzerland) 12, 12 (2012), 16937--16953.

[101]

Bjorn Schuller, Felix Friedmann, and Florian Eyben. 2013. Automatic recognition of physiological parameters in the human voice: Heart rate and skin conductance. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’13). 7219--7223.

[102]

Björn Schuller, Manfred Lang, and Gerhard Rigoll. 2006. Recognition of spontaneous emotions by speech within automotive environment. Tagungsband Fortschritte der Akustik (DAGA’06). 57--58. http://www.mmk.ei.tum.de/publ/pdf/06/06sch5.pdf.

[103]

Björn Schuller, Gerhard Rigoll, and Manfred Lang. 2004. Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture. Acoustics, Speech, and Signal Processing 1 (2004), 577--580.

[104]

Bjoern Schuller, Matthias Wimmer, Dejan Arsic, Tobias Moosmayr, and Gerhard Rigoll. 2008. Detection of security related affect and behaviour in passenger transport. In Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH’08). 265--268.

[105]

Bjoern W. Schuller. 2008. Speaker, noise, and acoustic space adaptation for emotion recognition in the automotive environment. In Proceedings of the ITG Conference on Voice Communication (SprachKommunikation’08). 1--4. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5759973

[106]

Liping Shen, Minjuan Wang, and Ruimin Shen. 2009. Affective e-learning: Using emotional data to improve learning in pervasive learning environment related work and the pervasive e-learning platform. Educational Technology 8 Society 12 (2009), 176--189.

[107]

Saul Shiffman, Arthur A. Stone, and Michael R. Hufford. 2008. Ecological momentary assessment. Annual Review of Clinical Psychology 4, 1 (2008), 1--32.

[108]

Felix W. Siebert, Michael Oehl, and H.-R. Pfister. 2010. The measurement of grip-strength in automobiles: A new approach to detect driver’s emotions. In Advances in Human Factors, Ergonomics, and Safety in Manufacturing and Service Industry, W. Karwowski and G. Salvendy (Eds.). CRC Press, Boca Raton, FL, 775--782.

[109]

Mohamad Hoseyn Sigari, Mahmood Fathy, and Mohsen Soryani. 2013. A driver face monitoring system for fatigue and distraction detection. International Journal of Vehicular Technology 2013 (2013), 73--100. arxiv:263983

[110]

Rajiv Ranjan Singh and Rahul Banerjee. 2010. Multi-parametric analysis of sensory data collected from automotive drivers for building a safety-critical wearable computing system. In Proceedings of the 2010 International Conference on Computer Engineering and Technology (ICCET’10), Vol. 1.355--360.

[111]

Rajiv Ranjan Singh, Sailesh Conjeti, and Rahul Banerjee. 2011. An approach for real-time stress-trend detection using physiological signals in wearable computing systems for automotive drivers. In Proceedings of the 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC’11). 1477--1482.

[112]

Rajiv Ranjan Singh, Sailesh Conjeti, and Rahul Banerjee. 2012. Biosignal based on-road stress monitoring for automotive drivers. In Proceedings of the 2012 National Conference on Communications (NCC’12). 8--9.

[113]

Rajiv Ranjan Singh, Sailesh Conjeti, and Rahul Banerjee. 2013. A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals. Biomedical Signal Processing and Control 8, 6 (2013), 740--754.

[114]

Kare Sjölander. 2004. The snack sound toolkit. http://www.speech.kth.se/snack.

[115]

Kåre Sjölander and Jonas Beskow. 2000. Wavesurfer—An open source speech tool. Interspeech 4 (2000), 464--467.

[116]

SourceForge. 2012. The Open Racing Car Simulator (TORCS). Retrieved May 12, 2020 from http://torcs.sourceforge.net/.

[117]

SourceForge. 2019. The BioSig Project. Retrieved May 12, 2020 from http://biosig.sourceforge.net/.

[118]

Olga Sourina, Yisi Liu, Qiang Wang, and Minh Khoa Nguyen. 2011. EEG-based personalized digital experience. In Universal Access in Human-Computer Interaction: Users Diversity. Lecture Notes in Computer Science, Vol. 6766. Springer, 591--599.

[119]

Ronnie Taib, Jeremy Tederry, and Benjamin Itzstein. 2014. Quantifying driver frustration to improve road safety. In Proceedings of the Extended Abstracts of the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI EA’14). 1777--1782.

Digital Library

[120]

Ashish Tawari and Mohan Trivedi. 2010. Speech emotion analysis in noisy real-world environment. In Proceedings of the International Conference on Pattern Recognition.

Digital Library

[121]

Ashish Tawari and Mohan M. Trivedi. 2010. Speech based emotion classification framework for driver assistance system. In Proceedings of the IEEE Intelligent Vehicles Symposium. 174--178.

[122]

Ashish Tawari and Mohan Manubhai Trivedi. 2010. Speech emotion analysis: Exploring the role of context. IEEE Transactions on Multimedia 12, 6 (2010), 502--509.

Digital Library

[123]

OpenCV Team. 2019. Open Source Computer Vision Library. Retrieved May 12, 2020 from https://opencv.org/.

[124]

Masaharu Terasaki, Youlchl Klshimoto, and Alto Koga. 1992. Construction of a multiple mood scale. Japanese Journal of Psychology 62, 6 (1992), 350--356.

[125]

Tessa Karina Tews, Michael Oehl, Felix W. Siebert, Rainer Höger, and Helmut Faasch. 2011. Emotional human-machine interaction: Cues from facial expressions. In Human Interface and the Management of Information: Interacting with Information. Lecture Notes in Computer Science, Vol. 6771. Springer, 641--650.

[126]

M. Tischler, C. Peter, M. Wimmer, and J. Voskamp. 2007. Application of emotion recognition methods in automotive research. In Proceedings of the Workshop on Emotion and Computing—Current Research and Future Impact. 50--55. http://ias.cs.tum.edu/_media/spezial/bib/tischler07application.pdf.

[127]

Geoffrey Underwood, Peter Chapman, Sharon Wright, and David Crundall. 1999. Anger while driving. Transportation Research Part F: Traffic Psychology and Behaviour 2, 1 (1999), 55--68.

[128]

Paul Viola and M. J. Jones. 2004. Robust real-time face detection. International Journal of Computer Vision 57, 2 (2004), 137--154. arxiv:arXiv:1011.1669v3

Digital Library

[129]

Jinjun Wang and Yihong Gong. 2008. Recognition of multiple drivers’ emotional state. In Proceedings of the 2008 19th International Conference on Pattern Recognition. 1--4.

[130]

Jeen Shing Wang, Che Wei Lin, and Ya Ting C. Yang. 2013. A k-nearest-neighbor classifier with heart rate variability feature-based transformation algorithm for driving stress recognition. Neurocomputing 116 (2013), 136--143.

[131]

D. Watson, L. A. Clark, and A. Tellegen. 1988. Development and validation of brief measures of positive and negative affect: The PANAS scales.Journal of Personality and Social Psychology 54, 6 (1988), 1063--1070.

[132]

Frank H. Wilhelm and Paul Grossman. 2010. Emotions beyond the laboratory: Theoretical fundaments, study design, and analytic strategies for advanced ambulatory assessment. Biological Psychology 84, 3 (2010), 552--569.

[133]

Kenton Williams, José Acevedo Flores, and Joshua Peters. 2014. Affective robot influence on driver adherence to safety, cognitive load reduction and sociability. In Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI’14). 1--8.

Digital Library

[134]

Mingmin Zhao, Fadel Adib, and Dina Katabi. 2016. Emotion recognition using wireless signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, New York, NY, 95--108.

Digital Library

Cited By

Ma FYuan YXie YRen HLiu IHe YRen FYu FNi S(2025)Generative technology for human emotion recognition: A scoping reviewInformation Fusion10.1016/j.inffus.2024.102753115(102753)Online publication date: Mar-2025
https://doi.org/10.1016/j.inffus.2024.102753
Mattioli MCabitza F(2024)Not in My Face: Challenges and Ethical Considerations in Automatic Face Emotion Recognition TechnologyMachine Learning and Knowledge Extraction10.3390/make60401096:4(2201-2231)Online publication date: 30-Sep-2024
https://doi.org/10.3390/make6040109
García-Hernández RLuna-García HCelaya-Padilla JGarcía-Hernández AReveles-Gómez LFlores-Chaires LDelgado-Contreras JRondon DVillalba-Condori K(2024)A Systematic Literature Review of Modalities, Trends, and Limitations in Emotion Recognition, Affective Computing, and Sentiment AnalysisApplied Sciences10.3390/app1416716514:16(7165)Online publication date: 15-Aug-2024
https://doi.org/10.3390/app14167165
Show More Cited By

Index Terms

Driver Emotion Recognition for Intelligent Vehicles: A Survey

Recommendations

Towards Empathetic Car Interfaces: Emotional Triggers while Driving
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Monitoring the emotions of drivers can play a critical role to reduce road accidents and enable novel driver-car interactions. To help understand the possibilities, this work systematically studies the in-road triggers that may lead to different ...
Improving automotive safety by pairing driver emotion and car voice emotion
CHI EA '05: CHI '05 Extended Abstracts on Human Factors in Computing Systems

This study examines whether characteristics of a car voice can affect driver performance and affect. In a 2 (driver emotion: happy or upset) x 2 (car voice emotion: energetic vs. subdued) experimental study, participants (N=40) had emotion induced ...
Emotion Recognition Using Physiological Signals
MIDI '15: Proceedings of the Mulitimedia, Interaction, Design and Innnovation

In this paper the problem of emotion recognition using physiological signals is presented. Firstly the problems with acquisition of physiological signals related to specific human emotions are described. It is not a trivial problem to elicit real ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 53, Issue 3

May 2021

787 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/3403423

Editor:
Albert Zomaya
University of Sydney, Australia

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 July 2020

Online AM: 07 May 2020

Accepted: 01 March 2020

Revised: 01 March 2020

Received: 01 November 2019

Published in CSUR Volume 53, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Survey
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

144
Total Citations
View Citations
3,337
Total Downloads

Downloads (Last 12 months)743
Downloads (Last 6 weeks)130

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ma FYuan YXie YRen HLiu IHe YRen FYu FNi S(2025)Generative technology for human emotion recognition: A scoping reviewInformation Fusion10.1016/j.inffus.2024.102753115(102753)Online publication date: Mar-2025
https://doi.org/10.1016/j.inffus.2024.102753
Mattioli MCabitza F(2024)Not in My Face: Challenges and Ethical Considerations in Automatic Face Emotion Recognition TechnologyMachine Learning and Knowledge Extraction10.3390/make60401096:4(2201-2231)Online publication date: 30-Sep-2024
https://doi.org/10.3390/make6040109
García-Hernández RLuna-García HCelaya-Padilla JGarcía-Hernández AReveles-Gómez LFlores-Chaires LDelgado-Contreras JRondon DVillalba-Condori K(2024)A Systematic Literature Review of Modalities, Trends, and Limitations in Emotion Recognition, Affective Computing, and Sentiment AnalysisApplied Sciences10.3390/app1416716514:16(7165)Online publication date: 15-Aug-2024
https://doi.org/10.3390/app14167165
Wu YZhang SLi P(2024)Improvement of Multimodal Emotion Recognition Based on Temporal-Aware Bi-Direction Multi-Scale Network and Multi-Head Attention MechanismsApplied Sciences10.3390/app1408327614:8(3276)Online publication date: 13-Apr-2024
https://doi.org/10.3390/app14083276
江靖(2024)Road Rage Identification and Real-Time Monitoring: Technological Advances and ChallengesAdvances in Psychology10.12677/ap.2024.14856714:08(447-460)Online publication date: 2024
https://doi.org/10.12677/ap.2024.148567
Kovacevic NHolz CGross MWampfler R(2024)On Multimodal Emotion Recognition for Human-Chatbot Interaction in the WildProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685759(12-21)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685759
Lapedriza AAlbo Canals JElhaouij NLee JHernandez JAmores JYun KJerez Molina CLahuerta LGonzalez NFernandez MAyats Soler MFerreira Quartino SBallesteros APicard RGrollman DBroadbent EJu WSoh HWilliams T(2024)Deploying a Robotic ride-on Car in the Hospital to Reduce the Stress of Pediatric Patients before SurgeryCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3641081(650-654)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3641081
Zheng ZAkash KMisu TGrollman DBroadbent EJu WSoh HWilliams T(2024)DualTake: Predicting Takeovers across Mobilities for Future Personalized Mobility ServicesCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640610(1194-1198)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3640610
Wu SZhou LHu ZLiu J(2024)Hierarchical Context-Based Emotion Recognition With Scene GraphsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.319683135:3(3725-3739)Online publication date: Mar-2024
https://doi.org/10.1109/TNNLS.2022.3196831
Li WCao DTan RShi TGao ZMa JGuo GHu HFeng JWang L(2024)Intelligent Cockpit for Intelligent Connected Vehicles: Definition, Taxonomy, Technology and EvaluationIEEE Transactions on Intelligent Vehicles10.1109/TIV.2023.33397989:2(3140-3153)Online publication date: Feb-2024
https://doi.org/10.1109/TIV.2023.3339798
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents