Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3577190.3614171acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article
Open access

Expanding the Role of Affective Phenomena in Multimodal Interaction Research

Published: 09 October 2023 Publication History

Abstract

In recent decades, the field of affective computing has made substantial progress in advancing the ability of AI systems to recognize and express affective phenomena, such as affect and emotions, during human-human and human-machine interactions. This paper describes our examination of research at the intersection of multimodal interaction and affective computing, with the objective of observing trends and identifying understudied areas. We examined over 16,000 papers from selected conferences in multimodal interaction, affective computing, and natural language processing: ACM International Conference on Multimodal Interaction, AAAC International Conference on Affective Computing and Intelligent Interaction, Annual Meeting of the Association for Computational Linguistics, and Conference on Empirical Methods in Natural Language Processing. We identified 910 affect-related papers and present our analysis of the role of affective phenomena in these papers. We find that this body of research has primarily focused on enabling machines to recognize or express affect and emotion; there has been limited research on how affect and emotion predictions might, in turn, be used by AI systems to enhance machine understanding of human social behaviors and cognitive states. Based on our analysis, we discuss directions to expand the role of affective phenomena in multimodal interaction research.

Supplemental Material

PDF File
Appendix

References

[1]
Ralph Adolphs and Antonio R Damasio. 2001. The interaction of affect and cognition: A neurobiological perspective. (2001).
[2]
Shazia Afzal, Bikram Sengupta, Munira Syed, Nitesh Chawla, G Alex Ambrose, and Malolan Chetlur. 2017. The ABC of MOOCs: Affect and its inter-play with behavior and cognition. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 279–284.
[3]
JinYeong Bak, Suin Kim, and Alice Oh. 2012. Self-disclosure and relationship strength in twitter conversations. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 60–64.
[4]
Matthew Barthet, Ahmed Khalifa, Antonios Liapis, and Georgios N Yannakakis. 2022. Play with Emotion: Affect-Driven Reinforcement Learning. In 2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1–8.
[5]
Antoine Bechara, Antonio R Damasio, Hanna Damasio, and Steven W Anderson. 1994. Insensitivity to future consequences following damage to human prefrontal cortex. Cognition 50, 1-3 (1994), 7–15.
[6]
Joan-Isaac Biel, Lucía Teijeiro-Mosquera, and Daniel Gatica-Perez. 2012. FaceTube: Predicting Personality from Facial Expressions of Emotion in Online Conversational Video. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (Santa Monica, California, USA) (ICMI ’12). Association for Computing Machinery, New York, NY, USA, 53–56. https://doi.org/10.1145/2388676.2388689
[7]
Kristy Boyer, Joseph F Grafsgaard, Eun Young Ha, Robert Phillips, and James Lester. 2011. An affect-enriched dialogue act classification model for task-oriented dialogue. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 1190–1199.
[8]
Dushyant Singh Chauhan, SR Dhanush, Asif Ekbal, and Pushpak Bhattacharyya. 2020. Sentiment and emotion help sarcasm? a multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 4351–4360.
[9]
Kushal Chawla, Rene Clever, Jaysa Ramirez, Gale Lucas, and Jonathan Gratch. 2021. Towards emotion-aware agents for negotiation dialogues. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1–8.
[10]
Kushal Chawla, Sopan Khosla, Niyati Chhaya, and Kokil Jaidka. 2019. Pre-trained affective word representations. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1–7.
[11]
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017).
[12]
Gerald L Clore and Donn Byrne. 1974. A reinforcement-affect model of attraction. Foundations of interpersonal attraction (1974), 143–170.
[13]
Gerald L Clore, Karen Gasper, and Erika Garvin. 2001. Affect as information. Handbook of affect and social cognition (2001), 121–144.
[14]
Gerald L Clore and John B Gormly. 1974. Knowing, feeling, and liking a psychophysiological study of attraction. Journal of Research in Personality 8, 3 (1974), 218–230.
[15]
Gerald L Clore and Jesse Pappas. 2007. The affective regulation of social interaction. Social Psychology Quarterly 70, 4 (2007), 333–339.
[16]
Gerald L Clore, Robert S Wyer, Bruce Dienes, Karen Gasper, Carol Gohm, and Linda Isbell. 2013. Affective feelings as feedback: Some cognitive consequences. In Theories of mood and cognition. Psychology Press, 27–62.
[17]
Elizabeth B Cloude, Franz Wortha, Daryn A Dever, and Roger Azevedo. 2021. Negative emotional dynamics shape cognition and performance with MetaTutor: toward building affect-aware systems. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1–8.
[18]
Verna Dankers, Marek Rei, Martha Lewis, and Ekaterina Shutova. 2019. Modelling the interplay of metaphor and emotion through multitask learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2218–2229.
[19]
Richard J Davidson, Klaus R Sherer, and H Hill Goldsmith. 2009. Handbook of affective sciences. Oxford University Press.
[20]
Sidney K D’mello and Jacqueline Kory. 2015. A review and meta-analysis of multimodal affect detection systems. ACM computing surveys (CSUR) 47, 3 (2015), 1–36.
[21]
Daniel Dukes, Kathryn Abrams, Ralph Adolphs, Mohammed E Ahmed, Andrew Beatty, Kent C Berridge, Susan Broomhall, Tobias Brosch, Joseph J Campos, Zanna Clay, 2021. The rise of affectivism. Nature human behaviour 5, 7 (2021), 816–820.
[22]
Joseph P Forgas. 2012. Affect in social thinking and behavior. Psychology Press.
[23]
Joseph P Forgas. 2013. The affect infusion model (AIM): An integrative theory of mood effects on cognition and judgments. In Theories of mood and cognition. Psychology Press, 99–134.
[24]
Daniel Gabana, Laurissa Tokarchuk, Emily Hannon, and Hatice Gunes. 2017. Effects of valence and arousal on working memory performance in virtual reality gaming. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 36–41.
[25]
Bilal Ghanem, Simone Paolo Ponzetto, Paolo Rosso, and Francisco Rangel. 2021. Fakeflow: Fake news detection by modeling the flow of affective information. arXiv preprint arXiv:2101.09810 (2021).
[26]
Sayan Ghosh, Moitreya Chatterjee, and Louis-Philippe Morency. 2014. A multimodal context-based approach for distress assessment. In Proceedings of the 16th International Conference on Multimodal Interaction. 240–246.
[27]
Jesse Hoey, Tobias Schroder, and Areej Alhothali. 2013. Bayesian affect control theory. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE, 166–172.
[28]
Kai Hong, Christian G Kohler, Mary E March, Amber A Parker, and Ani Nenkova. 2012. Lexical differences in autobiographical narratives from schizophrenic patients and healthy controls. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 37–47.
[29]
Pere-Lluís Huguet-Cabot, David Abadi, Agneta Fischer, and Ekaterina Shutova. 2021. Us vs. them: a dataset of populist attitudes, news bias and emotions. arXiv preprint arXiv:2101.11956 (2021).
[30]
Alice M Isen. 1987. Positive affect, cognitive processes, and social behavior. In Advances in experimental social psychology. Vol. 20. Elsevier, 203–253.
[31]
Alice M Isen and Barbara Means. 1983. The influence of positive affect on decision-making strategy. Social cognition 2, 1 (1983), 18–31.
[32]
Tiffany A Ito and John T Cacioppo. 2001. Affect and attitudes: A social neuroscience approach. (2001).
[33]
Hyeju Jang, Yohan Jo, Qinlan Shen, Michael Miller, Seungwhan Moon, and Carolyn Rose. 2016. Metaphor detection with topic transition, emotion and cognition in context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 216–225.
[34]
Apostolos Kalatzis, Vishnunarayan Girishan Prabhu, Saidur Rahman, Mike Wittie, and Laura Stanley. 2022. Emotions Matter: Towards Personalizing Human-System Interactions Using a Two-layer Multimodal Approach. In Proceedings of the 2022 International Conference on Multimodal Interaction. 63–72.
[35]
Karl Christoph Klauer and Jochen Musch. 2003. Affective priming: Findings and theories. The psychology of evaluation: Affective processes in cognition and emotion 7 (2003), 49.
[36]
Shiro Kumano, Kazuhiro Otsuka, Dan Mikami, and Junji Yamato. 2009. Recognizing Communicative Facial Expressions for Discovering Interpersonal Emotions in Group Meetings. In Proceedings of the 2009 International Conference on Multimodal Interfaces (Cambridge, Massachusetts, USA) (ICMI-MLMI ’09). Association for Computing Machinery, New York, NY, USA, 99–106. https://doi.org/10.1145/1647314.1647333
[37]
Allison Lahnala, Charles Welch, Béla Neuendorf, and Lucie Flek. 2022. Mitigating toxic degeneration with empathetic data: Exploring the relationship between toxicity and empathy. arXiv preprint arXiv:2205.07233 (2022).
[38]
Jina Lee, Helmut Prendinger, Alena Neviarouskaya, and Stacy Marsella. 2009. Learning models of speaker head nods with affective information. In 2009 3rd international conference on affective computing and intelligent interaction and workshops. IEEE, 1–6.
[39]
Minha Lee, Jaebok Kim, Khiet Truong, Yvonne de Kort, Femke Beute, and Wijnand IJsselsteijn. 2017. Exploring moral conflicts in speech: multidisciplinary analysis of affect and stress. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 407–414.
[40]
Elizabeth A Lemerise and William F Arsenio. 2000. An integrated model of emotion processes and cognition in social information processing. Child development 71, 1 (2000), 107–118.
[41]
Jinying Lin, Zhen Ma, Randy Gomez, Keisuke Nakamura, Bo He, and Guangliang Li. 2020. A Review on Interactive Reinforcement Learning From Human Social Feedback. IEEE Access 8 (2020), 120757–120765. https://doi.org/10.1109/ACCESS.2020.3006254
[42]
Lucien Maman, Mohamed Chetouani, Laurence Likforman-Sulem, and Giovanna Varni. 2021. Using Valence Emotion to Predict Group Cohesion’s Dynamics: Top-down and Bottom-up Approaches. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1–8.
[43]
Leena Mathur and Maja J Matarić. 2020. Introducing representations of facial affect in automated multimodal deception detection. In Proceedings of the 2020 International Conference on Multimodal Interaction. 305–314.
[44]
Kazuyuki Matsumoto, Kyosuke Akita, Minoru Yoshida, Kenji Kita, and Fuji Ren. 2015. Estimate the intimacy of the characters based on their emotional states for application to non-task dialogue. In 2015 International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 327–333.
[45]
Daniel McDuff, Rana El Kaliouby, Evan Kodra, and Rosalind Picard. 2013. Measuring voter’s candidate preference based on affective responses to election debates. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE, 369–374.
[46]
Bert S Moore and Alice M Isen. 1990. Affect and social behavior. Cambridge University Press.
[47]
Robert R Morris, Mira Dontcheva, Adam Finkelstein, and Elizabeth Gerber. 2013. Affect and creative performance on crowdsourcing platforms. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE, 67–72.
[48]
Paula M Niedenthal, Jamin B Halberstadt, Jonathan Margolin, and Åse H Innes-Ker. 2000. Emotional state and the detection of change in facial expression of emotion. European journal of social psychology 30, 2 (2000), 211–222.
[49]
Silviu Vlad Oprea, Steven Wilson, and Walid Magdy. 2022. Should a Chatbot be Sarcastic? Understanding User Preferences Towards Sarcasm Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 7686–7700.
[50]
Maja Pantic and Leon JM Rothkrantz. 2003. Toward an affect-sensitive multimodal human-computer interaction. Proc. IEEE 91, 9 (2003), 1370–1390.
[51]
Srinivas Parthasarathy and Carlos Busso. 2017. Predicting speaker recognition reliability by considering emotional content. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 434–439.
[52]
Jing Peng, Anna Feldman, and Ekaterina Vylomova. 2018. Classifying idiomatic and literal expressions using topic models and intensity of emotions. arXiv preprint arXiv:1802.09961 (2018).
[53]
Rosalind W Picard. 2000. Affective computing. MIT press.
[54]
Kosmas Pinitas, Konstantinos Makantasis, Antonios Liapis, and Georgios N. Yannakakis. 2022. Supervised Contrastive Learning for Affect Modelling. In Proceedings of the 2022 International Conference on Multimodal Interaction (Bengaluru, India) (ICMI ’22). Association for Computing Machinery, New York, NY, USA, 531–539. https://doi.org/10.1145/3536221.3556584
[55]
Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. 2017. A review of affective computing: From unimodal analysis to multimodal fusion. Information fusion 37 (2017), 98–125.
[56]
Santhosh Rajamanickam, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2020. Joint modelling of emotion and abusive language detection. arXiv preprint arXiv:2005.14028 (2020).
[57]
Vikram Ramanarayanan, Chee Wee Leong, Lei Chen, Gary Feng, and David Suendermann-Oeft. 2015. Evaluating speech, face, emotion and body movement time-series features for automated multimodal presentation scoring. In Proceedings of the 2015 acm on international conference on multimodal interaction. 23–30.
[58]
Sandratra Rasendrasoa, Alexandre Pauchet, Julien Saunier, and Sébastien Adam. 2022. Real-Time Multimodal Emotion Recognition in Conversation for Multi-Party Interactions. In Proceedings of the 2022 International Conference on Multimodal Interaction (Bengaluru, India) (ICMI ’22). Association for Computing Machinery, New York, NY, USA, 395–403. https://doi.org/10.1145/3536221.3556601
[59]
Tulika Saha, Aditya Patra, Sriparna Saha, and Pushpak Bhattacharyya. 2020. Towards emotion-aided multi-modal dialogue act classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 4361–4372.
[60]
Tulika Saha, Apoorva Upadhyaya, Sriparna Saha, and Pushpak Bhattacharyya. 2021. Towards sentiment and emotion aided multi-modal speech act classification in twitter. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 5727–5737.
[61]
Ramit Sawhney, Harshit Joshi, Lucie Flek, and Rajiv Shah. 2021. Phase: Learning emotional phase-aware representations for suicide ideation detection on social media. In Proceedings of the 16th conference of the European Chapter of the Association for Computational Linguistics: main volume. 2415–2428.
[62]
Ramit Sawhney, Harshit Joshi, Saumya Gandhi, and Rajiv Shah. 2020. A time-aware transformer based model for suicide ideation detection on social media. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). 7685–7697.
[63]
Ramit Sawhney, Harshit Joshi, Rajiv Shah, and Lucie Flek. 2021. Suicide ideation detection via social and temporal user representations using hyperbolic learning. In Proceedings of the 2021 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies. 2176–2190.
[64]
Ramit Sawhney, Puneet Mathur, Taru Jain, Akash Kumar Gautam, and Rajiv Shah. 2021. Multitask learning for emotionally analyzing sexual abuse disclosures. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4881–4892.
[65]
Stefan Scherer, Giota Stratou, and Louis-Philippe Morency. 2013. Audiovisual behavior descriptors for depression assessment. In Proceedings of the 15th ACM on International conference on multimodal interaction. 135–140.
[66]
Taylan Sen, Mohammad Rafayet Ali, Mohammed Ehsan Hoque, Ronald Epstein, and Paul Duberstein. 2017. Modeling doctor-patient communication with affective text analysis. In 2017 seventh international conference on affective computing and intelligent interaction (ACII). IEEE, 170–177.
[67]
Behjat Siddiquie, Dave Chisholm, and Ajay Divakaran. 2015. Exploiting multimodal affect and semantics to identify politically persuasive web videos. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. 203–210.
[68]
Gizem Sogancioglu, Heysem Kaya, and Albert Ali Salah. 2021. Can mood primitives predict apparent personality?. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1–8.
[69]
Samuel Spaulding and Cynthia Breazeal. 2019. Frustratingly Easy Personalization for Real-time Affect Interpretation of Facial Expression. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). 531–537. https://doi.org/10.1109/ACII.2019.8925515
[70]
Giota Stratou, Rens Hoegen, Gale Lucas, and Jonathan Gratch. 2015. Emotional signaling in a social dilemma: An automatic analysis. In 2015 International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 180–186.
[71]
Jianhua Tao and Tieniu Tan. 2005. Affective computing: A review. In International Conference on Affective computing and intelligent interaction. Springer, 981–995.
[72]
Elsbeth Turcan, Smaranda Muresan, and Kathleen McKeown. 2021. Emotion-infused models for explainable psychological stress detection. In Proceedings of the 2021 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies. 2895–2909.
[73]
Julia Wache, Ramanathan Subramanian, Mojtaba Khomami Abadi, Radu-Laurentiu Vieriu, Nicu Sebe, and Stefan Winkler. 2015. Implicit user-centric personality recognition based on physiological responses to emotional videos. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. 239–246.
[74]
Yan Wang, Wei Song, Wei Tao, Antonio Liotta, Dawei Yang, Xinlei Li, Shuyong Gao, Yixuan Sun, Weifeng Ge, Wei Zhang, 2022. A systematic review on affective computing: Emotion models, databases, and recent advances. Information Fusion 83 (2022), 19–52.
[75]
Adrienne Wood, Jennie Lipson, Olivia Zhao, and Paula Niedenthal. 2021. Forms and functions of affective synchrony. Handbook of embodied psychology: Thinking, feeling, and acting (2021), 381–402.
[76]
Zhaojun Yang, Boqing Gong, and Shrikanth Narayanan. 2017. Weighted geodesic flow kernel for interpersonal mutual influence modeling and emotion recognition in dyadic interactions. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 236–241.
[77]
Zixiaofan Yang, Shayan Hooshmand, and Julia Hirschberg. 2021. CHoRaL: Collecting humor reaction labels from millions of social media users. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 4429–4435.
[78]
Yiqun Yao, Michalis Papakostas, Mihai Burzo, Mohamed Abouelenien, and Rada Mihalcea. 2021. Muser: Multimodal stress detection using emotion recognition as an auxiliary task. arXiv preprint arXiv:2105.08146 (2021).
[79]
Amir Zadeh, Paul Pu Liang, and Louis-Philippe Morency. 2020. Foundations of multimodal co-learning. Information Fusion 64 (2020), 188–193.
[80]
Bowen Zhang, Min Yang, Xutao Li, Yunming Ye, Xiaofei Xu, and Kuai Dai. 2020. Enhancing cross-target stance detection with transferable semantic-emotion knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 3188–3197.

Cited By

View all
  • (2024)Automatic mild cognitive impairment estimation from the group conversation of coimagination methodProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685754(355-360)Online publication date: 4-Nov-2024

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '23: Proceedings of the 25th International Conference on Multimodal Interaction
October 2023
858 pages
ISBN:9798400700552
DOI:10.1145/3577190
This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 October 2023

Check for updates

Author Tags

  1. affect
  2. affective computing
  3. artificial social intelligence
  4. emotion
  5. human-centered AI
  6. social signals

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICMI '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)323
  • Downloads (Last 6 weeks)50
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Automatic mild cognitive impairment estimation from the group conversation of coimagination methodProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685754(355-360)Online publication date: 4-Nov-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media