A First Step Towards Considering Nonverbal Communicative Cues in Dialog Systems
Pages 125 - 129
Abstract
Currently, spoken dialogue systems mostly consider only voice interactions and do not consider nonverbal cues. This work aims to explore the domain of interaction with conversational assistants, taking into account not only verbal cues, but also the importance of integrating nonverbal aspects to make interactions between humans and agents more natural, richer, and more efficient. This work presents an initial proof-of-concept for the modules needed for dialogue systems to take into account verbal and nonverbal inputs, as well as their articulation, demonstrating their viability. To this end, we use a fusion module that can directly or indirectly affect the communicative acts, depending on the verbal and nonverbal communication inputs. Preliminary results demonstrate the implemented system behaviour of how verbal content is affected by nonverbal cues.
References
[1]
Nuno Almeida, António Teixeira, Samuel Silva, and Maksym Ketsmur. 2019. The AM4I architecture and framework for multimodal interaction and its application to smart environments. Sensors 19, 11 (2019), 2587.
[2]
P.A. Andersen. 2008. Nonverbal Communication: Forms and Functions. Waveland Press Incorporated. https://books.google.pt/books?id=Np0oSgAACAAJ
[3]
Richard A Bolt. 1980. “Put-that-there” Voice and gesture at the graphics interface. In Proceedings of the 7th annual conference on Computer graphics and interactive techniques. 262–270.
[4]
Justine Cassell. 2000. Embodied conversational interface agents. Commun. ACM 43, 4 (2000), 70–78.
[5]
Daniel Chandler and Rod Munday. 2011. Verbal Communication. https://doi.org/10.1093/acref/9780199568758.013.2910
[6]
Deborah A Dahl. 2013. The W3C multimodal architecture and interfaces standard. Journal on Multimodal User Interfaces 7, 3 (2013), 171–182.
[7]
Stephan Diederich, Alfred Benedikt Brendel, Stefan Morana, and Lutz Kolbe. 2022. On the design of and interaction with conversational agents: An organizing and assessing review of human-computer interaction research. Journal of the Association for Information Systems 23, 1 (2022), 96–138.
[8]
Manaal Faruqui and Dilek Hakkani-Tür. 2022. Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems. Computational Linguistics 48, 1 (2022), 221–232.
[9]
Jasper Feine, Ulrich Gnewuch, Stefan Morana, and Alexander Maedche. 2019. A taxonomy of social cues for conversational agents. International Journal of Human-Computer Studies 132 (2019), 138–161.
[10]
Lucie Galland, Catherine Pelachaud, and Florian Pecune. 2022. Adapting conversational strategies in information-giving human-agent interaction. Frontiers in Artificial Intelligence 5 (2022), 1029340.
[11]
Judith A Hall, Terrence G Horgan, and Nora A Murphy. 2019. Nonverbal communication. Annual review of psychology 70 (2019), 271–294.
[12]
Jia-Hao Hsu, Ming-Hsiang Su, Chung-Hsien Wu, and Yi-Hsuan Chen. 2021. Speech emotion recognition considering nonverbal vocalization in affective conversations. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021), 1675–1686.
[13]
Kristina Jokinen and Michael McTear. 2022. Spoken dialogue systems. Springer Nature.
[14]
Shun Katada, Shogo Okada, Yuki Hirano, and Kazunori Komatani. 2020. Is she truly enjoying the conversation? analysis of physiological signals toward adaptive dialogue systems. In Proceedings of the 2020 International Conference on Multimodal Interaction. 315–323.
[15]
Jina Lee and Stacy Marsella. 2006. Nonverbal behavior generator for embodied conversational agents. In International Workshop on Intelligent Virtual Agents. Springer, 243–255.
[16]
Nikolaos Mavridis. 2015. A review of verbal and non-verbal human–robot interactive communication. Robotics and Autonomous Systems 63 (2015), 22–35.
[17]
Michael McTear. 2022. Conversational ai: Dialogue systems, conversational agents, and chatbots. Springer Nature.
[18]
Albert Mehrabian and Susan R Ferris. 1967. Inference of attitudes from nonverbal communication in two channels.Journal of consulting psychology 31, 3 (1967), 248.
[19]
Quim Motger, Xavier Franch, and Jordi Marco. 2022. Software-based dialogue systems: survey, taxonomy, and challenges. Comput. Surveys 55, 5 (2022), 1–42.
[20]
Merle M Reimann, Florian A Kunneman, Catharine Oertel, and Koen V Hindriks. 2023. A Survey on Dialogue Management in Human-Robot Interaction. arXiv preprint arXiv:2307.10897 (2023).
[21]
Katie Seaborn, Norihisa P Miyake, Peter Pennefather, and Mihoko Otake-Matsuura. 2021. Voice in human–agent interaction: A survey. ACM Computing Surveys (CSUR) 54, 4 (2021), 1–43.
[22]
Emiliana R Simon-Thomas, Dacher J Keltner, Disa Sauter, Lara Sinicropi-Yao, and Anna Abramson. 2009. The voice conveys specific emotions: evidence from vocal burst displays.Emotion 9, 6 (2009), 838.
[23]
Timo Spring, Jacky Casas, Karl Daher, Elena Mugellini, and Omar Abou Khaled. 2019. Empathic response generation in chatbots. In Proceedings of 4th Swiss Text Analytics Conference (SwissText 2019), 18-19 June 2019, Wintherthur, Switzerland. 18-19 June 2019.
[24]
Sheila Steinberg. 2007. An introduction to communication studies. Juta and Company Ltd.
[25]
Lynn H Turner and Richard West. 2018. An introduction to communication. Cambridge University Press.
[26]
Jacqueline Urakami and Katie Seaborn. 2023. Nonverbal Cues in Human–Robot Interaction: A Communication Studies Perspective. ACM Transactions on Human-Robot Interaction 12, 2 (2023), 1–21.
[27]
Michelle ME Van Pinxteren, Mark Pluymaekers, and Jos GAM Lemmink. 2020. Human-like communication in conversational agents: a literature review and research agenda. Journal of Service Management 31, 2 (2020), 203–225.
[28]
Kai Vogeley and Gary Bente. 2010. “Artificial humans”: Psychology and neuroscience perspectives on embodiment and nonverbal communication. Neural Networks 23, 8-9 (2010), 1077–1090.
[29]
Ruchen Wen, Alyssa Hanson, Zhao Han, and Tom Williams. 2023. Fresh Start: Encouraging Politeness in Wakeword-Driven Human-Robot Interaction. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. 112–121.
[30]
Jiaqi Xu, Cheng Luo, Weicheng Xie, Linlin Shen, Xiaofeng Liu, Lu Liu, Hatice Gunes, and Siyang Song. 2023. MRecGen: Multimodal Appropriate Reaction Generator. arXiv preprint arXiv:2307.02609 (2023).
[31]
Steve Young. 2021. Hey Cyba: The Inner Workings of a Virtual Personal Assistant. Cambridge University Press.
[32]
Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proc. IEEE 101, 5 (2013), 1160–1179.
[33]
Wen Zhiyuan, Cao Jiannong, Yang Ruosong, Liu Shuaiqi, and Shen Jiaxing. 2021. Automatically select emotion for response via personality-affected emotion transition. arXiv preprint arXiv:2106.15846 (2021).
[34]
Chi Zhou, Tengyue Bian, and Kang Chen. 2022. Gesturemaster: Graph-based speech-driven gesture generation. In Proceedings of the 2022 International Conference on Multimodal Interaction. 764–770.
Index Terms
- A First Step Towards Considering Nonverbal Communicative Cues in Dialog Systems
Recommendations
Authoring Communicative Behaviors for Situated, Embodied Characters
ICMI '14: Proceedings of the 16th International Conference on Multimodal InteractionEmbodied conversational agents hold great potential as multimodal interfaces due to their ability to communicate naturally using speech and nonverbal cues. The goal of my research is to enable animators and designers to endow ECAs with interactive ...
Enabling Embodied Conversational Agents to Respond to Nonverbal Behavior of the Communication Partner
Human-Computer Interaction. User Experience and BehaviorAbstractHumans communicate on three levels: words, paralanguage, and nonverbal. While conversational agents focus mainly on the interpretation of words that are being spoken, recently the focus has also shifted to how we say those words with our tone, ...
Towards Naturally Responsive Spoken Dialog Systems by Modelling Pragmatic-Prosody Correlations of Discourse Markers
IUI '20 Companion: Companion Proceedings of the 25th International Conference on Intelligent User InterfacesHuman speakers in a dialog adapt their responses and the way they convey them to their interlocutors by appropriately tuning their prosody, taking into account the context in which the dialog takes place. Today's spoken dialog systems are incapable of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2024
708 pages
ISBN:9798400717604
DOI:10.1145/3652037
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 June 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Funding Sources
- Fundação para a Ciência e a Tecnologia
- Institute of Electronics and Informatics Engineering of Aveiro
Conference
PETRA '24
PETRA '24: The PErvasive Technologies Related to Assistive Environments Conference
June 26 - 28, 2024
Crete, Greece
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 94Total Downloads
- Downloads (Last 12 months)94
- Downloads (Last 6 weeks)13
Reflects downloads up to 16 Nov 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in