Abstract
This paper presents Zenon, an affective, multi-modal conversational agent (chatbot) specifically designed for treatment of brain diseases like multiple sclerosis and stroke. Zenon collects information from patients in a non-intrusive way and records user sentiment using two different modalities: text and video. A user-friendly interface is designed to meet users’ needs and achieve an efficient conversation flow. What makes Zenon unique is the support of multiple languages, the combination of two information sources for tracking sentiment, and the deployment of a semantic knowledge graph that ensures machine-interpretable information exchange.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
References
Agathangelou, P., Katakis, I.: Balancing between holistic and cumulative sentiment classification. Online Soc. Netw. Media 29, 100199 (2022)
Alnashwan, R., Sorensen, H., O’Riordan, A., Hoare, C.: Multiclass sentiment classification of online health forums using both domain-independent and domain-specific features. In: Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, Austin, Texas, USA, pp. 75–83. ACM (2017)
Amato, F., Marrone, S., Moscato, V., Piantadosi, G., Picariello, A., Sansone, C.: Chatbots meet ehealth: automatizing healthcare. In: WAIAH@AI*IA (2017)
Anastasiadou, M., Alexiadis, A., Polychronidou, E., Votis, K., Tzovaras, D.: A prototype educational virtual assistant for diabetes management. In: 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), Cincinnati, OH, USA, pp. 999–1004. IEEE (2020)
Bagherzadeh, P., Bergler, S.: Multi-input recurrent independent mechanisms for leveraging knowledge sources: case studies on sentiment analysis and health text mining. In: Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 108–118. Association for Computational Linguistics (2021)
Cai, Y., Zheng, W., Zhang, T., Li, Q., Cui, Z., Ye, J.: Video based emotion recognition using CNN and BRNN. In: Tan, T., Li, X., Chen, X., Zhou, J., Yang, J., Cheng, H. (eds.) CCPR 2016. CCIS, vol. 663, pp. 679–691. Springer, Singapore (2016). https://doi.org/10.1007/978-981-10-3005-5_56
Chang, F.J., Tran, A.T., Hassner, T., Masi, I., Nevatia, R., Medioni, G.: Expnet: landmark-free, deep, 3D facial expressions (2018)
Cheatham, M., Pesquita, C.: Semantic data integration. In: Zomaya, A.Y., Sakr, S. (eds.) Handbook of Big Data Technologies, pp. 263–305. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-49340-4_8
Clark, L., et al.: What makes a good conversation? Challenges in designing truly conversational agents. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2019)
World Wide Web Consortium: SPARQL 1.1 overview (2013)
World Wide Web Consortium: RDF 1.1 concepts and abstract syntax (2014)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database, pp. 248–255 (2009)
Diederich, S., Brendel, A., Morana, S., Kolbe, L.: On the design of and interaction with conversational agents: an organizing and assessing review of human-computer interaction research. J. Assoc. Inf. Syst. 23(1), 96–138 (2022)
Fan, Y., Lu, X., Li, D., Liu, Y.: Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 445–450 (2016)
Gogate, U., Parate, A., Sah, S., Narayanan, S.: Real time emotion recognition and gender classification, pp. 138–143 (2020)
OWL Working Group: OWL 2 web ontology language document overview: W3C recommendation 27 October 2009 (2009)
Guo, Y., Wang, J., Wu, R., Li, Z., Sun, L.: Designing for trust: a set of design principles to increase trust in chatbot. CCF Trans. Pervasive Comput. Interact. 1–8 (2022)
Hommersom, A., et al.: MoSHCA - my mobile and smart health care assistant. In: 2013 IEEE 15th International Conference on e-Health Networking, Applications and Services (Healthcom 2013), Lisbon, Portugal, pp. 188–192. IEEE (2013)
Hossain, M.S., Muhammad, G.: An audio-visual emotion recognition system using deep learning fusion for a cognitive wireless framework. IEEE Wirel. Commun. 26(3), 62–68 (2019)
Hossain, M.S., Muhammad, G.: Emotion recognition using deep learning approach from audio-visual emotional big data. Inf. Fusion 49, 69–78 (2019)
Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv (2017)
Hu, Y., Qu, Y., Maus, A., Mutlu, B.: Polite or direct? Conversation design of a smart display for older adults based on politeness theory. In: CHI Conference on Human Factors in Computing Systems, pp. 1–15 (2022)
Jelodar, H., Wang, Y., Orji, R., Huang, S.: Deep sentiment classification and topic discovery on novel coronavirus or COVID-19 online discussions: NLP using LSTM recurrent neural network approach. IEEE J. Biomed. Health Inform. 24(10), 2733–2742 (2020)
Laranjo, L., et al.: Conversational agents in healthcare: a systematic review. J. Am. Med. Inform. Assoc. 25(9), 1248–1258 (2018)
Maga-Nteve, C., et al.: A semantic technologies toolkit for bridging early diagnosis and treatment in brain diseases: report from the ongoing EU-funded research project ALAMEDA. In: Garoufallou, E., Ovalle-Perandones, M.-A., Vlachidis, A. (eds.) MTSR 2021. CCIS, vol. 1537, pp. 349–354. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98876-0_30
Oh, K.J., Lee, D., Ko, B., Choi, H.J.: A chatbot for psychiatric counseling in mental healthcare service based on emotional dialogue analysis and sentence generation. In: 2017 18th IEEE International Conference on Mobile Data Management (MDM), Daejeon, South Korea, pp. 371–375. IEEE (2017)
Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.: Mobilenetv 2: inverted residuals and linear bottlenecks. In: CVPR, pp. 4510–4520. Computer Vision Foundation/IEEE Computer Society (2018)
Teixeira, M.S., Maran, V., Dragoni, M.: The interplay of a conversational ontology and AI planning for health dialogue management. In: Proceedings of the 36th Annual ACM Symposium on Applied Computing, Virtual Event Republic of Korea, pp. 611–619. ACM (2021)
Wafa, H., Wolfe, C., Emmett, E., Roth, G., Johnson, C., Wang, Y.: Burden of stroke in Europe: thirty-year projections of incidence, prevalence, deaths, and disability-adjusted life years. Stroke 51, 2418–2427 (2020)
Walker, T., et al.: Developing an intelligent virtual agent to stratify people with cognitive complaints: a comparison of human-patient and intelligent virtual agent-patient interaction. Dementia 19, 1173–1188 (2018)
Acknowledgement
This research received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No GA 101017558 (ALAMEDA).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chira, C. et al. (2023). An Affective Multi-modal Conversational Agent for Non Intrusive Data Collection from Patients with Brain Diseases. In: Følstad, A., et al. Chatbot Research and Design. CONVERSATIONS 2022. Lecture Notes in Computer Science, vol 13815. Springer, Cham. https://doi.org/10.1007/978-3-031-25581-6_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-25581-6_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25580-9
Online ISBN: 978-3-031-25581-6
eBook Packages: Computer ScienceComputer Science (R0)