Abstract
Chatbots have been adopted in the health domain and their number grew during the current COVID-19 pandemic. As a new kind of interface, chatbots combine visual elements with natural conversation. While conversational capabilities of chatbots improve, little attention has been given to the evaluation of the user experience and chatbot usability. This paper presents the results of a heuristic review of 24 COVID-19 chatbots on different channels (webchat vs messengers), for diverse topics (symptom-checker vs FAQ) and with varying interaction styles (visual-centric vs content-centric vs conversation-centric). It proposes a generic evaluation framework with 12 heuristics based on Nielsen’s ten heuristics and adapted to the conversational interface context. The results point at the strengths (immediate feedback, familiar language, consistent wording and visual design) as well as shortcomings (little user control and freedom, missing permanent menu and help options, lack of context understanding and interaction management capabilities) of COVID-19 chatbots. The paper furthermore gives recommendations for chatbot design in similar contexts.
Supported by the Luxembourg National Research Fund (FNR), SLANT, 13320890.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bongard-Blanchy, K., Bouchard, C.: Dimensions of user experience-from the product design perspective (2014)
Brandtzaeg, P.B., Følstad, A.: Why people use chatbots. In: Kompatsiaris, I., et al. (eds.) INSCI 2017. LNCS, vol. 10673, pp. 377–392. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70284-1_30
Brandtzaeg, P.B., Følstad, A.: Chatbots: changing user needs and motivations. Interactions 25(5), 38–43 (2018). https://doi.org/10.1145/3236669
Budiu, R.: The user experience of chatbots. Nielsen Norman Group (2018)
Chaix, B., Delamon, G., Guillemasse, A., Brouard, B., Bibault, J.E.: Psychological distress during the Covid-19 pandemic in France: a national assessment of at-risk populations. medRxiv (2020)
Danilava, S., Busemann, S., Schommer, C., Ziegler, G.: Towards computational models for a long-term interaction with an artificial conversational companion. In: Proceeding of ICAART’2013 (2013)
Fadhil, A., Schiavo, G.: Designing for health chatbots. arXiv preprint arXiv:1902.09022 (2019)
Artificial Companion for Second Language Conversation. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-15504-9_14
Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., McTear, M.: Usability testing of a healthcare chatbot: can we use conventional methods to assess conversational user interfaces? In: Proceedings of ECCE, pp. 207–214 (2019)
Hone, K.S., Graham, R.: Towards a tool for the subjective assessment of speech system interfaces (sassi). Nat. Lang. Eng. 6, 287–303 (2000)
Lallemand, C., Gronier, G.: Méthodes de design UX: 30 méthodes fondamentales pour concevoir des expériences optimales. Eyrolles (2018)
Luger, E., Sellen, A.: Like having a really bad PA: the gulf between user expectation and experience of conversational agents. In: Proceedings CHI’2016, pp. 5286–5297. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2858036.2858288
McTear, M., Callejas, Z., Griol, D.: The Conversational Interface: Talking to Smart Devices. Springer (2016)
Moore, R.J., Arar, R.: Conversational UX Design: A Practitioner’s Guide to the Natural Conversation Framework. ACM Books (2019)
MTI-Singapore: Ministry of trade and industry Singapore, MTI Chat for Biz (2020). https://www.mti.gov.sg/Chatbot/chat
Munsch, N., et al.: A benchmark of online COVID-19 symptom checkers. medRxiv (2020)
Nielsen, J.: Ten usability heuristics (2005). http://www.nngroup.com/articles/ten-usability-heuristics/. Accessed 25 Jun 2020
Semigran, H.L., Linder, J.A., Gidengil, C., Mehrotra, A.: Evaluation of symptom checkers for self diagnosis and triage: audit study. bmj 351, h3480 (2015)
Shariat, J., Saucier, C.S.: Tragic Design: The Impact of Bad Product Design and How to Fix It. O’Reilly, United States (2017)
Shevat, A.: Designing Bots: Creating Conversational Experiences. O’Reilly, United States (2017)
Sugisaki, K., Bleiker, A.: Usability guidelines and evaluation criteria for conversational user interfaces: a heuristic and linguistic approach. In: Proceedings of the Conference on Mensch und Computer, pp. 309–319 (2020)
TARS: Chatbot templates to fight coronavirus (Covid-19) pandemic, June 2020. https://hellotars.com/chatbot-templates/coronavirus-covid19-fight/
Verma, V.: 10 usability heuristics every chatbot company should follow. UX Collective, Medium (2019)
Vijayakumar, B., Höhn, S., Schommer, C.: Quizbot: exploring formative feedback with conversational interfaces. In: Draaijer, S., Joosten-ten Brinke, D., Ras, E. (eds.) TEA 2018. CCIS, vol. 1014, pp. 102–120. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25264-9_8
Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A.: Evaluating spoken dialogue agents with PARADISE: two case studies. Comp. Speech Lang. 12(4), 317–348 (1998)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Höhn, S., Bongard-Blanchy, K. (2021). Heuristic Evaluation of COVID-19 Chatbots. In: Følstad, A., et al. Chatbot Research and Design. CONVERSATIONS 2020. Lecture Notes in Computer Science(), vol 12604. Springer, Cham. https://doi.org/10.1007/978-3-030-68288-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-68288-0_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68287-3
Online ISBN: 978-3-030-68288-0
eBook Packages: Computer ScienceComputer Science (R0)