research-article

Chatbot or Chat-Blocker: Predicting Chatbot Popularity before Deployment

Authors:

Rama AkkirajuAuthors Info & Claims

DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems Conference

Pages 1458 - 1469

https://doi.org/10.1145/3461778.3462147

Published: 28 June 2021 Publication History

Abstract

Chatbots are widely employed in various scenarios. However, given the high costs of chatbot development and chatbots’ tremendous social influence, chatbot failures may inevitably lead to a huge economic loss. Previous chatbot evaluation frameworks rely heavily on human evaluation, lending little support for automatic early-stage chatbot examination prior to deployment. To reduce the risk of potential loss, we propose a computational approach to extracting features and training models that make a priori prediction about chatbots’ popularity, which indicates chatbot general performance. The features we extract cover chatbot Intent, Conversation Flow, and Response Design. We studied 1050 customer service chatbots on one of the most popular chatbot service platforms. Our model achieves 77.36% prediction accuracy among very popular and very unpopular chatbots, making the first step towards computational feedback before chatbot deployment. Our evaluation results also reveal the key design features associated with chatbot popularity and offer guidance on chatbot design.

References

[1]

2016. New DigitasLBi Research Shows More than 1 in 3 Americans are willing to make Purchases via Chatbots. digitas.com.

[2]

2017. Enterprises by business size. data.oecd.org.

[3]

2019. The Ultimate Guide for chatbot conversation flow. rakebots.com.

[4]

2020. 20+ Metrics for Chatbot Analytics in 2020: The Ultimate Guide. research.aimultiple.com.

[5]

2021. 8 Epic Chatbot/Conversational Bot Failures. research.aimultiple.com.

[6]

Hadeel Al-Zubaide and Ayman A Issa. 2011. Ontbot: Ontology based chatbot. In International Symposium on Innovations in Information and Communications Technology. IEEE, 7–12. https://doi.org/10.1109/ISIICT.2011.6149594

[7]

Zahra Ashktorab, Mohit Jain, Q Vera Liao, and Justin D Weisz. 2019. Resilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 254. https://doi.org/10.1145/3290605.3300484

Digital Library

[8]

Paul Boutin. 2017. Why Most Chatbots Fail. chatbotsmagazine.com.

[9]

Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. 2010. The balanced accuracy and its posterior distribution. In 2010 20th International Conference on Pattern Recognition. IEEE, 3121–3124. https://doi.org/10.1109/ICPR.2010.764

Digital Library

[10]

Heloisa Candello, Claudio Pinhanez, and Flavio Figueiredo. 2017. Typefaces and the perception of humanness in natural language chatbots. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3476–3487. https://doi.org/10.1145/3025453.3025919

Digital Library

[11]

Heloisa Candello, Claudio Pinhanez, Mauro Pichiliani, Paulo Cavalin, Flavio Figueiredo, Marisa Vasconcelos, and Haylla Do Carmo. 2019. The Effect of Audiences on the User Experience with Conversational Interfaces in Physical Spaces. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 90. https://doi.org/10.1145/3290605.3300320

Digital Library

[12]

Mark Cirillo. 2019. 6 Examples of SMART Chatbot to Improve Your Customer Services. leadsbridge.com.

[13]

David Coniam. 2014. The linguistic accuracy of chatbots: usability from an ESL perspective. Text & Talk 34, 5 (2014), 545–567. https://doi.org/10.1515/text-2014-0018

[14]

Menal Dahiya. 2017. A tool of conversation: Chatbot. International Journal of Computer Sciences and Engineering 5, 5(2017), 158–161.

[15]

Edgar Dale and Jeanne S Chall. 1948. A formula for predicting readability: Instructions. Educational research bulletin(1948), 37–54.

[16]

Christopher Dossman. 2019. AI Scholar: Chatbots that improve after deployment. towardsdatascience.com.

[17]

Jon Doyle. 1992. Rationality and its roles in reasoning. Computational Intelligence 8, 2 (1992), 376–409. https://doi.org/10.1111/j.1467-8640.1992.tb00371.x

[18]

Akash Dubey. 2018. Feature Selection Using Random forest. towardsdatascience.com.

[19]

Ahmed Fadhil, Gianluca Schiavo, Yunlong Wang, and Bereket A Yilma. 2018. The effect of emojis when interacting with conversational interface assisted health coaching system. In Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare. ACM, 378–383. https://doi.org/10.1145/3240925.3240965

Digital Library

[20]

Ethan Fast, Binbin Chen, and Michael S Bernstein. 2016. Empath: Understanding topic signals in large-scale text. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4647–4657. https://doi.org/10.1145/2858036.2858535

Digital Library

[21]

Ong Sing Goh, Cemal Ardil, Wilson Wong, and Chun Che Fung. 2007. A black-box approach for response quality evaluation of conversational agent systems. International Journal of Computational Intelligence 3, 3(2007), 195–203. https://doi.org/10.5281/zenodo.1076854

[22]

Jonathan Grudin and Richard Jacques. 2019. Chatbots, Humbots, and the Quest for Artificial General Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 209. https://doi.org/10.1145/3290605.3300439

Digital Library

[23]

Robert Gunning. 1969. The fog index after twenty years. Journal of Business Communication 6, 2 (1969), 3–13.

[24]

Lakisha Hall. 2018. 6 steps to successful conversational design. ibm.com.

[25]

Chan Chun Ho, Ho Lam Lee, Wing Kwan Lo, and Kwok Fai Andrew Lui. 2018. Developing a chatbot for college student programme advisement. In 2018 International Symposium on Educational Technology (ISET). IEEE, 52–56. https://doi.org/10.1109/ISET.2018.00021

[26]

HN Io and CB Lee. 2017. Chatbots and conversational agents: A bibliometric analysis. In 2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM). IEEE, 215–219. https://doi.org/10.1109/IEEM.2017.8289883

[27]

Mohammed Kaleem, Omar Alobadi, James O’Shea, and Keeley Crockett. 2016. Framework for the formulation of metrics for conversational agent evaluation. In RE-WOCHAT: Workshop on Collecting and Generating Resources for Chatbots and Conversational Agents-Development and Evaluation Workshop Programme (May 28 th, 2016). 20.

[28]

Ralph L Keeney and Howard Raiffa. 1993. Decisions with multiple objectives: preferences and value trade-offs. Cambridge university press. https://doi.org/10.1017/CBO9781139174084

[29]

Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 86. https://doi.org/10.1145/3290605.3300316

Digital Library

[30]

J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. (1975).

[31]

Karolina Kuligowska. 2015. Commercial chatbot: Performance evaluation, usability metrics and quality standards of embodied conversational agents. Professionals Center for Business Research 2 (2015).

[32]

Justin Lee. 2018. The practical guide to chatbot metrics and analytics. blog.growthbot.org.

[33]

Michael McTear. 2018. Conversation modelling for chatbots: current approaches and future directions. Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2018 (2018), 175–185.

[34]

M Meira and AM de P Canuto. 2015. Evaluation of emotional agents’ architectures: an approach based on quality metrics and the influence of emotions on users. In Proceedings of the world congress on engineering, Vol. 1. 1–8.

[35]

Isaac Oswalt. 2017. What Will a Chatbot Cost Me - And Is It Worth It?21handshake.com.

[36]

David J Pasta. 2009. Learning when to be discrete: continuous vs. categorical predictors. In SAS Global Forum, Vol. 248.

[37]

James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates 71, 2001 (2001), 2001.

[38]

Juanan Pereira and Oscar Díaz. 2018. A quality analysis of facebook messenger’s most popular chatbots. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing. ACM, 2144–2150. https://doi.org/10.1145/3167132.3167362

Digital Library

[39]

Nicole M Radziwill and Morgan C Benton. 2017. Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579(2017).

[40]

Prateek Saxena. 2018. How Much Does it Cost to Develop A Chatbot. appinventiv.com.

[41]

Joao Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison-Burch. 2019. Chateval: A tool for chatbot evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). 60–65. https://doi.org/10.18653/v1/N19-4011

[42]

RJ Senter and Edgar A Smith. 1967. Automated readability index. Technical Report. CINCINNATI UNIV OH.

[43]

Bayan Abu Shawar and Eric Atwell. 2007. Chatbots: are they really useful?. In Ldv forum, Vol. 22. 29–49.

[44]

Bayan Abu Shawar and Eric Atwell. 2007. Different measurements metrics to evaluate a chatbot system. In Proceedings of the workshop on bridging the gap: Academic and industrial research in dialog technologies. Association for Computational Linguistics, 89–96.

Digital Library

[45]

Heung-Yeung Shum, Xiao-dong He, and Di Li. 2018. From Eliza to XiaoIce: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering 19, 1(2018), 10–26.

[46]

Emma VA Sylvester, Paul Bentzen, Ian R Bradbury, Marie Clément, Jon Pearce, John Horne, and Robert G Beiko. 2018. Applications of random forest feature selection for fine-scale genetic population assignment. Evolutionary applications 11, 2 (2018), 153–165. https://doi.org/10.1111/eva.12524

[47]

Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, 2018. On evaluating and comparing conversational agents. arXiv preprint arXiv:1801.03625 4 (2018), 60–68.

[48]

Michael Vetter. 2002. Quality aspects of bots. In Software quality and software testing in internet times. Springer, 165–184. https://doi.org/10.1007/978-3-642-56333-1_11

[49]

Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. PARADISE: A Framework for Evaluating Spoken Dialogue Agents. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics. 271–280. https://doi.org/10.3115/976909.979652

Digital Library

[50]

Joseph Weizenbaum. 1966. ELIZA—a computer program for the study of natural language communication between man and machine. Commun. ACM 9, 1 (1966), 36–45. https://doi.org/10.1145/365153.365168

Digital Library

[51]

Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A new chatbot for customer service on social media. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3506–3510. https://doi.org/10.1145/3025453.3025496

Digital Library

[52]

Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020. The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics 46, 1 (2020), 53–93. https://doi.org/10.1162/coli_a_00368

Digital Library

[53]

Michelle X Zhou, Gloria Mark, Jingyi Li, and Huahai Yang. 2019. Trusting Virtual Agents: The Effect of Personality. ACM Transactions on Interactive Intelligent Systems (TiiS) 9, 2-3(2019), 10. https://doi.org/10.1145/3232077

Digital Library

[54]

Rui Zhou, Jasmine Hentschel, and Neha Kumar. 2017. Goodbye text, hello emoji: mobile communication on wechat in China. In Proceedings of the 2017 CHI conference on human factors in computing systems. ACM, 748–759. https://doi.org/10.1145/3025453.3025800

Digital Library

Cited By

Guo Y(2024)The Current Research Status of Normal Chatbots and Government ChatbotsDigital Government and Public Interaction10.4018/979-8-3693-3665-6.ch003(63-88)Online publication date: 13-Sep-2024
https://doi.org/10.4018/979-8-3693-3665-6.ch003
Mo WSingh AHolloway C(2024)From Information Seeking to Empowerment: Using Large Language Model Chatbot in Supporting Wheelchair Life in Low Resource SettingsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675609(1-18)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675609
Rese ATränkner P(2024)Perceived conversational ability of task-based chatbots – Which conversational elements influence the success of text-based dialogues?International Journal of Information Management: The Journal for Information Professionals10.1016/j.ijinfomgt.2023.10269974:COnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.ijinfomgt.2023.102699
Show More Cited By

Recommendations

Chatbot with Touch and Graphics: An Interaction of Users for Emotional Expression and Turn-taking
CUI '20: Proceedings of the 2nd Conference on Conversational User Interfaces

Use of chatbots for emotional exchange is recently increasing in various domains. However, as existing chatbots have been considered in terms of natural language processing techniques for interaction with text-based chatting, chatbot interaction with ...
Multi-platform Chatbot Modeling and Deployment with the Jarvis Framework
Advanced Information Systems Engineering
Abstract
Chatbot applications are increasingly adopted in various domains such as e-commerce or customer services as a direct communication channel between companies and end-users. Multiple frameworks have been developed to ease their definition and ...
E-commerce Distributed Chatbot System
BCI'19: Proceedings of the 9th Balkan Conference on Informatics

Chatbot system attract huge interest in the recent years in many different fields in an attempt to increase the efficiency and shortens the business process execution time replacing the human-human communication with a human-machine conversations and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems Conference

June 2021

2082 pages

ISBN:9781450384766

DOI:10.1145/3461778

Editors:
Wendy Ju
Jacobs Technion-Cornell Institute at Cornell Tech
,
Lora Oehlberg
University of Calgary
,
Sean Follmer
Stanford University
,
Sarah Fox
Carnegie Mellon University
,
Stacey Kuznetsov
Arizona State University

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

DIS '21

Sponsor:

SIGCHI

DIS '21: Designing Interactive Systems Conference 2021

June 28 - July 2, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
755
Total Downloads

Downloads (Last 12 months)185
Downloads (Last 6 weeks)12

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Guo Y(2024)The Current Research Status of Normal Chatbots and Government ChatbotsDigital Government and Public Interaction10.4018/979-8-3693-3665-6.ch003(63-88)Online publication date: 13-Sep-2024
https://doi.org/10.4018/979-8-3693-3665-6.ch003
Mo WSingh AHolloway C(2024)From Information Seeking to Empowerment: Using Large Language Model Chatbot in Supporting Wheelchair Life in Low Resource SettingsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675609(1-18)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675609
Rese ATränkner P(2024)Perceived conversational ability of task-based chatbots – Which conversational elements influence the success of text-based dialogues?International Journal of Information Management: The Journal for Information Professionals10.1016/j.ijinfomgt.2023.10269974:COnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.ijinfomgt.2023.102699
Milana FCostanza EFischer J(2023)Chatbots as Advisers: the Effects of Response Variability and Reply Suggestion ButtonsProceedings of the 5th International Conference on Conversational User Interfaces10.1145/3571884.3597132(1-10)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3571884.3597132
Schloß DGnewuch UMaedche A(2023)Chatbot Analytics mittels BetriebsdatenRobotik in der Wirtschaftsinformatik10.1007/978-3-658-39621-3_9(167-192)Online publication date: 1-Aug-2023
https://doi.org/10.1007/978-3-658-39621-3_9
Kilani NRajaobelina L(2022)Impact of Live Chat Service Quality on Behavioral Intentions and Relationship Quality: A Meta-AnalysisInternational Journal of Human–Computer Interaction10.1080/10447318.2022.214412640:7(1558-1585)Online publication date: 22-Nov-2022
https://doi.org/10.1080/10447318.2022.2144126

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents