research-article

Chat with Smart Conversational Agents: How to Evaluate Chat Experience in Smart Home

Authors:

Tian Wu,

Daisong GuanAuthors Info & Claims

MobileHCI '19: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services

Article No.: 60, Pages 1 - 6

https://doi.org/10.1145/3338286.3344408

Published: 01 October 2019 Publication History

Get Access

Abstract

There are more and more smart devices equipped with smart conversational agents, which can engage in chat or free conversation with human. However, the human-machine chat is still in the early stage of development, and there is a lack of effective methods to evaluate chat experience. In this study we proposed an approach to evaluate chat experience with smart conversational agents in smart home. We collected evaluation metrics and applied them in user testing, and then optimized the metrics and constructed an evaluation system. We applied the evaluation system to compare chat experience of five different smart conversational agents.

References

[1]

B Banchs, R. E., and Li, H. IRIS: a chat-oriented dialogue system based on the vector space model. In Proceedings of the Association for Computational Linguistics, System Demonstrations, 2012. 37--42.

Google Scholar

[2]

Stallard D. Talk'n'travel: a conversational system for air travel planning. In Proc. of the Sixth Conference on Applied NLP, 2000. 68--75. https://doi.org/10.3115/974147.974157.

Crossref

Google Scholar

[3]

Yu, Z., Papangelis, A., Rudnicky, A. Ticktock: A non-goal-oriented multimodal dialog system with engagement awareness. In Proceedings of the AAAI Spring Symposium. 2015.108-111.

Google Scholar

[4]

Robin IM Dunbar, Anna Marriott, and Neil DC Duncan. Human conversational behavior. Human nature, 1997. 8(3), 231--246. https://doi.org/10.1007/bf02912493.

Crossref

Google Scholar

[5]

Mor Naaman, Jeffrey Boase, and Chih-Hui Lai. Is it really about me?: message content in social awareness streams. In Proceedings of the 2010 ACM conference on Computer supported cooperative work, 2010. 189--192. https://doi.org/10.1145/1718918.1718953.

Crossref

Google Scholar

[6]

Jeesoo Bang, Hyungjong Noh, Yonghee Kim, and Gary Geunbae Lee. Example-based chat-oriented dialogue system with personalized long- term memory. In Proc. BigComp, 2015. 238--243. https://doi.org/10.1109/35021bigcomp.2015.7072837.

Crossref

Google Scholar

[7]

McTear M. Spoken dialogue technology: enabling the conversational user interface. ACM Computing Surveys. 2002. 34(1), 90--169. https://doi.org/10.1145/505282.505285.

Crossref

Google Scholar

[8]

B. A. Shawar, and E. Atwell. Different measurements metrics to evaluate a chatbot system, Proc. of the 2nd Workshop on TextGraphs: Graph-Based Algorithms for Natural Language Processing. 2007. 89--96. https://doi.org/10.3115/1556328.1556341.

Crossref

Google Scholar

[9]

Gandhe, S., Traum, D.R. Creating spoken dialogue characters from corpora without annotations. In: INTERSPEECH. 2007. 2201--2204.

Google Scholar

[10]

Grice, Paul. Logic and conversation. In Cole, P.; Morgan, J. Syntax and semantics.3: Speech acts. New York: Academic Press. 1975. 41--58. https://doi.org/10.1057/9780230005853_5.

Crossref

Google Scholar

[11]

Leech, G.N. Principles of Pragmatics. London: Longman. 1983.

Google Scholar

Cited By

View all

Sanguinetti MAtzori M(2024)Conversational Agents for Energy Awareness and Efficiency: A SurveyElectronics10.3390/electronics1302040113:2(401)Online publication date: 18-Jan-2024
https://doi.org/10.3390/electronics13020401
Yao YHuang LHe YMa ZXu XMi H(2023)Reviewing and Reflecting on Smart Home Research from the Human-Centered PerspectiveProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580842(1-21)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580842
Li SYuan XZhao XYang S(2023)Leaders or Team-Mates: Exploring the Role-Based Relationship Between Multiple Intelligent Agents in Driving ScenariosHCI in Mobility, Transport, and Automotive Systems10.1007/978-3-031-35678-0_9(144-165)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-35678-0_9
Show More Cited By

Recommendations

How to Evaluate a Good Conversation? An Evaluation Framework for Chat Experience in Smart Home
Human-Computer Interaction. Theory, Methods and Tools
Abstract
With the development of artificial intelligence technology, more and more smart devices equipped with smart conversational agents, which can engage in chat or free conversation with human. However, the human-machine chat is still in the early ...
CHAT for chat

This mixed methods study addresses learning in online chat virtual reference service at a large university library. Cultural-Historical Activity Theory (CHAT) provides the guiding principle and philosophy for the investigation, understanding, and ...
Conversational Agents: Acting on the Wave of Research and Development
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

In the last five years, work on software that interacts with people via typed or spoken natural language, called chatbots, intelligent assistants, social bots, virtual companions, non-human players, and so on, increased dramatically. Chatbots burst into ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

MobileHCI '19: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services

October 2019

646 pages

ISBN:9781450368254

DOI:10.1145/3338286

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

MobileHCI '19

Sponsor:

SIGCHI

MobileHCI '19: 21st International Conference on Human-Computer Interaction with Mobile Devices and Services

October 1 - 4, 2019

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 202 of 906 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
451
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)3

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Sanguinetti MAtzori M(2024)Conversational Agents for Energy Awareness and Efficiency: A SurveyElectronics10.3390/electronics1302040113:2(401)Online publication date: 18-Jan-2024
https://doi.org/10.3390/electronics13020401
Yao YHuang LHe YMa ZXu XMi H(2023)Reviewing and Reflecting on Smart Home Research from the Human-Centered PerspectiveProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580842(1-21)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580842
Li SYuan XZhao XYang S(2023)Leaders or Team-Mates: Exploring the Role-Based Relationship Between Multiple Intelligent Agents in Driving ScenariosHCI in Mobility, Transport, and Automotive Systems10.1007/978-3-031-35678-0_9(144-165)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-35678-0_9
Coppola RArdito L(2021)Quality Assessment Methods for Textual Conversational Interfaces: A Multivocal Literature ReviewInformation10.3390/info1211043712:11(437)Online publication date: 21-Oct-2021
https://doi.org/10.3390/info12110437
Kumar RChauhan DDias GEkbal A(2021)Modelling Personalized Dialogue Generation in Multi-Party Settings2021 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN52387.2021.9534278(1-6)Online publication date: 18-Jul-2021
https://doi.org/10.1109/IJCNN52387.2021.9534278
Humphry JChesher C(2020)Preparing for smart voice assistants: Cultural histories and media innovationsNew Media & Society10.1177/146144482092367923:7(1971-1988)Online publication date: 22-May-2020
https://doi.org/10.1177/1461444820923679
Adel NCrockett KChandran DCarvalho J(2020)Interpreting Human Responses in Dialogue Systems using Fuzzy Semantic Similarity Measures2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)10.1109/FUZZ48607.2020.9177605(1-8)Online publication date: Jul-2020
https://doi.org/10.1109/FUZZ48607.2020.9177605

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

How to Evaluate a Good Conversation? An Evaluation Framework for Chat Experience in Smart Home

CHAT for chat

Conversational Agents: Acting on the Wave of Research and Development