demonstration

Perceived Usefulness of Conversational Agents Predicts Search Performance in Aerospace Domain

Authors:

Ying-Hsang LiuAuthors Info & Claims

CUI '20: Proceedings of the 2nd Conference on Conversational User Interfaces

Article No.: 57, Pages 1 - 3

https://doi.org/10.1145/3405755.3406172

Published: 22 July 2020 Publication History

Get Access

Abstract

This paper presents a user-centered approach to the design and evaluation of conversational search user interfaces to support the pilot in cockpits. Our findings of a controlled user experiment suggest that user perception of the usefulness of the system in completing the search task and the system's responses to the relevance of the topic are good predictors of search performance. User satisfaction with the system's responses may not be a good predictor of search performance in the context of safety of life scenarios such as cockpit procedures for pilots.

References

[1]

Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019. Asking Clarifying Questions in Open-Domain Information-Seeking Conversations. In Proceedings of the ACM SIGIR Conference (SIGIR '19). ACM, New York, 475--484. https://doi.org/10.1145/3331184.3331265

Digital Library

Google Scholar

[2]

Alexandre Arnold, Gérard Dupont, Catherine Kobus, and François Lancelot. 2019. Conversational agent for aerospace question answering: A position paper. In Proceedings of the 1st Workshop on Conversational Interaction Systems (WCIS at SIGIR). Paris. https://drive.google.com/file/d/1yEw4Kj9c04PScdnaHkwPTiACBJ8AL-G3/view

Google Scholar

[3]

Tom Bocklisch, Joey Faulkner, Nick Pawlowski, and Alan Nichol. 2017. Rasa: Open Source Language Understanding and Dialogue Management. arXiv:arXiv:1712.05181

Google Scholar

[4]

Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What Do You Mean Exactly?: Analyzing Clarification Questions in CQA. In Proceedings of the ACM CHIIR Conference (CHIIR '17). ACM, New York, 345--348. https://doi.org/10.1145/3020165.3022149

Digital Library

Google Scholar

[5]

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada. https://doi.org/10.18653/v1/p17--1171

Crossref

Google Scholar

[6]

Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2020. The Second Conversational Intelligence Challenge (ConvAI2).In The NeurIPS '18 Competition: From Machine Learning to Intelligent Conversations, Sergio Escalera and Ralf Herbrich (Eds.). Springer International Publishing, Cham, 187--208. https://doi.org/10.1007/978--3-030-29135-8_7

Crossref

Google Scholar

[7]

Roger E. Kirk. 2013. Experimental Design: Procedures for the Behavioral Sciences (4th ed.). Brooks/Cole, Pacific Grove, CA.

Crossref

Google Scholar

[8]

Chen Qu, Liu Yang, W Bruce Croft, Yongfeng Zhang, Johanne R Trippas, and Minghui Qiu. 2019. User Intent Prediction in Information-seeking Conversations. In Proceedings of the ACM CHIIR Conference (CHIIR '19). ACM, New York, 25--33. https://doi.org/10.1145/3295750.3298924

Digital Library

Google Scholar

[9]

Filip Radlinski and Nick Craswell. 2017. A Theoretical Framework for Conversational Search. In Proceedings of the ACM CHIIR Conference (CHIIR '17). ACM, New York, 117--126. https://doi.org/10.1145/3020165.3020183

Digital Library

Google Scholar

[10]

Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr. 3, 4 (2009), 333--389. https://doi.org/10.1561/1500000019

Digital Library

Google Scholar

[11]

Doug Turnbull and John Berryman. 2016. Relevant Search: With Applications for Solr and Elasticsearch. Manning Publications, Shelter Island, NY.

Google Scholar

Cited By

View all

Buschmeyer KHatfield SZenner J(2023)Psychological assessment of AI-based decision support systems: tool development and expected benefitsFrontiers in Artificial Intelligence10.3389/frai.2023.12493226Online publication date: 25-Sep-2023
https://doi.org/10.3389/frai.2023.1249322
Liu YArnold ADupont GKobus CLancelot FGranger GRouillard YDuchevet AImbert JMatton N(2023)User Evaluation of Conversational Agents for Aerospace DomainInternational Journal of Human–Computer Interaction10.1080/10447318.2023.223954440:19(5549-5568)Online publication date: 2-Aug-2023
https://doi.org/10.1080/10447318.2023.2239544
Gosper STrippas JRichards HAllison FSear CKhorasani SMattioli F(2021)Understanding the Utility of Digital Flight Assistants: A Preliminary AnalysisProceedings of the 3rd Conference on Conversational User Interfaces10.1145/3469595.3469627(1-5)Online publication date: 27-Jul-2021
https://dl.acm.org/doi/10.1145/3469595.3469627
Show More Cited By

Index Terms

Perceived Usefulness of Conversational Agents Predicts Search Performance in Aerospace Domain
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. Laboratory experiments
    2. Interaction paradigms
      1. Natural language interfaces
2. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
      1. Search interfaces

Recommendations

The Mediating of Perceived Usefulness and Perceived Ease of Use: The Case of Mobile Banking in Yemen

While there are a wide range of business opportunities available via mobile technologies, mobile banking services have not been widely accepted by bank clients in Yemen. This article aims to test the mediation effect of TAM core constructs between the ...
Enhancing user experience with conversational agent for movie recommendation

This study investigates how user satisfaction and intention to use for an interactive movie recommendation system is determined by communication variables and relationship between conversational agent and user. By adopting the Computers-Are-Social-...
Exploring Effects of Conversational Fillers on User Perception of Conversational Agents
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Through technological advancements in various areas of our lives, Conversational Agents progressed in their human-likeness. In the field of HCI, however, the use of conversational fillers (e.g., "um," "uh," etc.) by Conversational Agents have not been ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

CUI '20: Proceedings of the 2nd Conference on Conversational User Interfaces

July 2020

271 pages

ISBN:9781450375443

DOI:10.1145/3405755

Conference Chairs:
María Inés Torres,
Stephan Schlögl,
Program Chairs:
Leigh Clark,
Martin Porcheron

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 July 2020

Check for updates

Author Tags

Qualifiers

Demonstration
Research
Refereed limited

Conference

CUI '20

CUI '20: 2nd Conference on Conversational User Interfaces

July 22 - 24, 2020

Bilbao, Spain

Acceptance Rates

CUI '20 Paper Acceptance Rate 13 of 39 submissions, 33%;

Overall Acceptance Rate 34 of 100 submissions, 34%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
139
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Buschmeyer KHatfield SZenner J(2023)Psychological assessment of AI-based decision support systems: tool development and expected benefitsFrontiers in Artificial Intelligence10.3389/frai.2023.12493226Online publication date: 25-Sep-2023
https://doi.org/10.3389/frai.2023.1249322
Liu YArnold ADupont GKobus CLancelot FGranger GRouillard YDuchevet AImbert JMatton N(2023)User Evaluation of Conversational Agents for Aerospace DomainInternational Journal of Human–Computer Interaction10.1080/10447318.2023.223954440:19(5549-5568)Online publication date: 2-Aug-2023
https://doi.org/10.1080/10447318.2023.2239544
Gosper STrippas JRichards HAllison FSear CKhorasani SMattioli F(2021)Understanding the Utility of Digital Flight Assistants: A Preliminary AnalysisProceedings of the 3rd Conference on Conversational User Interfaces10.1145/3469595.3469627(1-5)Online publication date: 27-Jul-2021
https://dl.acm.org/doi/10.1145/3469595.3469627
Wu DHuang Y(2021)Why Do You Trust Siri?Proceedings of the Association for Information Science and Technology10.1002/pra2.46458:1(366-379)Online publication date: 13-Oct-2021

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

The Mediating of Perceived Usefulness and Perceived Ease of Use: The Case of Mobile Banking in Yemen

Enhancing user experience with conversational agent for movie recommendation

Exploring Effects of Conversational Fillers on User Perception of Conversational Agents