research-article

Open access

Zero-shot Clarifying Question Generation for Conversational Search

Authors:

Qingyao AiAuthors Info & Claims

WWW '23: Proceedings of the ACM Web Conference 2023

Pages 3288 - 3298

https://doi.org/10.1145/3543507.3583420

Published: 30 April 2023 Publication History

All formats PDF

Abstract

A long-standing challenge for search and conversational assistants is query intention detection in ambiguous queries. Asking clarifying questions in conversational search has been widely studied and considered an effective solution to resolve query ambiguity. Existing work have explored various approaches for clarifying question ranking and generation. However, due to the lack of real conversational search data, they have to use artificial datasets for training, which limits their generalizability to real-world search scenarios. As a result, the industry has shown reluctance to implement them in reality, further suspending the availability of real conversational search interaction data. The above dilemma can be formulated as a cold start problem of clarifying question generation and conversational search in general. Furthermore, even if we do have large-scale conversational logs, it is not realistic to gather training data that can comprehensively cover all possible queries and topics in open-domain search scenarios. The risk of fitting bias when training a clarifying question retrieval/generation model on incomprehensive dataset is thus another important challenge.

In this work, we innovatively explore generating clarifying questions in a zero-shot setting to overcome the cold start problem and we propose a constrained clarifying question generation system which uses both question templates and query facets to guide the effective and precise question generation. The experiment results show that our method outperforms existing state-of-the-art zero-shot baselines by a large margin. Human annotations to our model outputs also indicate our method generates 25.2% more natural questions, 18.1% more useful questions, 6.1% less unnatural and 4% less useless questions.

References

[1]

Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2020. ConvAI3: Generating clarifying questions for open-domain dialogue systems (ClariQ). arXiv preprint arXiv:2009.11352 (2020).

[2]

Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42nd international acm sigir conference on research and development in information retrieval. 475–484.

Digital Library

[3]

Avishek Anand, Lawrence Cavedon, Matthias Hagen, Hideo Joho, Mark Sanderson, and Benno Stein. 2021. Dagstuhl Seminar 19461 on Conversational Search: Seminar Goals and Working Group Outcomes. SIGIR Forum 54, 1, Article 3 (feb 2021), 11 pages. https://doi.org/10.1145/3451964.3451967

Digital Library

[4]

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided Open Vocabulary Image Captioning with Constrained Beam Search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, 936–945. https://doi.org/10.18653/v1/D17-1098

[5]

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Association for Computational Linguistics, Ann Arbor, Michigan, 65–72. https://aclanthology.org/W05-0909

[6]

Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In 11th conference of the european chapter of the association for computational linguistics. 313–320.

[7]

Michael Bendersky, Donald Metzler, and W. Bruce Croft. 2010. Learning Concept Importance Using a Weighted Dependence Model. In Proceedings of the Third ACM International Conference on Web Search and Data Mining (New York, New York, USA) (WSDM ’10). Association for Computing Machinery, New York, NY, USA, 31–40. https://doi.org/10.1145/1718487.1718492

Digital Library

[8]

Keping Bi, Qingyao Ai, and W Bruce Croft. 2021. Asking Clarifying Questions Based on Negative Feedback in Conversational Search. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. 157–166.

Digital Library

[9]

Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020).

[10]

Charles L. A. Clarke, Nick Craswell, and Ian Soboroff. 2009. Overview of the TREC 2009 Web Track. In TREC.

[11]

J Shane Culpepper, Fernando Diaz, and Mark D Smucker. 2018. Research frontiers in information retrieval: Report from the third strategic workshop on information retrieval in lorne (swirl 2018). In ACM SIGIR Forum, Vol. 52. ACM New York, NY, USA, 34–90.

Digital Library

[12]

Van Dang and Bruce W Croft. 2010. Query reformulation using anchor text. In Proceedings of the third ACM international conference on Web search and data mining. 41–50.

Digital Library

[13]

Kaustubh D Dhole. 2020. Resolving intent ambiguities by retrieving discriminative clarifying questions. arXiv preprint arXiv:2008.07559 (2020).

[14]

Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that¿ learning to rewrite questions-in-context. Can You Unpack That¿ Learning to Rewrite Questions-in-Context (2019).

[15]

Zuohui Fu, Yikun Xian, Yongfeng Zhang, and Yi Zhang. 2020. Tutorial on Conversational Recommendation Systems. In Fourteenth ACM Conference on Recommender Systems. 751–753.

[16]

Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational ai. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 1371–1374.

[17]

Jianfeng Gao, Chenyan Xiong, Paul Bennett, and Nick Craswell. 2022. Neural approaches to conversational information retrieval. arXiv preprint arXiv:2201.05176 (2022).

[18]

Claudia Hauff, Julia Kiseleva, Mark Sanderson, Hamed Zamani, and Yongfeng Zhang. 2021. Conversational Search and Recommendation: Introduction to the Special Issue.

[19]

Chris Hokamp and Qun Liu. 2017. Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 1535–1546. https://doi.org/10.18653/v1/P17-1141

[20]

Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv preprint arXiv:1905.01969 (2019).

[21]

Kimiya Keyvan and Jimmy Xiangji Huang. 2022. How to Approach Ambiguous Queries in Conversational Search¿ A Survey of Techniques, Approaches, Tools and Challenges. ACM Computing Surveys (CSUR) (2022).

[22]

Antonios Minas Krasakis, Mohammad Aliannejadi, Nikos Voskarides, and Evangelos Kanoulas. 2020. Analysing the effect of clarifying questions on document ranking in conversational search. In Proceedings of the 2020 acm sigir on international conference on theory of information retrieval. 129–132.

Digital Library

[23]

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021).

[24]

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 1823–1840. https://doi.org/10.18653/v1/2020.findings-emnlp.165

[25]

Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013

[26]

Chang Liu, Jacek Gwizdka, Jingjing Liu, Tao Xu, and Nicholas J Belkin. 2010. Analysis and evaluation of query reformulations in different task types. Proceedings of the American Society for Information Science and Technology 47, 1 (2010), 1–9.

Digital Library

[27]

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 (2021).

Digital Library

[28]

Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, 2021. Neurologic a* esque decoding: Constrained text generation with lookahead heuristics. arXiv preprint arXiv:2112.08726 (2021).

[29]

Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Neurologic decoding:(un) supervised neural text generation with predicate logic constraints. arXiv preprint arXiv:2010.12884 (2020).

[30]

Gary Marchionini. 2006. Exploratory Search: From Finding to Understanding. Commun. ACM 49, 4 (apr 2006), 41–46. https://doi.org/10.1145/1121949.1121979

Digital Library

[31]

Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. arXiv preprint arXiv:1809.01984 (2018).

[32]

Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Vol. 24. Elsevier, 109–165.

[33]

Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. Cgmh: Constrained sentence generation by metropolis-hastings sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 6834–6842.

Digital Library

[34]

Wenjie Ou and Yue Lin. 2020. A clarifying question selection system from ntes_along in convai3 challenge. arXiv preprint arXiv:2010.14202 (2020).

[35]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 311–318. https://doi.org/10.3115/1073083.1073135

Digital Library

[36]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.

[37]

Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. In Proceedings of the 2017 conference on conference human information interaction and retrieval. 117–126.

Digital Library

[38]

Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. arXiv preprint arXiv:1805.04655 (2018).

[39]

Sudha Rao and Hal Daumé III. 2019. Answer-based adversarial training for generating clarification questions. arXiv preprint arXiv:1904.02281 (2019).

[40]

Corbin Rosset, Chenyan Xiong, Xia Song, Daniel Campos, Nick Craswell, Saurabh Tiwary, and Paul Bennett. 2020. Leading conversational search by suggesting useful questions. In Proceedings of The Web Conference 2020. 1160–1170.

Digital Library

[41]

Rodrygo LT Santos, Craig Macdonald, Iadh Ounis, 2015. Search result diversification. Foundations and Trends® in Information Retrieval 9, 1 (2015), 1–90.

Digital Library

[42]

Timo Schick and Hinrich Schütze. 2021. Few-shot text generation with natural language instructions. Association for Computational Linguistics.

[43]

David Schlangen. 2004. Causes and strategies for requesting clarification in dialogue. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004. 136–143.

[44]

Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2021. Towards Facet-Driven Generation of Clarifying Questions for Conversational Search. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. 167–175.

Digital Library

[45]

Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In proceedings of the 24th ACM international on conference on information and knowledge management. 553–562.

Digital Library

[46]

Alexandra Vtyurina, Denis Savenkov, Eugene Agichtein, and Charles LA Clarke. 2017. Exploring conversational search with humans, assistants, and wizards. In Proceedings of the 2017 chi conference extended abstracts on human factors in computing systems. 2187–2193.

Digital Library

[47]

Jian Wang and Wenjie Li. 2021. Template-guided Clarifying Question Generation for Web Search Clarification. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 3468–3472.

Digital Library

[48]

Sean Welleck, Kianté Brantley, Hal Daumé Iii, and Kyunghyun Cho. 2019. Non-monotonic sequential text generation. In International Conference on Machine Learning. PMLR, 6716–6726.

[49]

Julia White, Gabriel Poesia, Robert Hawkins, Dorsa Sadigh, and Noah Goodman. 2021. Open-domain clarification question generation without question examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 563–570. https://doi.org/10.18653/v1/2021.emnlp-main.44

[50]

Jingjing Xu, Yuechen Wang, Duyu Tang, Nan Duan, Pengcheng Yang, Qi Zeng, Ming Zhou, and Xu Sun. 2019. Asking Clarification Questions in Knowledge-Based Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 1618–1629. https://doi.org/10.18653/v1/D19-1172

[51]

Liu Yang, Hamed Zamani, Yongfeng Zhang, Jiafeng Guo, and W Bruce Croft. 2017. Neural matching models for question retrieval and next question prediction in conversation. arXiv preprint arXiv:1707.05409 (2017).

[52]

Shi Yu, Jiahua Liu, Jingqin Yang, Chenyan Xiong, Paul Bennett, Jianfeng Gao, and Zhiyuan Liu. 2020. Few-shot generative conversational query rewriting. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 1933–1936.

Digital Library

[53]

Hamed Zamani and Nick Craswell. 2020. Macaw: An extensible conversational information seeking platform. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2193–2196.

Digital Library

[54]

Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck. 2020. Generating clarifying questions for information retrieval. In Proceedings of The Web Conference 2020. 418–428.

Digital Library

[55]

Hamed Zamani, Johanne R Trippas, Jeff Dalton, and Filip Radlinski. 2022. Conversational information seeking. arXiv preprint arXiv:2201.08808 (2022).

[56]

Maosen Zhang, Nan Jiang, Lei Li, and Yexiang Xue. 2020. Language Generation via Combinatorial Constraint Satisfaction: A Tree Search Enhanced Monte-Carlo Approach. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 1286–1298. https://doi.org/10.18653/v1/2020.findings-emnlp.115

[57]

Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W Bruce Croft. 2018. Towards conversational search and recommendation: System ask, user respond. In Proceedings of the 27th acm international conference on information and knowledge management. 177–186.

Digital Library

[58]

Ziliang Zhao, Zhicheng Dou, Jiaxin Mao, and Ji-Rong Wen. 2022. Generating Clarifying Questions with Web Search Results. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 234–244.

Digital Library

Cited By

Liu QZhu JYang YDai QDu ZWu XZhao ZZhang RDong ZBaeza-Yates RBonchi F(2024)Multimodal Pretraining, Adaptation, and Generation for Recommendation: A SurveyProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671473(6566-6576)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671473
Zhao ZDou ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Generating Multi-turn Clarification for Web Information SeekingProceedings of the ACM Web Conference 202410.1145/3589334.3645712(1539-1548)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645712
Liu ZChen ZRen ZGao SMa JRen P(2024)Center-retained fine-tuning for conversational question ranking through unsupervised center identificationInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10357861:2Online publication date: 12-Apr-2024
https://dl.acm.org/doi/10.1016/j.ipm.2023.103578
Show More Cited By

Index Terms

Zero-shot Clarifying Question Generation for Conversational Search
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval

Recommendations

Zero-shot Query Reformulation for Conversational Search
ICTIR '23: Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval

As the popularity of voice assistants continues to surge, conversational search has gained increased attention in Information Retrieval. However, data sparsity issues in conversational search significantly hinder the progress of supervised conversational ...
Improving and Understanding Clarifying Question Generation in Conversational Search
Text, Speech, and Dialogue
Abstract
Conversational information-seeking systems (CISs), such as chatbots and virtual personal assistants, encounter difficulty when processing ambiguous user requests (URs) and generate an accurate response, especially when multiple search results ...
Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Search
ICTIR '20: Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval

Recent research on conversational search highlights the importance of mixed-initiative in conversations. To enable mixed-initiative, the system should be able to ask clarifying questions to the user. However, the ability of the underlying ranking models ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '23: Proceedings of the ACM Web Conference 2023

April 2023

4293 pages

ISBN:9781450394161

DOI:10.1145/3543507

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 April 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '23

Sponsor:

SIGWEB

WWW '23: The ACM Web Conference 2023

April 30 - May 4, 2023

TX, Austin, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
834
Total Downloads

Downloads (Last 12 months)579
Downloads (Last 6 weeks)75

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu QZhu JYang YDai QDu ZWu XZhao ZZhang RDong ZBaeza-Yates RBonchi F(2024)Multimodal Pretraining, Adaptation, and Generation for Recommendation: A SurveyProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671473(6566-6576)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671473
Zhao ZDou ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Generating Multi-turn Clarification for Web Information SeekingProceedings of the ACM Web Conference 202410.1145/3589334.3645712(1539-1548)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645712
Liu ZChen ZRen ZGao SMa JRen P(2024)Center-retained fine-tuning for conversational question ranking through unsupervised center identificationInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10357861:2Online publication date: 12-Apr-2024
https://dl.acm.org/doi/10.1016/j.ipm.2023.103578
Ni SBi KGuo JCheng X(2023)A Comparative Study of Training Objectives for Clarification Facet GenerationProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3625332(1-10)Online publication date: 26-Nov-2023
https://dl.acm.org/doi/10.1145/3624918.3625332
Meng CAliannejadi Mde Rijke MFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)System Initiative Prediction for Multi-turn Conversational Information SeekingProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615070(1807-1817)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615070
Liu BHu YAi QLiu YWu YLi CShen WFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Leveraging Event Schema to Ask Clarifying Questions for Conversational Legal Case RetrievalProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614953(1513-1522)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614953
He SZhang SZhang XFeng Z(2023)Improve Conversational Search with Multi-document InformationNeural Information Processing10.1007/978-981-99-8178-6_1(3-15)Online publication date: 30-Nov-2023
https://doi.org/10.1007/978-981-99-8178-6_1

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents