Abstract
A user input to a schema-driven dialogue information navigation system, such as venue search, is typically constrained by the underlying database which restricts the user to specify a predefined set of preferences, or slots, corresponding to the database fields. We envision a more natural information navigation dialogue interface where a user has flexibility to specify unconstrained preferences that may not match a predefined schema. We propose to use information retrieval from unstructured knowledge to identify entities relevant to a user request. We construct an up-to-date database of restaurants in Cambridge, including unstructured knowledge snippets (reviews and information from the web) and annotate a set of query-snippet pairs with relevance labels. We use the annotated dataset to train and evaluate snippet relevance classifiers, as a proxy to evaluating recommendation accuracy. We show that with a pretrained transformer model as an encoder, an unsupervised/supervised classifier achieves a weighted F1 of 0.661/0.856.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
A system may ask to narrow down the search criteria and a user may specify additional preferences in consecutive turns.
- 2.
The dataset is compiled by crawling the Web in January 2021.
- 3.
As the task was very short, the participants were not paid.
- 4.
The number of top results (5) was chosen empirically since a user of a dialogue system may navigate over multiple search results.
- 5.
Three query-snippet pairs with a known label are used to monitor work quality. The workers who did not pass the quality test were rejected.
- 6.
- 7.
We use empirically chosen J \(=\) 5 and N \(=\) 5 in this work.
- 8.
The vocabulary size in our domain is 30124.
- 9.
Experiments were conducted with i7-6 cores CPU and single GTX 1080 GPU.
- 10.
The project was completed during a 4-month internship and the annotation costs were under $300.
References
Budzianowski P, Wen T-H, Tseng B-H, Casanueva I, Ultes S, Ramadan O, Gavsic M (2018) MultiWOZ—a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In: Proceedings of the 2018 conference on empirical methods in natural language processing, Brussels, Belgium, pp 5016–5026
Rastogi A, Zang X, Sunkara S, Gupta R, Khaitan P (2020) Schema-guided dialogue state tracking task at DSTC8. In: Proceedings of the AAAI dialog system technology challenges workshop
Stoyanchev S, Keizer S, Doddipatla R (2021) Action state update approach to dialogue management. In: ICASSP 2021–2021 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 7398–7402
Henderson M, Thomson B, Williams JD (2014) The second dialog state tracking challenge. In: Proceedings of the 15th annual SIGdial meeting on discourse and dialogue, pp 263–272
Trippas J.R, Spina D, Cavedon L, Joho H, Sanderson M (2018) Informing the design of spoken conversational search: Perspective paper. In: Proceedings of the 2018 conference on human information interaction & retrieval. CHIIR ’18. ACM, New York, NY, USA, pp 32–41
Yan Z, Duan N, Chen P, Zhou M, Zhou J, Li Z (2017) Building task-oriented dialogue systems for online shopping. In: Proceedings of the thirty-first AAAI conference on artificial intelligence. AAAI’17. AAAI Press, San Francisco, CA, USA , pp 4618–4625
Vakulenko S, Revoredo K, Di Ciccio C, de Rijke M (2019) Qrfa: a data-driven model of information-seeking dialogues. In: Azzopardi L, Stein B, Fuhr N, Mayr P, Hauff C, Hiemstra D (eds) Advances in information retrieval. Springer, Cham, pp 541–557
Komatani K, Ueno S, Kawahara T, Okuno HG (2005) User modeling in spoken dialogue systems to generate flexible guidance. User Model User Adapt Interact 15(1–2):169–183
Tomko S, Rosenfeld R (2004) Speech graffiti versus natural language: Assessing the user experience. In: Proceedings of HLT-NAACL 2004: short papers. Association for Computational Linguistics, Boston, MA, USA, pp 73–76
Kim S, Eric M, Gopalakrishnan K, Hedayatnia B, Liu Y, Hakkani-Tur D (2020) Beyond domain APIs: Task-oriented conversational modeling with unstructured knowledge access. In: Proceedings of the 21th annual meeting of the special interest group on discourse and dialogue. Association for Computational Linguistics, 1st virtual meeting, pp 278–289
Louis A, Roth D, Radlinski F (2020) “I’d rather just go to bed”: Understanding indirect answers. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). Association for Computational Linguistics, Online, pp 7411–7425
Radlinski F, Balog K, Byrne B, Krishnamoorthi K (2019) Coached conversational preference elicitation: a case study in understanding movie preferences. In: Proceedings of the annual SIGdial meeting on discourse and dialogue
Devlin J, Chang M.-W, Lee K, Toutanova K (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT
Wang Z, Ng P, Ma X, Nallapati R, Xiang B (2019) Multi-passage bert: a globally normalized bert model for open-domain question answering. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 5881–5885
Sakata W, Shibata T, Tanaka R, Kurohashi S (2019) Faq retrieval using query-question similarity and bert-based query-answer relevance. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp 1113–1116
Izacard G, Grave E (2021) Leveraging passage retrieval with generative models for open domain question answering. In: Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pp 874–880
Zhan J, Mao J, Liu Y, Zhang M, Ma S (2020) An analysis of BERT in document ranking. Association for Computing Machinery, New York, NY, USA, pp 1941–1944
Bowman SR, Angeli G, Potts C, Manning CD (2015) A large annotated corpus for learning natural language inference. In: EMNLP, pp 632–642
Fleiss JL, Cohen J (1973) The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas 33(3):613–619
Salton G, Buckley C (1988) Term-weighting approaches in automatic text retrieval. Inf Process Manag 24(5):513–523
Reimers N, Gurevych I (2019) Sentence-bert: sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 3982–3992
Gao Y, Colombo N, Wang W (2021) Adapting by pruning: a case study on BERT. arXiv:2105.03343
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Pandey, S., Stoyanchev, S., Doddipatla, R. (2022). Towards Handling Unconstrained User Preferences in Dialogue. In: Stoyanchev, S., Ultes, S., Li, H. (eds) Conversational AI for Natural Human-Centric Interaction. Lecture Notes in Electrical Engineering, vol 943. Springer, Singapore. https://doi.org/10.1007/978-981-19-5538-9_6
Download citation
DOI: https://doi.org/10.1007/978-981-19-5538-9_6
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-5537-2
Online ISBN: 978-981-19-5538-9
eBook Packages: Computer ScienceComputer Science (R0)