Abstract
Conversational agents are usually designed for closed-world environments. Unfortunately, users can behave unexpectedly. Based on the open-world environment, we often encounter the situation that the training and test data are sampled from different distributions. Then, data from different distributions are called out-of-domain (OOD). A robust conversational agent needs to react to these OOD utterances adequately. Thus, the importance of robust OOD detection is emphasized. Unfortunately, collecting OOD data is a challenging task. We have designed an OOD detection algorithm independent of OOD data that outperforms a wide range of current state-of-the-art algorithms on publicly available datasets. Our algorithm is based on a simple but efficient approach of combining metric learning with adaptive decision boundary. Furthermore, compared to other algorithms, we have found that our proposed algorithm has significantly improved OOD performance in a scenario with a lower number of classes while preserving the accuracy for in-domain (IND) classes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Deep Average Network (DAN) and with Transformer encoder (TRAN).
- 2.
AWS ml.m5.4xlarge.
- 3.
References
Casanueva, I., Temcinas, T., Gerz, D., Henderson, M., Vulic, I.: Efficient Intent Detection with Dual Sentence Encoders. Association for Computational Linguistics (2020)
Cer, D., et al.: Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018)
Chen, D., Yu, Z.: Gold: improving out-of-scope detection in dialogues using data augmentation. In: Empirical Methods in Natural Language Processing, pp. 402–434 (2021)
Chen, W., Chen, X., Zhang, J., Huang, K.: Beyond triplet loss: a deep quadruplet network for person re-identification. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1320–1329 (2017)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: North American Association for Computational Linguistics, pp. 68–94 (2019)
Finch, J.D., et al.: Emora: an inquisitive social chatbot who cares for you. In: Alexa Prize Processing, vol. 3 (2020)
Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks (2016)
Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: Feragen, A., Pelillo, M., Loog, M. (eds.) Similarity-Based Pattern Recognition, pp. 84–92 (2015)
Keet, C.M.: Open World Assumption, pp. 1567–1567. Springer, New York (2013). https://doi.org/10.1007/978-1-4419-9863-7_734
Konrád, J., et al.: Alquist 4.0: towards social intelligence using generative models and dialogue personalization. In: Alexa Prize Proceeding, vol. 4 (2021)
Larson, S., et al.: An evaluation dataset for intent classification and out-of-scope prediction. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (2019)
Lin, T.E., Xu, H.: Deep unknown intent detection with margin loss. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5491–5496 (2019)
Moore, R.J., Arar, R.: Conversational UX design: an introduction. In: Moore, R.J., Szymanski, M.H., Arar, R., Ren, G.-J. (eds.) Studies in Conversational UX Design. HIS, pp. 1–16. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-95579-7_1
Pichl, J., Marek, P., Konrád, J., Lorenc, P., Ta, V.D., Sedivý, J.: Alquist 3.0: Alexa prize bot using conversational knowledge graph. In: Alexa Prize Processing, vol. 3 (2020)
Reimers, N., Gurevych, I.: Sentence-Bert: sentence embeddings using Siamese Bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (2019)
Shafaei, A., Schmidt, M., Little, J.J.: A less biased evaluation of out-of-distribution sample detectors. In: BMVC, p. 3 (2019)
Shu, L., Benajiba, Y., Mansour, S., Zhang, Y.: Odist: open world classification via distributionally shifted instances. In: Empirical Methods in Natural Language Processing (2021)
Shu, L., Xu, H., Liu, B.: Doc: deep open classification of text documents. In: Empirical Methods in Natural Language Processing, pp. 2911–2916 (2017)
Sun, Y., et al.: Circle loss: a unified perspective of pair similarity optimization. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6397–6406 (2020)
Wang, H., et al.: Cosface: large margin cosine loss for deep face recognition. In: CVPR, pp. 5612–5634 (2018)
Zhang, H., Xu, H., Lin, T.E.: Deep open intent classification with adaptive decision boundary. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 14374–14382 (2021)
Acknowledgments
This research was partially supported by the Grant Agency of the Czech Technical University in Prague, grant (SGS22/082/OHK3/1T/37).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lorenc, P. et al. (2022). Metric Learning and Adaptive Boundary for Out-of-Domain Detection. In: Rosso, P., Basile, V., Martínez, R., Métais, E., Meziane, F. (eds) Natural Language Processing and Information Systems. NLDB 2022. Lecture Notes in Computer Science, vol 13286. Springer, Cham. https://doi.org/10.1007/978-3-031-08473-7_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-08473-7_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-08472-0
Online ISBN: 978-3-031-08473-7
eBook Packages: Computer ScienceComputer Science (R0)