Abstract
Continual relation extraction (CRE) aims to continually learn new relations while maintaining knowledge of previous relations in the data streams. Recently, continual few-shot relation extraction (CFRE) is introduced, in which only the first step of training has enough data, while the later steps have only a handful of samples, since emerging relations are often difficult to obtain enough training data quickly in realistic scenarios. Some previous work has proved that storing samples from previous tasks can alleviate the problem of catastrophic forgetting in continual learning. However, such approaches rely heavily on memory size and fail to fully exploit the knowledge associated with memorized samples in the pre-trained language model (PLM). To solve these problems, we propose a novel Prompt-based Contrastive Learning method, namely PCLfor CFRE with two optimizations. Firstly, to make better use of the knowledge in the limited number of training samples, we propose to obtain better representations of the relations by a prompt-based encoder with the assistance of a well-designed template. A mix-up data augmentation strategy is harnessed to increase the robustness of the model and prevent few-shot tasks from over-fitting. Secondly, to alleviate forgetting previous knowledge, in addition to replaying samples in the memory, our method maintains the stability of instance embeddings and retains the knowledge of the learned relations by an instance-level distillation loss. With extensive experiments on two benchmarks, we demonstrate that our model significantly outperforms state-of-the-art baselines and increases the generalization and robustness of the CFRE model on the imbalanced dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Pang, N., Tan, Z., Zhao, X., Zeng, W., Xiao, W.: Domain relation extraction from noisy Chinese texts. Neurocomputing 418, 21–35 (2020)
Huang, P., Fang, Y., Zhu, H., Xiao, W.: End-to-end knowledge triplet extraction combined with adversarial training. J. Comput. Res. Develop. 56, 2536–2548 (2019)
Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725. Berlin, Germany (2016)
Han, X.: Continual relation learning via episodic memory activation and reconsolidation. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6429–6440 (2020)
Wu, T., et al.: Curriculum-meta learning for order-robust continual relation extraction. In: AAAI Conference on Artificial Intelligence (2021)
Qin, C., Joty, S.: Continual few-shot relation learning via embedding space regularization and data augmentation. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2776–2789. Dublin, Ireland (2022)
McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109–165. Elsevier (1989)
Parnami, A., Lee, M.: Learning from few examples: a summary of approaches to few-shot learning. arXiv:2203.04291 (2022)
Ritter, H., Botev, A., Barber, D.: Online structured laplace approximations for overcoming catastrophic forgetting. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Mallya, A., Davis, D., Lazebnik, S.: Piggyback: adapting a single network to multiple tasks by learning to mask weights. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 67–82 (2018)
Sun, F., Ho, C., Lee, H.: LAMOL: language modeling for lifelong language learning. In: 8th International Conference on Learning Representations, ICLR 2020. Addis Ababa, Ethiopia (2020)
Cui, L., et al.: Refining sample embeddings with relation prototypes to enhance continual relation extraction. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 232–243 (2021)
Zhao, K., Xu, H., Yang, J., et al.: Consistent representation learning for continual relation extraction. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 3402–3411. Dublin, Ireland (2022)
Chen, K., Lee, C.G.: Incremental few-shot learning via vector quantization in deep embedded space. In: International Conference on Learning Representations (2021)
Pang, N., Tan, Z., Hao, X., Xiao, W.: Boosting knowledge base automatically via few-shot relation classification. Front. Neurorobot. 14, 584192 (2020)
Pang, N., Zhao, X., Wang, W., Xiao, W., Guo, D.: Few-shot text classification by leveraging bi-directional attention and cross-class knowledge. Sci. China Inf. Sci. 64, 1–13 (2021)
Fei, J., Zeng, W., Zhao, X., Li, X., Xiao, W.: Few-shot relational triple extraction with perspective transfer network. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 488–498 (2022)
Zhang, L., Deng, Z., Kawaguchi, K., Ghorbani, A., Zou, J.: How does mixup help with robustness and generalization? arXiv:2010.04819 (2020)
He, K., Girshick, R., Dollár, P.: Rethinking imagenet pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918–4927 (2019)
Jingyao, W., Zhao, Z., Sun, C., Yan, R., Chen, X.: Few-shot transfer learning for intelligent fault diagnosis of machine. Measurement 166, 108202 (2020)
Tian, R., Shi, H.: Momentum memory contrastive learning for transfer-based few-shot classification. In: Applied Intelligence, pp. 1–15 (2022)
Gao, B., Zhao, X., Zhao, H.: An active and contrastive learning framework for fine-grained off-road semantic segmentation. In: IEEE Transactions on Intelligent Transportation Systems (2022)
Chen, X., et al.: Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In: Proceedings of the ACM Web Conference 2022, pp. 2778–2788 (2022)
Liu, P., Yuan, W., Jinlan, F., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9), 1–35 (2023)
Scao, T.L., Rush, A.M.: How many data points is a prompt worth? arXiv:2103.08493 (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)
Soares, L.B., FitzGerald, N., Ling, J., Kwiatkowski, T.: Matching the blanks: Distributional similarity for relation learning. arXiv:1906.03158 (2019)
Han, X., et al.: Fewrel: a large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. arXiv:1810.10147 (2018)
Zhang, Y., Zhong, V., Chen, D., Angeli, G., Manning, C.D.: Position-aware attention and supervised data improve slot filling. In: Conference on Empirical Methods in Natural Language Processing (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wu, F., Zhang, C., Tan, Z., Xu, H., Ge, B. (2024). Continual Few-Shot Relation Extraction with Prompt-Based Contrastive Learning. In: Song, X., Feng, R., Chen, Y., Li, J., Min, G. (eds) Web and Big Data. APWeb-WAIM 2023. Lecture Notes in Computer Science, vol 14334. Springer, Singapore. https://doi.org/10.1007/978-981-97-2421-5_21
Download citation
DOI: https://doi.org/10.1007/978-981-97-2421-5_21
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-2420-8
Online ISBN: 978-981-97-2421-5
eBook Packages: Computer ScienceComputer Science (R0)