research-article

Enhancing text-based knowledge graph completion with zero-shot large language models: : A focus on semantic enhancement

Authors: Rui Yang, Jiahao Zhu, Jianping Man, Li Fang,

Yi ZhouAuthors Info & Claims

Volume 300, Issue C

https://doi.org/10.1016/j.knosys.2024.112155

Published: 18 November 2024 Publication History

Abstract

The design and development of text-based knowledge graph completion (KGC) methods leveraging textual entity descriptions are at the forefront of research. These methods involve advanced optimization techniques such as soft prompts and contrastive learning to enhance KGC models. The effectiveness of text-based methods largely hinges on the quality and richness of the training data. Large language models (LLMs) can utilize straightforward prompts to alter text data, thereby enabling data augmentation for KGC. Nevertheless, LLMs typically demand substantial computational resources. To address these issues, we introduce a framework termed constrained prompts for KGC (CP-KGC). This CP-KGC framework designs prompts that adapt to different datasets to enhance semantic richness. Additionally, CP-KGC employs a context constraint strategy to effectively identify polysemous entities within KGC datasets. Through extensive experimentation, we have verified the effectiveness of this framework. Even after quantization, the LLM (Qwen-7B-Chat-int4) still enhances the performance of text-based KGC methods. Code and datasets are available at https://github.com/sjlmg/CP-KGC. This study extends the performance limits of existing models and promotes further integration of KGC with LLMs.

References

[1]

H. Sun, T. Bedrax-Weiss, W.W. Cohen, Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text, in: Proceedings Ofthe 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP, 2019, pp. 2380–2390.

[2]

J. Huang, W.X. Zhao, H. Dou, J.-R. Wen, E.Y. Chang, Improving sequential recommendation with knowledge-enhanced memory networks, in: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018, pp. 505–514.

[3]

Bordes A., Usunier N., Garcia-Duran A., Weston J., Yakhnenko O., Translating embeddings for modeling multi-relational data, Adv. Neural Inf. Process. Syst. 26 (2013).

[4]

Z. Wang, J. Zhang, J. Feng, Z. Chen, Knowledge graph embedding by translating on hyperplanes, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 28, 2014.

[5]

Z. Sun, Z.-H. Deng, J.-Y. Nie, J. Tang, Rotate: Knowledge graph embedding by relational rotation in complex space, in: 7th International Conference on Learning Representations, ICLR 2019, 2019.

[6]

L. Wang, W. Zhao, Z. Wei, J. Liu, SimKGC: Simple contrastive knowledge graph completion with pre-trained language models, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, 1: Long Papers, 2022, pp. 4281–4294.

[7]

Yao L., Mao C., Luo Y., KG-BERT: BERT for knowledge graph completion, 2019, arXiv preprint arXiv:1909.03193.

[8]

Singhal K., Azizi S., Tu T., Mahdavi S.S., Wei J., Chung H.W., Scales N., Tanwani A., Cole-Lewis H., Pfohl S., et al., Large language models encode clinical knowledge, Nature 620 (7972) (2023) 172–180,.

[9]

F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A.H. Miller, S. Riedel, Language models as knowledge bases?, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP, 2019, pp. 2463–2473.

[10]

Amatriain X., Prompt design and engineering: Introduction and advanced methods, 2024, arXiv preprint arXiv:2401.14423.

[11]

Bai J., Bai S., Chu Y., Cui Z., Dang K., Deng X., Fan Y., Ge W., Han Y., Huang F., Hui B., Ji L., Li M., Lin J., Lin R., Liu D., Liu G., Lu C., Lu K., Ma J., Men R., Ren X., Ren X., Tan C., Tan S., Tu J., Wang P., Wang S., Wang W., Wu S., Xu B., Xu J., Yang A., Yang H., Yang J., Yang S., Yao Y., Yu B., Yuan H., Yuan Z., Zhang J., Zhang X., Zhang Y., Zhang Z., Zhou C., Zhou J., Zhou X., Zhu T., Qwen technical report, 2023, arXiv preprint arXiv:2309.16609.

[12]

Touvron H., Martin L., Stone K., Albert P., Almahairi A., Babaei Y., Bashlykov N., Batra S., Bhargava P., Bhosale S., et al., Llama 2: Open foundation and fine-tuned chat models, 2023, arXiv preprint arXiv:2307.09288.

[13]

Achiam J., Adler S., Agarwal S., Ahmad L., Akkaya I., Aleman F.L., Almeida D., Altenschmidt J., Altman S., Anadkat S., et al., Gpt-4 technical report, 2023, arXiv preprint arXiv:2303.08774.

[14]

Y. Lin, Z. Liu, M. Sun, Y. Liu, X. Zhu, Learning entity and relation embeddings for knowledge graph completion, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 29, 2015.

[15]

Trouillon T., Welbl J., Riedel S., Gaussier É., Bouchard G., Complex embeddings for simple link prediction, in: International Conference on Machine Learning, PMLR, 2016, pp. 2071–2080.

[16]

Wang X., Gao T., Zhu Z., Zhang Z., Liu Z., Li J., Tang J., KEPLER: A unified model for knowledge embedding and pre-trained language representation, Trans. Assoc. Comput. Linguist. 9 (2021) 176–194.

[17]

B. Wang, T. Shen, G. Long, T. Zhou, Y. Wang, Y. Chang, Structure-augmented text representation learning for efficient knowledge graph completion, in: Proceedings of the Web Conference 2021, 2021, pp. 1737–1748.

[18]

C. Chen, Y. Wang, B. Li, K.-Y. Lam, Knowledge is flat: A seq2seq generative framework for various knowledge graph completion, in: Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 4005–4017.

[19]

Chen C., Wang Y., Sun A., Li B., Lam K.-Y., Dipping plms sauce: Bridging structure and text for effective knowledge graph completion via conditional soft prompting, in: Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 11489–11503.

[20]

Zhao W.X., Zhou K., Li J., Tang T., Wang X., Hou Y., Min Y., Zhang B., Zhang J., Dong Z., et al., A survey of large language models, 2023, arXiv preprint arXiv:2303.18223.

[21]

Wei J., Tay Y., Bommasani R., Raffel C., Zoph B., Borgeaud S., Yogatama D., Bosma M., Zhou D., Metzler D., et al., Emergent abilities of large language models, 2022, arXiv preprint arXiv:2206.07682.

[22]

Pan S., Luo L., Wang Y., Chen C., Wang J., Wu X., Unifying large language models and knowledge graphs: A roadmap, IEEE Trans. Knowl. Data Eng. (2024).

[23]

Y. Zhou, A.I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, J. Ba, Large language models are human-level prompt engineers, in: NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.

[24]

T. Shin, Y. Razeghi, R.L. Logan IV, E. Wallace, S. Singh, Autoprompt: Eliciting knowledge from language models with automatically generated prompts, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, 2020, pp. 4222–4235.

[25]

Brown T., Mann B., Ryder N., Subbiah M., Kaplan J.D., Dhariwal P., Neelakantan A., Shyam P., Sastry G., Askell A., et al., Language models are few-shot learners, Adv. Neural Inf. Process. Syst. 33 (2020) 1877–1901.

[26]

Jiang Z., Xu F.F., Araki J., Neubig G., How can we know what language models know?, Trans. Assoc. Comput. Linguist. 8 (2020) 423–438.

[27]

Wei J., Wang X., Schuurmans D., Bosma M., Xia F., Chi E., Le Q.V., Zhou D., et al., Chain-of-thought prompting elicits reasoning in large language models, Adv. Neural Inf. Process. Syst. 35 (2022) 24824–24837.

[28]

Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I., Attention is all you need, Adv. Neural Inf. Process. Syst. 30 (2017).

Digital Library

[29]

Devlin J., Chang M.-W., Lee K., Toutanova K., Bert: Pre-training of deep bidirectional transformers for language understanding, 2018, arXiv preprint arXiv:1810.04805.

[30]

Liu Y., Ott M., Goyal N., Du J., Joshi M., Chen D., Levy O., Lewis M., Zettlemoyer L., Stoyanov V., Roberta: A robustly optimized bert pretraining approach, 2019, arXiv preprint arXiv:1907.11692.

[31]

Raffel C., Shazeer N., Roberts A., Lee K., Narang S., Matena M., Zhou Y., Li W., Liu P.J., Exploring the limits of transfer learning with a unified text-to-text transformer, J. Mach. Learn. Res. 21 (140) (2020) 1–67.

[32]

Bano S., Khalid S., Tairan N.M., Shah H., Khattak H.A., Summarization of scholarly articles using BERT and BiGRU: Deep learning-based extractive approach, J. King Saud Univ.-Comput. Inf. Sci. 35 (9) (2023).

[33]

Bano S., Khalid S., BERT-based extractive text summarization of scholarly articles: A novel architecture, in: 2022 International Conference on Artificial Intelligence of Things, ICAIoT, IEEE, 2022, pp. 1–5.

[34]

B. Yang, W.-t. Yih, X. He, J. Gao, L. Deng, Embedding entities and relations for learning and inference in knowledge bases, in: 3rd International Conference on Learning Representations, ICLR 2015, 2014.

[35]

T. Dettmers, P. Minervini, P. Stenetorp, S. Riedel, Convolutional 2d knowledge graph embeddings, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, 2018.

[36]

Kaplan J., McCandlish S., Henighan T., Brown T.B., Chess B., Child R., Gray S., Radford A., Wu J., Amodei D., Scaling laws for neural language models, 2020, arXiv preprint arXiv:2001.08361.

Index Terms

Enhancing text-based knowledge graph completion with zero-shot large language models: A focus on semantic enhancement
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Data management systems
    1. Database design and models
  2. Information retrieval
    1. Document representation

Index terms have been assigned to the content through auto-classification.

Recommendations

On the security of a strong provably secure identity-based encryption scheme without bilinear pairing

The identity-based encryption scheme enables a sender to generate the ciphertext using a receiver's identity and system's parameters. Because of its convenience, the identity-based encryption scheme has been widely used in many practical applications. ...
Making Large Language Models Perform Better in Knowledge Graph Completion
MM '24: Proceedings of the 32nd ACM International Conference on Multimedia

Large language model (LLM) based knowledge graph completion (KGC) aims to predict the missing triples in the KGs with LLMs. However, research about LLM-based KGC fails to sufficiently harness LLMs' inference proficiencies, overlooking critical structural ...
Enhancing Low-Resource NER via Knowledge Transfer from LLM
Computational Collective Intelligence
Abstract
This paper presents a study for low-resource language NER via knowledge transfer using large pre-trained language models. The goals of the study are to enhance the performance of the proposed model for low-resource language NER through knowledge ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Knowledge-Based Systems

Knowledge-Based Systems Volume 300, Issue C

Sep 2024

1714 pages

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 18 November 2024

Author Tags

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents