research-article

A distributed joint extraction framework for sedimentological entities and relations with federated learning

Authors:

Guorui WangAuthors Info & Claims

Volume 213, Issue PC

https://doi.org/10.1016/j.eswa.2022.119216

Published: 01 March 2023 Publication History

Abstract

Sedimentological knowledge graphs can be used to identify natural resources in earth layers, which may help geologists analyze the distribution of oil crude in earth, and therefore locating the oilfield that is unknown. The building of such knowledge graphs mainly counts on the methods of joint extraction for pairwise entities and the corresponding relations on large-scale data. However, the whole sedimentological data is fairly owned by the different parties with the possibly inconsistent format. Centralized processing on sedimentological data as a whole will be either securely or structurally impractical. Therefore, this paper proposes a framework of distributed joint extraction in order to harvest knowledge triplets on distributed sedimentological corpus that are from many disparate sources without data transmission. The experimental studies demonstrate our methods not only approach the previous state-of-the-art but also protect the data privacy and security for data holders.

Highlights

•

A corpus is built upon sedimentological literature abstracts on our own.

•

Manually annotate training and testing texts.

•

Distributed deep learning framework is proposed for the joint extraction task.

•

Federated learning is used to implement distributed deep learning framework.

•

Federated learning with two novel parameter-sharing strategies.

References

[1]

Chen Z., Guo C., A pattern-first pipeline approach for entity and relation extraction, Neurocomputing 494 (2022) 182–191,.

[2]

Chen M., Shlezinger N., Poor H.V., Eldar Y.C., Cui S., Communication-efficient federated learning, Proceedings of the National Academy of Sciences (17) (2021).

[3]

Chen C., Xu H., Wang W., Li B., Li B., Chen L., et al., Communication-efficient federated learning with adaptive parameter freezing, in: 2021 IEEE 41st International conference on distributed computing systems, IEEE, 2021, pp. 1–11.

[4]

Doddington G.R., Mitchell A., Przybocki M.A., Ramshaw L.A., Strassel S.M., Weischedel R.M., The automatic content extraction (ace) program-tasks, data, and evaluation, in: Lrec, no. 1, Lisbon, 2004, pp. 837–840.

[5]

Fallah A., Mokhtari A., Ozdaglar A., Personalized federated learning: A meta-learning approach, 2020, arXiv preprint arXiv:2002.07948.

[6]

Fei H., Zhang Y., Ren Y., Ji D., A span-graph neural model for overlapping entity relation extraction in biomedical texts, Bioinformatics 37 (11) (2021) 1581–1589.

[7]

Ge S., Wu F., Wu C., Qi T., Huang Y., Xie X., Fedner: Privacy-preserving medical named entity recognition with federated learning, 2020, arXiv preprint arXiv:2003.09288.

[8]

Geng Z., Zhang Y., Han Y., Joint entity and relation extraction model based on rich semantics, Neurocomputing 429 (2021) 132–140.

[9]

Goerttler T., Obermayer K., Similarity of pre-trained and fine-tuned representations, 2022, arXiv preprint arXiv:2207.09225.

[10]

Gurulingappa H., Rajput A.M., Roberts A., Fluck J., Hofmann-Apitius M., Toldo L., Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports, Journal of Biomedical Informatics 45 (5) (2012) 885–892.

Digital Library

[11]

He H., Ren Y., Li Z., Xue J., Adaptive knowledge distillation for efficient relation classification, in: International conference on artificial neural networks, Springer, 2022, pp. 148–158.

[12]

Ji S., Pan S., Cambria E., Marttinen P., Philip S.Y., A survey on knowledge graphs: Representation, acquisition, and applications, IEEE Transactions on Neural Networks and Learning Systems 33 (2) (2021) 494–514.

[13]

Kingma D.P., Ba J., Adam: A method for stochastic optimization, 2014, arXiv preprint arXiv:1412.6980.

[14]

Kornblith S., Norouzi M., Lee H., Hinton G., Similarity of neural network representations revisited, in: International conference on machine learning, PMLR, 2019, pp. 3519–3529.

[15]

Li, Q., Diao, Y., Chen, Q., & He, B. (2022). Federated Learning on Non-IID Data Silos: An Experimental Study. In 2022 IEEE 38th International conference on data engineering (pp. 965–978). https://doi.org/10.1109/ICDE53745.2022.00077.

[16]

Li, J., Fei, H., Liu, J., Wu, S., Zhang, M., Teng, C., et al. (2022). Unified named entity recognition as word-word relation classification. In Proceedings of the AAAI Conference on artificial intelligence, no. 10 (pp. 10965–10973).

[17]

Li, Q., He, B., & Song, D. (2021). Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition (pp. 10713–10722).

[18]

Li, Q., & Ji, H. (2014). Incremental Joint Extraction of Entity Mentions and Relations. In ACL (1) (pp. 402–412).

[19]

Li R., Li D., Yang J., Xiang F., Ren H., Jiang S., et al., Joint extraction of entities and relations via an entity correlated attention neural model, Information Sciences 581 (2021) 179–193.

Digital Library

[20]

Lin B.Y., He C., Zeng Z., Wang H., Huang Y., Soltanolkotabi M., et al., Fednlp: A research platform for federated learning in natural language processing, 2021, arXiv preprint arXiv:2104.08815.

[21]

Liu X., Chen H., Xia W., Overview of named entity recognition, Journal of Contemporary Educational Research 6 (5) (2022) 65–68.

[22]

Luan Y., He L., Ostendorf M., Hajishirzi H., Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction, 2018, arXiv preprint arXiv:1808.09602.

[23]

McMahan B., Moore E., Ramage D., Hampson S., y Arcas B.A., Communication-efficient learning of deep networks from decentralized data, in: Artificial Intelligence and Statistics, PMLR, 2017, pp. 1273–1282.

[24]

Min B., Ross H., Sulem E., Veyseh A.P.B., Nguyen T.H., Sainz O., et al., Recent advances in natural language processing via large pre-trained language models: A survey, 2021, arXiv preprint arXiv:2111.01243.

[25]

Połap D., Woźniak M., Meta-heuristic as manager in federated learning approaches for image processing purposes, Applied Soft Computing 113 (2021).

[26]

Połap D., Woźniak M., A hybridization of distributed policy and heuristic augmentation for improving federated learning approach, Neural Networks 146 (2022) 130–140.

Digital Library

[27]

Qiao B., Zou Z., Huang Y., Fang K., Zhu X., Chen Y., A joint model for entity and relation extraction based on BERT, Neural Computing and Applications 34 (5) (2022) 3471–3481.

[28]

Reimers N., Gurevych I., Sentence-bert: Sentence embeddings using siamese bert-networks, 2019, arXiv preprint arXiv:1908.10084.

[29]

Roth D., Yih W.-t., A linear programming formulation for global inference in natural language tasks, Illinois Univ at Urbana-Champaign Dept of Computer Science, 2004.

[30]

Sattler F., Müller K.-R., Samek W., Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints, IEEE Transactions on Neural Networks and Learning Systems (8) (2020) 3710–3722.

[31]

Shang Y.-M., Huang H., Sun X., Wei W., Mao X.-L., A pattern-aware self-attention network for distant supervised relation extraction, Information Sciences 584 (2022) 269–279.

[32]

Sherstinsky A., Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network, Physica D: Nonlinear Phenomena 404 (2020).

[33]

Sidahmed H., Xu Z., Garg A., Cao Y., Chen M., Efficient and private federated learning with partially trainable networks, 2021, arXiv preprint arXiv:2110.03450.

[34]

Sui, D., Chen, Y., Zhao, J., Jia, Y., Xie, Y., & Sun, W. (2020). Feded: Federated learning via ensemble distillation for medical relation extraction. In Proceedings of the 2020 Conference on empirical methods in natural language processing (pp. 2118–2128).

[35]

Sun Q., Zhang K., Lv L., Li X., Huang K., Zhang T., Joint extraction of entities and overlapping relations by improved graph convolutional networks, Applied Intelligence 52 (5) (2022) 5212–5224.

[36]

Walker C., Strassel S., Medero J., Maeda K., ACE 2005 multilingual training corpus, Linguistic Data Consortium, Philadelphia 57 (2006) 45.

[37]

Wang J., Lu W., Two are better than one: Joint entity and relation extraction with table-sequence encoders, 2020, arXiv preprint arXiv:2010.03851.

[38]

Wang S., Zhang Y., Che W., Liu T., Joint extraction of entities and relations based on a novel graph scheme., in: IJCAI, Yokohama, 2018, pp. 4461–4467.

[39]

Xu Y., Fu L., Lin Z., Qi J., Wang X., INFINITY: A simple yet effective unsupervised framework for graph-text mutual conversion, 2022, arXiv preprint arXiv:2209.10754.

[40]

Yang T.-J., Guliani D., Beaufays F., Motta G., Partial variable training for efficient on-device federated learning, in: ICASSP 2022-2022 IEEE International conference on acoustics, speech and signal processing, IEEE, 2022, pp. 4348–4352.

[41]

Zadeh Nojoo Kambar, M. E., Esmaeilzadeh, A., & Heidari, M. (2022). A Survey on Deep Learning Techniques for Joint Named Entities and Relation Extraction. In 2022 IEEE World AI IoT congress (AIIoT) (pp. 218–224). https://doi.org/10.1109/AIIoT54504.2022.9817231.

[42]

Zeng, X., Zeng, D., He, S., Liu, K., & Zhao, J. (2018). Extracting relational facts by an end-to-end neural model with copy mechanism. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 506–514).

[43]

Zhang Z., Gu Y., Han X., Chen S., Xiao C., Sun Z., et al., Cpm-2: Large-scale cost-effective pre-trained language models, AI Open 2 (2021) 216–224.

[44]

Zhang C., Xie Y., Bai H., Yu B., Li W., Gao Y., A survey on federated learning, Knowledge-Based Systems 216 (2021).

[45]

Zheng H., Wen R., Chen X., Yang Y., Zhang Y., Zhang Z., et al., PRGC: Potential relation and global correspondence based joint relational triple extraction, 2021, arXiv preprint arXiv:2106.09895.

[46]

Zhong Z., Chen D., A frustratingly easy approach for entity and relation extraction, 2020, arXiv preprint arXiv:2010.12812.

[47]

Zhou Y., Pan L., Bai C., Luo S., Wu Z., Self-selective attention using correlation between instances for distant supervision relation extraction, Neural Networks 142 (2021) 213–220.

Digital Library

[48]

Zhu X., Li Z., Wang X., Jiang X., Sun P., Wang X., et al., Multi-modal knowledge graph construction and application: A survey, 2022, arXiv preprint arXiv:2202.05786.

Cited By

Sharma SGuleria K(2024)A comprehensive review on federated learning based models for healthcare applicationsArtificial Intelligence in Medicine10.1016/j.artmed.2023.102691146:COnline publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.artmed.2023.102691
Liu ZLi HWang HLiao YLiu XWu G(2023)A novel pipelined end-to-end relation extraction framework with entity mentions and contextual semantic representationExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.120435228:COnline publication date: 15-Oct-2023
https://dl.acm.org/doi/10.1016/j.eswa.2023.120435

Index Terms

A distributed joint extraction framework for sedimentological entities and relations with federated learning
1. Computing methodologies
  1. Artificial intelligence
  2. Machine learning
2. Information systems
  1. Information systems applications

Index terms have been assigned to the content through auto-classification.

Recommendations

Analysis and robust extraction of changing named entities
NEWS '09: Proceedings of the 2009 Named Entities Workshop: Shared Task on Transliteration

This paper focuses on the change of named entities over time and its influence on the performance of the named entity tagger. First, we analyze Japanese named entities which appear in Mainichi Newspaper articles published in 1995, 1996, 1997, 1998 and ...
Joint Extraction of Nested Entities and Relations Based on Multi-task Learning
Knowledge Science, Engineering and Management
Abstract
Nested named entity recognition and relation extraction are two crucial tasks in information extraction. Traditional systems often treat them as separate, sequential tasks, which can lead to error propagation. To mitigate this issue, we present a ...
NEREL: a Russian information extraction dataset with rich annotation for nested entities, relations, and wikidata entity links
Abstract
This paper describes NEREL—a Russian news dataset suited for three tasks: nested named entity recognition, relation extraction, and entity linking. Compared to flat entities, nested named entities provide a richer and more complete annotation ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Expert Systems with Applications: An International Journal

Expert Systems with Applications: An International Journal Volume 213, Issue PC

Mar 2023

1402 pages

ISSN:0957-4174

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 March 2023

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sharma SGuleria K(2024)A comprehensive review on federated learning based models for healthcare applicationsArtificial Intelligence in Medicine10.1016/j.artmed.2023.102691146:COnline publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.artmed.2023.102691
Liu ZLi HWang HLiao YLiu XWu G(2023)A novel pipelined end-to-end relation extraction framework with entity mentions and contextual semantic representationExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.120435228:COnline publication date: 15-Oct-2023
https://dl.acm.org/doi/10.1016/j.eswa.2023.120435

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents