research-article

Open access

Procedural Text Mining with Large Language Models

Authors:

Jennifer D'SouzaAuthors Info & Claims

K-CAP '23: Proceedings of the 12th Knowledge Capture Conference 2023

Pages 9 - 16

https://doi.org/10.1145/3587259.3627572

Published: 05 December 2023 Publication History

All formats PDF

Abstract

Recent advancements in the field of Natural Language Processing, particularly the development of large-scale language models that are pretrained on vast amounts of knowledge, are creating novel opportunities within the realm of Knowledge Engineering. In this paper, we investigate the usage of large language models (LLMs) in both zero-shot and in-context learning settings to tackle the problem of extracting procedures from unstructured PDF text in an incremental question-answering fashion. In particular, we leverage the current state-of-the-art GPT-4 (Generative Pre-trained Transformer 4) model, accompanied by two variations of in-context learning that involve an ontology with definitions of procedures and steps and a limited number of samples of few-shot learning. The findings highlight both the promise of this approach and the value of the in-context learning customisations. These modifications have the potential to significantly address the challenge of obtaining sufficient training data, a hurdle often encountered in deep learning-based Natural Language Processing techniques for procedure extraction.

References

[1]

Patrizio Bellan, Mauro Dragoni, and Chiara Ghidini. 2021. Process Extraction from Text: state of the art and challenges for the future. CoRR abs/2110.03754 (2021).

[2]

Patrizio Bellan, Mauro Dragoni, and Chiara Ghidini. 2022. Extracting Business Process Entities and Relations from Text Using Pre-trained Language Models and In-Context Learning. In Enterprise Design, Operations, and Computing - 26th International Conference, EDOC 2022, Bozen-Bolzano, Italy, October 3-7, 2022, Proceedings(Lecture Notes in Computer Science, Vol. 13585). Springer, 182–199. https://doi.org/10.1007/978-3-031-17604-3_11

Digital Library

[3]

Piergiorgio Bertoli, Francesco Corcoglioniti, Chiara Di Francescomarino, Mauro Dragoni, Chiara Ghidini, and Marco Pistore. 2022. Semantic modeling and analysis of complex data-aware processes and their executions. Expert Syst. Appl. 198 (2022), 116702.

[4]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022).

[5]

Zihui Dong, Shiladitya Paul, Karl Tassenberg, Geoff Melton, and Hongbiao Dong. 2021. Transformation from human-readable documents and archives in arc welding domain to machine-interpretable data. Comput. Ind. 128 (2021), 103439.

[6]

Daniel Garijo and Yolanda Gil. 2012. Augmenting prov with plans in p-plan: scientific processes as linked data. In 2nd International Workshop on Linked Science (ISWC2012). CEUR Workshop Proceedings.

[7]

Xinyang Geng and Hao Liu. 2023. OpenLLaMA: An Open Reproduction of LLaMA. https://github.com/openlm-research/open_llama

[8]

Mohamad Yaser Jaradeh, Kuldeep Singh, Markus Stocker, Andreas Both, and Sören Auer. 2021. Better Call the Plumber: Orchestrating Dynamic Information Extraction Pipelines. In ICWE(Lecture Notes in Computer Science, Vol. 12706). Springer, 240–254.

[9]

Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, 74–81. https://aclanthology.org/W04-1013

[10]

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688 (2023).

[11]

Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective. arxiv:2301.06627 [cs.CL]

[12]

Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey. ACM Comput. Surv. (jun 2023). https://doi.org/10.1145/3605943

Digital Library

[13]

OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774 [cs.CL]

[14]

Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2023. Unifying Large Language Models and Knowledge Graphs: A Roadmap. arxiv:2306.08302 [cs.CL]

[15]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551.

Digital Library

[16]

Adrian Rebmann and Han van der Aa. 2021. Extracting Semantic Process Information from the Natural Language in Event Logs. In CAiSE, Vol. 12751. Springer, 57–74.

[17]

Anisa Rula, Gloria Re Calegari, Antonia Azzini, Davide Bucci, Alessio Carenini, Ilaria Baroni, and Irene Celino. 2023. K-Hub: A Modular Ontology to Support Document Retrieval and Knowledge Extraction in Industry 5.0. In ESWC 2023 Proceedings(LNCS, Vol. 13870). Springer, 454–470.

Digital Library

[18]

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022).

[19]

Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2023. Head-to-Tail: How Knowledgeable are Large Language Models (LLM)? A.K.A. Will LLMs Replace Knowledge Graphs?arxiv:2308.10168 [cs.CL]

[20]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).

[21]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).

[22]

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021).

[23]

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022).

Cited By

S OP JA JT R K(2024)AI Based Chatbot for Educational Institutions2024 Ninth International Conference on Science Technology Engineering and Mathematics (ICONSTEM)10.1109/ICONSTEM60960.2024.10568662(1-7)Online publication date: 4-Apr-2024
https://doi.org/10.1109/ICONSTEM60960.2024.10568662
Filetti SFenza GGallo A(2024)Research design and writing of scholarly articles: new artificial intelligence tools available for researchersEndocrine10.1007/s12020-024-03977-z85:3(1104-1116)Online publication date: 31-Jul-2024
https://doi.org/10.1007/s12020-024-03977-z
Carriero VAzzini ABaroni IScrocca MCelino I(2024)Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language ModelsKnowledge Engineering and Knowledge Management10.1007/978-3-031-77792-9_26(434-452)Online publication date: 20-Nov-2024
https://doi.org/10.1007/978-3-031-77792-9_26
Show More Cited By

Index Terms

Procedural Text Mining with Large Language Models
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Mining relational data from text: From strictly supervised to weakly supervised learning

This paper approaches the relation classification problem in information extraction framework with different machine learning strategies, from strictly supervised to weakly supervised. A number of learning algorithms are presented and empirically ...
Partially labeled topic models for interpretable text mining
KDD '11: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining

Abstract Much of the world's electronic text is annotated with human-interpretable labels, such as tags on web pages and subject codes on academic publications. Effective text mining in this setting requires models that can flexibly account for the ...
TnT-LLM: Text Mining at Scale with Large Language Models
KDD '24: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Transforming unstructured text into structured and meaningful forms, organized by useful category labels, is a fundamental step in text mining for downstream analysis and application. However, most existing methods for producing label taxonomies and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

K-CAP '23: Proceedings of the 12th Knowledge Capture Conference 2023

December 2023

270 pages

ISBN:9798400701412

DOI:10.1145/3587259

Editors:
Brent Venable
University of West Florida and Institute for Human and Machine Cognition, Pensacola, FL, USA
,
Daniel Garijo
Ontology Engineering Group, Universidad Politécnica de Madrid, Spain
,
Brian Jalaian
University of West Florida and Institute for Human & Machine Cognition, Pensacola, FL, USA

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 December 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

German BMBF project SCINEXT (ID 01lS22070)
MICS (Made in Italy ? Circular and Sustainable) Extended Partnership and received funding from Next-GenerationEU (Italian PNRR ? M4 C2, Invest 1.3 ? D.D. 1551.11-10-2022)

Conference

K-CAP '23

Sponsor:

SIGAI

K-CAP '23: Knowledge Capture Conference 2023

December 5 - 7, 2023

FL, Pensacola, USA

Acceptance Rates

Overall Acceptance Rate 55 of 198 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
569
Total Downloads

Downloads (Last 12 months)569
Downloads (Last 6 weeks)96

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

S OP JA JT R K(2024)AI Based Chatbot for Educational Institutions2024 Ninth International Conference on Science Technology Engineering and Mathematics (ICONSTEM)10.1109/ICONSTEM60960.2024.10568662(1-7)Online publication date: 4-Apr-2024
https://doi.org/10.1109/ICONSTEM60960.2024.10568662
Filetti SFenza GGallo A(2024)Research design and writing of scholarly articles: new artificial intelligence tools available for researchersEndocrine10.1007/s12020-024-03977-z85:3(1104-1116)Online publication date: 31-Jul-2024
https://doi.org/10.1007/s12020-024-03977-z
Carriero VAzzini ABaroni IScrocca MCelino I(2024)Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language ModelsKnowledge Engineering and Knowledge Management10.1007/978-3-031-77792-9_26(434-452)Online publication date: 20-Nov-2024
https://doi.org/10.1007/978-3-031-77792-9_26
Utrilla Guerrero CCorcho OGarijo D(2024)Automated Extraction of Research Software Installation Instructions from README Files: An Initial AnalysisNatural Scientific Language Processing and Research Knowledge Graphs10.1007/978-3-031-65794-8_8(114-133)Online publication date: 26-May-2024
https://dl.acm.org/doi/10.1007/978-3-031-65794-8_8

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents