research-article

PROSPER: Extracting Protocol Specifications Using Large Language Models

Authors:

Prakhar Sharma,

Vinod YegneswaranAuthors Info & Claims

HotNets '23: Proceedings of the 22nd ACM Workshop on Hot Topics in Networks

Pages 41 - 47

https://doi.org/10.1145/3626111.3628205

Published: 28 November 2023 Publication History

Abstract

We explore the application of Large Language Models (LLMs) (specifically GPT-3.5-turbo) to extract specifications and automating understanding of networking protocols from Internet Request for Comments (RFC) documents. LLMs have proven successful in specialized domains like medical and legal text understanding, and this work investigates their potential in automatically comprehending RFCs. We develop Artifact Miner, a tool to extract diagram artifacts from RFCs. We then couple extracted artifacts with natural language text to extract protocol automata using GPT-turbo 3.5 (chatGPT) and present our zero-shot and few-shot extraction results. We call this framework for FSM extraction 'PROSPER: Protocol Specification Miner'. We compare PROSPER with existing state-of-the-art techniques for protocol FSM state and transition extraction. Our experiments indicate that employing artifacts along with text for extraction can lead to lower false positives and better accuracy for both extracted states and transitions. Finally, we discuss efficient prompt engineering techniques, the errors we encountered, and pitfalls of using LLMs for knowledge extraction from specialized domains such as RFC documents.

References

[1]

Ibrahim Abdullah and Daniel Menascé. 2003. Protocol specification and automatic implementation using XML and CBSE. Proceedings of the Second IASTED International Conference on Communications, Internet and Information Technology.

[2]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. CoRR abs/2005.14165 (2020). arXiv:2005.14165 https://arxiv.org/abs/2005.14165

[3]

Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. 2022. Recurrent Memory Transformer. Advances in Neural Information Processing Systems. arXiv:cs.CL/2207.06881

[4]

Chia Yuan Cho, Domagoj Babic, Eui Chul, Richard Shin, and Dawn Song. 2010. Inference and Analysis of Formal Models of Botnet Command and Control Protocols. In ACM Conference on Computer and Communications Security (CCS).

[5]

Chia Yuan Cho, Domagoj Babić, Pongsin Poosankam, Kevin Zhijie Chen, Edward XueJun Wu, and Dawn Song. 2011. MACE: Model-Inference-Assisted Concolic Exploration for Protocol and Vulnerability Discovery. In USENIX Security Symposium.

[6]

Paolo Milani Comparetti, Gilbert Wondracek, Christopher Kruegel&dagger;, and Engin Kirda&Dagger;. 2009. Prospex: Protocol Specification Extraction. In IEEE Symposium on Security and Privacy (SP).

[7]

J.C. Corbett, M.B. Dwyer, J. Hatcliff, S. Laubach, C.S. Pasareanu, and Hongjun Zheng Robby. 2000. Bandera: Extracting Finite-State Models from Java Source Code. In IEEE/ACM International Conference on Software Engineering (ICSE).

Digital Library

[8]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. (2019). arXiv:cs.CL/1810.04805

[9]

Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. (2022). arXiv:cs.CL/2203.05794

[10]

Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 (2022).

[11]

Bin Ji. 2023. VicunaNER: Zero/Few-shot Named Entity Recognition using Vicuna. (2023). arXiv:cs.CL/2305.03253

[12]

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large Language Models are Zero-Shot Reasoners. (2023). arXiv:cs.CL/2205.11916

[13]

Nupur Kothari, Todd Millstein, and Ramesh Govindan. 2008. Deriving State Machines from TinyOS Programs Using Symbolic Execution. In International Conference on Information Processing in Sensor Networks (IPSN).

Digital Library

[14]

Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. (2023).

[15]

David Lie, Andy Chou, Dawson Engler, and David L. Dill. 2001. A Simple Method for Extracting Models from Protocol Code. In IEEE International Symposium on Computer Architecture (ISCA).

[16]

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv:cs.CL/2203.02155

[17]

Maria Lenore Pacheco, Max von Hippel, Ben Weintraub, Dan Goldwasser, and Cristina Nita-Rotaru. 2022. Automated Attack Synthesis by Extracting Finite State Machines from Protocol Specification Documents. (2022).

[18]

Rahul Pandita, Xusheng Xiao, Wei Yang, William Enck, and Tao Xie. 2013. WHYPER: Towards Automating Risk Assessment of Mobile Applications. (2013).

[19]

Panupong Pasupat and Percy Liang. 2015. Compositional Semantic Parsing on Semi-Structured Tables. (July 2015), 1470--1480. https://doi.org/10.3115/v1/P15-1142

[20]

István Pelle, Felicián Németh, and András Gulyás. 2017. A Little Less Interaction, A Little More Action: A Modular Framework for Network Troubleshooting. CoRR abs/1702.08827 (2017). arXiv:1702.08827 http://arxiv.org/abs/1702.08827

[21]

Chenxiong Qian, Hong Hu, Mansour Alharthi, Pak Ho Chung, Taesoo Kim, and Wenke Lee. 2019. Razor: A Framework for Post-deployment Software Debloating. (2019).

[22]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. (2023). arXiv:cs.CL/2302.13971

[23]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems. 5998--6008.

[24]

Max von Hippel, Cole Vick, Stavros Tripakis, and Cristina Nita-Rotaru. 2022. Automated Attacker Synthesis for Distributed Protocols. (2022). arXiv:cs.CR/2004.01220

[25]

Yipeng Wang, Zhibin Zhang, Buyun Qu Danfeng (Daphne) Yao, and Li Guo. 2011. Inferring Protocol State Machine from Network Traces: A Probabilistic Approach. In Applied Cryptography and Network Security (ACNS).

[26]

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent Abilities of Large Language Models. (2022). arXiv:cs.CL/2206.07682

[27]

Edmund Wong, Lei Zhang, Song Wang, Taiyue Liu, and Lin Tan. 2015. DASE: Document-Assisted Symbolic Execution for Improving Automated Software Testing. In ACM/IEEE International Conference on Software Engineering (ICSE).

[28]

Hui Yang, Sifu Yue, and Yunzhong He. 2023. Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions. (2023). arXiv:cs.AI/2306.02224

[29]

Jane Yen, Tamás Lévai, Qinyuan Ye, Xiang Ren, Ramesh Govindan, and Barath Raghavan. 2020. Semi-Automated Protocol Disambiguation and Code Generation. Special Interest Group on Data Communication (SIGCOMM)aa (2020).

[30]

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large Language Models Are Human-Level Prompt Engineers. (2022). arXiv:cs.LG/2211.01910

Cited By

Hollósi GFiczere DVarga P(2024)Generative AI for Low-Level NETCONF Configuration in Network Management Based on YANG Models2024 20th International Conference on Network and Service Management (CNSM)10.23919/CNSM62983.2024.10814410(1-7)Online publication date: 28-Oct-2024
https://doi.org/10.23919/CNSM62983.2024.10814410
Ma RQiu LHu W(2024)SurfOS: Towards an Operating System for Programmable Radio EnvironmentsProceedings of the 23rd ACM Workshop on Hot Topics in Networks10.1145/3696348.3696861(132-141)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3696348.3696861
Arkko JLindbo DKlitte M(2024)Do Large Language Models Dream of Sockets?Proceedings of the 2024 Applied Networking Research Workshop10.1145/3673422.3674900(103-105)Online publication date: 23-Jul-2024
https://dl.acm.org/doi/10.1145/3673422.3674900
Show More Cited By

Index Terms

PROSPER: Extracting Protocol Specifications Using Large Language Models
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Information extraction
      2. Summarization
2. Networks
  1. Network protocols
    1. Network protocol design
    2. Protocol correctness
      1. Protocol testing and verification

Recommendations

Semi-automated protocol disambiguation and code generation
SIGCOMM '21: Proceedings of the 2021 ACM SIGCOMM 2021 Conference

For decades, Internet protocols have been specified using natural language. Given the ambiguity inherent in such text, it is not surprising that protocol implementations have long exhibited bugs. In this paper, we apply natural language processing (NLP) ...
Characterising the IETF through the lens of RFC deployment
IMC '21: Proceedings of the 21st ACM Internet Measurement Conference

Protocol standards, defined by the Internet Engineering Task Force (IETF), are crucial to the successful operation of the Internet. This paper presents a large-scale empirical study of IETF activities, with a focus on understanding collaborative ...
An efficient method for protocol conversion
ICCCN '95: Proceedings of the 4th International Conference on Computer Communications and Networks

Abstract: We propose an efficient algorithm for constructing optimized protocol converters to achieve interoperability between heterogeneous computer networks. This method first generates constraints from existing protocols and imposes them to channel ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

HotNets '23: Proceedings of the 22nd ACM Workshop on Hot Topics in Networks

November 2023

306 pages

ISBN:9798400704154

DOI:10.1145/3626111

Copyright © 2023 ACM.

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 November 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Office of Naval Research

Conference

HotNets '23

Sponsor:

SIGCOMM

HotNets '23: The 22nd ACM Workshop on Hot Topics in Networks

November 28 - 29, 2023

MA, Cambridge, USA

Acceptance Rates

Overall Acceptance Rate 110 of 460 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
376
Total Downloads

Downloads (Last 12 months)306
Downloads (Last 6 weeks)40

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hollósi GFiczere DVarga P(2024)Generative AI for Low-Level NETCONF Configuration in Network Management Based on YANG Models2024 20th International Conference on Network and Service Management (CNSM)10.23919/CNSM62983.2024.10814410(1-7)Online publication date: 28-Oct-2024
https://doi.org/10.23919/CNSM62983.2024.10814410
Ma RQiu LHu W(2024)SurfOS: Towards an Operating System for Programmable Radio EnvironmentsProceedings of the 23rd ACM Workshop on Hot Topics in Networks10.1145/3696348.3696861(132-141)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3696348.3696861
Arkko JLindbo DKlitte M(2024)Do Large Language Models Dream of Sockets?Proceedings of the 2024 Applied Networking Research Workshop10.1145/3673422.3674900(103-105)Online publication date: 23-Jul-2024
https://dl.acm.org/doi/10.1145/3673422.3674900
Wang CScazzariello MFarshin AFerlin SKostić DChiesa M(2024)NetConfEval: Can LLMs Facilitate Network Configuration?Proceedings of the ACM on Networking10.1145/36562962:CoNEXT2(1-25)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3656296
Tóthfalusi TCsiszár ZVarga P(2024)Utilizing Generative AI for Test Data Generation - use-cases for IoT and 5G Core SignalingNOMS 2024-2024 IEEE Network Operations and Management Symposium10.1109/NOMS59830.2024.10574974(1-6)Online publication date: 6-May-2024
https://doi.org/10.1109/NOMS59830.2024.10574974
Aykurt KBlenk AKellerer W(2024)NetLLMBench: A Benchmark Framework for Large Language Models in Network Configuration Tasks2024 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN)10.1109/NFV-SDN61811.2024.10807499(1-6)Online publication date: 5-Nov-2024
https://doi.org/10.1109/NFV-SDN61811.2024.10807499
Wang YKang XLi T(2024)Toward an Open Trust Establishment Infrastructure for Future Network: Motivations, Models, and TechnologiesIEEE Access10.1109/ACCESS.2024.343968912(111196-111205)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3439689
Jiao LShao YSun LLiu FYang SMa WLi LLiu XHou BZhang XShang RLi YWang STang XGuo Y(2024)Advanced Deep Learning Models for 6G: Overview, Opportunities, and ChallengesIEEE Access10.1109/ACCESS.2024.341890012(133245-133314)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3418900

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents