research-article

Open access

Enhancing REST API Testing with NLP Techniques

Authors:

Davide Corradini,

Alessandro Orso,

Michele Pasqua,

Rachel Tzoref-Brill,

Mariano CeccatoAuthors Info & Claims

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 1232 - 1243

https://doi.org/10.1145/3597926.3598131

Published: 13 July 2023 Publication History

Abstract

RESTful services are commonly documented using OpenAPI specifications. Although numerous automated testing techniques have been proposed that leverage the machine-readable part of these specifications to guide test generation, their human-readable part has been mostly neglected. This is a missed opportunity, as natural language descriptions in the specifications often contain relevant information, including example values and inter-parameter dependencies, that can be used to improve test generation. In this spirit, we propose NLPtoREST, an automated approach that applies natural language processing techniques to assist REST API testing. Given an API and its specification, NLPtoREST extracts additional OpenAPI rules from the human-readable part of the specification. It then enhances the original specification by adding these rules to it. Testing tools can transparently use the enhanced specification to perform better test case generation. Because rule extraction can be inaccurate, due to either the intrinsic ambiguity of natural language or mismatches between documentation and implementation, NLPtoREST also incorporates a validation step aimed at eliminating spurious rules. We performed studies to assess the effectiveness of our rule extraction and validation approach, and the impact of enhanced specifications on the performance of eight state-of-the-art REST API testing tools. Our results are encouraging and show that NLPtoREST can extract many relevant rules with high accuracy, which can in turn significantly improve testing tools’ performance.

References

[1]

J. C. Alonso, A. Martin-Lopez, S. Segura, J. Garcia, and A. Ruiz-Cortes. 2023. ARTE: Automated Generation of Realistic Test Inputs for Web APIs. IEEE Transactions on Software Engineering, 49, 01 (2023), jan, 348–363. issn:1939-3520 https://doi.org/10.1109/TSE.2022.3150618

[2]

Andrea Arcuri. 2019. RESTful API Automated Test Case Generation with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM), 28, 1 (2019), Article 3, jan, 37 pages. issn:1049-331X https://doi.org/10.1145/3293455

Digital Library

[3]

Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. In Proceedings of the 41st International Conference on Software Engineering (ICSE ’19). IEEE Press, Piscataway, NJ, USA. 748–758. https://doi.org/10.1109/ICSE.2019.00083

Digital Library

[4]

Mourad Badri, Linda Badri, and Marius Naha. 2004. A Use Case Driven Testing Process: Towards a Formal Approach Based on UML Collaboration Diagrams. In Formal Approaches to Software Testing, Alexandre Petrenko and Andreas Ulrich (Eds.). Springer Berlin Heidelberg, Berlin, Germany. 223–235. isbn:978-3-540-24617-6 https://doi.org/10.1007/978-3-540-24617-6_16

[5]

Christian Bizer, Jens Lehmann, Georgi Kobilarov, Sören Auer, Christian Becker, Richard Cyganiak, and Sebastian Hellmann. 2009. Dbpedia-a crystallization point for the web of data. Journal of web semantics, 7, 3 (2009), 154–165.

Digital Library

[6]

Arianna Blasi, Alberto Goffi, Konstantin Kuznetsov, Alessandra Gorla, Michael D. Ernst, Mauro Pezze, and Sergio Delgado Castellanos. 2018. Translating Code Comments to Procedure Specifications. In Proceedings of the 2018 International Symposium on Software Testing and Analysis (ISSTA 2018) (ISSTA ’18). Association for Computing Machinery, New York, NY, USA. 242–253. isbn:9781450356992 https://doi.org/10.1145/3213846.3213872

Digital Library

[7]

Arianna Blasi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezze. 2022. Call Me Maybe: Using NLP to Automatically Generate Unit Test Cases Respecting Temporal Constraints. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE ’22). Association for Computing Machinery, New York, NY, USA. Article 19, 11 pages. isbn:9781450394758 https://doi.org/10.1145/3551349.3556961

Digital Library

[8]

Hanyang Cao, Jean-Rémy Falleri, and Xavier Blanc. 2017. Automated Generation of REST API Specification from Plain HTML Documentation. In Service-Oriented Computing, Michael Maximilien, Antonio Vallecillo, Jianmin Wang, and Marc Oriol (Eds.). Springer International Publishing, New York, NY, USA. 453–461. isbn:978-3-319-69035-3 https://doi.org/10.1007/978-3-319-69035-3_32

Digital Library

[9]

Davide Corradini, Amedeo Zampieri, Michele Pasqua, Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2022. Automated black-box testing of nominal and error scenarios in RESTful APIs. Software Testing, Verification and Reliability, 32 (2022), 01, https://doi.org/10.1002/stvr.1808

[10]

Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-Based Software Architectures. Ph. D. Dissertation. University of California, Irvine.

[11]

The Linux Foundation. 2022. OpenAPI specification. https://spec.openapis.org/oas/v3.1.0

[12]

Yoav Goldberg and Omer Levy. 2014. word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arxiv:1402.3722.

[13]

Google. 2023. Google Bard. https://bard.google.com/

[14]

Nitin Hardeniya, Jacob Perkins, Deepti Chopra, Nisheeth Joshi, and Iti Mathur. 2016. Natural language processing: python and NLTK. Packt Publishing Ltd, Birmingham, UK.

[15]

Zac Hatfield-Dodds and Dmitry Dygalo. 2022. Deriving Semantics-Aware Fuzzers from Web API Schemas. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 345–346. isbn:9781450392235 https://doi.org/10.1145/3510454.3528637

Digital Library

[16]

Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. FastText.zip: Compressing text classification models. arxiv:1612.03651.

[17]

Daniel Jurafsky and James H. Martin. 2021. Speech and Language Processing: Constituency Parsing. https://web.stanford.edu/ jurafsky/slp3/13.pdf

[18]

Myeongsoo Kim, Qi Xin, Saurabh Sinha, and Alessandro Orso. ’22. Automated test generation for REST APIs: no time to rest yet. In ISSTA ’22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, South Korea, July 18 - 22, 2022, Sukyoung Ryu and Yannis Smaragdakis (Eds.). ACM, New York, NY, USA. 289–301. https://doi.org/10.1145/3533767.3534401

Digital Library

[19]

Kerry Kimbrough. 2023. Tcases. https://github.com/Cornutum/tcases

[20]

Dan Klein and Christopher D. Manning. 2003. Accurate Unlexicalized Parsing. In Proceedings of the 41st annual meeting of the association for computational linguistics. Association for Computational Linguistics, Edinburgh, Scotland. 423–430.

[21]

LanguageTool. 2023. LanguageTool REST API. https://languagetool.org/proofreading-api

[22]

Nuno Laranjeiro, João Agnelo, and Jorge Bernardino. 2021. A Black Box Tool for Robustness Testing of REST Services. IEEE Access, 9 (2021), 24738–24754. https://doi.org/10.1109/ACCESS.2021.3056505

[23]

Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. 2022. Morest: Model-Based RESTful API Testing with Execution Feedback. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 1406–1417. isbn:9781450392211 https://doi.org/10.1145/3510003.3510133

Digital Library

[24]

Bogdan Marculescu, Man Zhang, and Andrea Arcuri. 2022. On the Faults Found in REST APIs by Automated Test Generation. ACM Trans. Softw. Eng. Methodol., 31, 3 (2022), Article 41, mar, 43 pages. https://doi.org/10.1145/3491038

Digital Library

[25]

Alberto Martin-Lopez, Sergio Segura, Carlos Müller, and Antonio Ruiz-Cortés. 2022. Specification and Automated Analysis of Inter-Parameter Dependencies in Web APIs. IEEE Transactions on Services Computing, 15, 4 (2022), 2342–2355. https://doi.org/10.1109/TSC.2021.3050610

[26]

Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. A Catalogue of Inter-Parameter Dependencies in RESTful Web APIs. In Service-Oriented Computing: 17th International Conference, ICSOC 2019, Toulouse, France, October 28–31, 2019, Proceedings. Springer-Verlag, Berlin, Heidelberg. 399–414. isbn:978-3-030-33701-8 https://doi.org/10.1007/978-3-030-33702-5_31

Digital Library

[27]

Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2021. RESTest: Automated Black-Box Testing of RESTful Web APIs. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021). Association for Computing Machinery, New York, NY, USA. 682–685. isbn:9781450384599 https://doi.org/10.1145/3460319.3469082

Digital Library

[28]

Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2022. Online Testing of RESTful APIs: Promises and Challenges. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’22). Association for Computing Machinery, New York, NY, USA. 408–420. isbn:9781450394130 https://doi.org/10.1145/3540250.3549144

Digital Library

[29]

Manish Motwani and Yuriy Brun. 2019. Automatically Generating Precise Oracles from Structured Natural Language Specifications. In Proceedings of the 41st International Conference on Software Engineering (ICSE ’19). IEEE Press, Piscataway, NJ, USA. 188–199. https://doi.org/10.1109/ICSE.2019.00035

Digital Library

[30]

Myeongsoo Kim and Davide Corradini. 2023. Experiment infrastructure and data for NLPtoREST. https://github.com/codingsoo/nlp2rest

[31]

OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774.

[32]

OpenAPI. 2023. OpenAPI standard. https://www.openapis.org

[33]

Johannes Ryser and Martin Glinz. 1999. A scenario-based approach to validating and testing software systems using statecharts. https://doi.org/10.5167/uzh-205008

[34]

SmartBear Software. 2023. OpenAPI Extensions. https://swagger.io/docs/specification/openapi-extensions/

[35]

EclEmma Team. 2023. JaCoCo. https://www.eclemma.org/jacoco/

[36]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arxiv:2302.13971.

[37]

Chunhui Wang, Fabrizio Pastore, and Lionel Briand. 2018. Automated Generation of Constraints from Use Case Specifications to Support System Testing. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). IEEE, Piscataway, NJ, USA. 23–33. https://doi.org/10.1109/ICST.2018.00013

[38]

Chunhui Wang, Fabrizio Pastore, Arda Goknil, and Lionel C. Briand. 2022. Automatic Generation of Acceptance Test Cases From Use Case Specifications: An NLP-Based Approach. IEEE Transactions on Software Engineering, 48, 2 (2022), 585–616. https://doi.org/10.1109/TSE.2020.2998503

[39]

Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial Testing of RESTful APIs. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 426–437. isbn:9781450392211 https://doi.org/10.1145/3510003.3510151

Digital Library

[40]

Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial Testing of RESTful APIs. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 426–437. isbn:9781450392211 https://doi.org/10.1145/3510003.3510151

Digital Library

[41]

Jinqiu Yang, Erik Wittern, Annie T. T. Ying, Julian Dolby, and Lin Tan. 2018. Towards Extracting Web API Specifications from Documentation. In Proceedings of the 15th International Conference on Mining Software Repositories (MSR ’18). Association for Computing Machinery, New York, NY, USA. 454–464. isbn:9781450357166 https://doi.org/10.1145/3196398.3196411

Digital Library

Cited By

Küçük EBalıkçı Çiçek İKüçükakçalı ZYetiş CÇolak C(2024)A Developed Graphical User Interface-Based on Different Generative Pre-trained Transformers ModelsODÜ Tıp Dergisi10.56941/odutip.141359711:1(18-32)Online publication date: 30-Apr-2024
https://doi.org/10.56941/odutip.1413597
Kanwal SRaza ABai CZhang DWenn JKumar D(2024)An Effective Machine Learning Approach with Hyper-parameter Tuning for Sentiment AnalysisData Intelligence10.3724/2096-7004.di.2024.0060Online publication date: 1-Nov-2024
https://doi.org/10.3724/2096-7004.di.2024.0060
Chen JChen YPan ZChen YLi YLi YZhang MShen Y(2024)DynER: Optimized Test Case Generation for Representational State Transfer (REST)ful Application Programming Interface (API) Fuzzers Guided by Dynamic Error ResponsesElectronics10.3390/electronics1317347613:17(3476)Online publication date: 1-Sep-2024
https://doi.org/10.3390/electronics13173476
Show More Cited By

Index Terms

Enhancing REST API Testing with NLP Techniques
1. Information systems
  1. World Wide Web
    1. Web services
      1. RESTful web services
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Leveraging Large Language Models to Improve REST API Testing
ICSE-NIER'24: Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results

The widespread adoption of REST APIs, coupled with their growing complexity and size, has led to the need for automated REST API testing tools. Current tools focus on the structured data in REST API specifications but often neglect valuable insights ...
Adaptive REST API Testing with Reinforcement Learning
ASE '23: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering

Modern web services increasingly rely on REST APIs. Effectively testing these APIs is challenging due to the vast search space to be explored, which involves selecting API operations for sequence creation, choosing parameters for each operation from a ...
Generating REST API Specifications through Static Analysis
ICSE '24: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

Web Application Programming Interfaces (APIs) allow services to be accessed over the network. RESTful (or REST) APIs, which use the REpresentation State Transfer (REST) protocol, are a popular type of web API. To use or test REST APIs, developers use ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2023

1554 pages

ISBN:9798400702211

DOI:10.1145/3597926

General Chair:
René Just
University of Washington, USA
,
Program Chair:
Gordon Fraser
University of Passau, Germany

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Union's Horizon Europe research and innovation programme
Italian Ministry of University and Research - PON programme for Research and Innovation (Action IV.6)
NSF (National Science Foundation)
DARPA
DOE U.S. Department of Energy

Conference

ISSTA '23

Sponsor:

SIGSOFT

ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 17 - 21, 2023

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
1,610
Total Downloads

Downloads (Last 12 months)1,184
Downloads (Last 6 weeks)143

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Küçük EBalıkçı Çiçek İKüçükakçalı ZYetiş CÇolak C(2024)A Developed Graphical User Interface-Based on Different Generative Pre-trained Transformers ModelsODÜ Tıp Dergisi10.56941/odutip.141359711:1(18-32)Online publication date: 30-Apr-2024
https://doi.org/10.56941/odutip.1413597
Kanwal SRaza ABai CZhang DWenn JKumar D(2024)An Effective Machine Learning Approach with Hyper-parameter Tuning for Sentiment AnalysisData Intelligence10.3724/2096-7004.di.2024.0060Online publication date: 1-Nov-2024
https://doi.org/10.3724/2096-7004.di.2024.0060
Chen JChen YPan ZChen YLi YLi YZhang MShen Y(2024)DynER: Optimized Test Case Generation for Representational State Transfer (REST)ful Application Programming Interface (API) Fuzzers Guided by Dynamic Error ResponsesElectronics10.3390/electronics1317347613:17(3476)Online publication date: 1-Sep-2024
https://doi.org/10.3390/electronics13173476
Zhang MArcuri ATeng PXue KWang WFilkov VRay BZhou M(2024)Seeding and Mocking in White-Box Fuzzing Enterprise RPC APIs: An Industrial Case StudyProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695265(2024-2034)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695265
Arcuri AZhang MGaleotti J(2024)Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIsACM Transactions on Software Engineering and Methodology10.1145/365215733:6(1-36)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3652157
Decrop AChristakis MPradel M(2024)Leveraging Natural Language Processing and Data Mining to Augment and Validate APIsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3685554(1906-1908)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3685554
Kim MStennett TShah DSinha SOrso ARoychoudhury APaiva AAbreu RStorey MHierons RMadeira H(2024)Leveraging Large Language Models to Improve REST API TestingProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results10.1145/3639476.3639769(37-41)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639476.3639769
Le TTran TCao DLe VNguyen TNguyen V(2024)KAT: Dependency-Aware Automated API Testing with Large Language Models2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00017(82-92)Online publication date: 27-May-2024
https://doi.org/10.1109/ICST60714.2024.00017
Baumann CKoroglu YWotawa F(2024)On the Impact of Input Models on the Fault Detection Capabilities of Combinatorial TestingSN Computer Science10.1007/s42979-024-03134-35:7Online publication date: 27-Aug-2024
https://dl.acm.org/doi/10.1007/s42979-024-03134-3
Kim MSinha SOrso A(2023)Adaptive REST API Testing with Reinforcement Learning2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00218(446-458)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00218

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents