short-paper

Open access

SBFT Tool Competition 2024 - CPS-UAV Test Case Generation Track

Authors:

Prasun Saurabh,

Timothy Zimmermann,

Charith Munasinghe,

Christian Birchler,

Sebastiano PanichellaAuthors Info & Claims

SBFT '24: Proceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing

Pages 29 - 32

https://doi.org/10.1145/3643659.3643931

Published: 10 September 2024 Publication History

Abstract

While simulation-based testing is critical for ensuring the safety of autonomous Unmanned Aerial Vehicles (UAVs), it has not been adequately researched yet. The UAV Testing Competition organized by the Search-Based and Fuzz Testing (SBFT) workshop is an initiative designed to inspire and encourage the software testing Community to direct their attention toward UAVs as a rapidly emerging and crucial domain. It provides a simple software platform and case study to facilitate their onboarding in the UAV domain and help them develop their first test generation tools for UAVs.

In this first edition of the competition, 7 tools were submitted, evaluated, and compared extensively against each other and the baseline approach. We evaluated their test generation performance for 6 different case studies using our novel benchmarking infrastructure. The generated test suites were scored and ranked based on the number and severity of the revealed faults, and the complexity, diversity, and execution time of the test cases. This paper describes the competition context, its platform, the competing tools, and the evaluation process and results.

References

[1]

Matteo Biagiola, Stefan Klikovits, Jarkko Peltomäki, and Vincenzo Riccio. 2023. SBFT Tool Competition 2023 - Cyber-Physical Systems Track. In International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE. IEEE, 45--48.

[2]

Andrea Di Sorbo, Fiorella Zampetti, Corrado A. Visaggio, Massimiliano Di Penta, and Sebastiano Panichella. 2022. Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software Platforms. ACM Trans. Softw. Eng. Methodol. (Sept. 2022). Place: New York, NY, USA Publisher: Association for Computing Machinery.

Digital Library

[3]

Nicolas Erni, Al-Ameen Mohammed Ali Mohammed, Christian Birchler, Pouria Derakhshanfar, Stephan Lukasczyk, and Sebastiano Panichella. 2024. SBFT Tool Competition 2024 - Python Test Case Generation Track. In IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2024.

[4]

Dmytro Humeniuk and Foutse Khomh. 2024. AmbieGen at the SBFT 2024 Tool Competition - CPS-UAV Track. In IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2024.

Digital Library

[5]

Gunel Jahangirova and Valerio Terragni. 2023. SBFT Tool Competition 2023 - Java Test Case Generation Track. In International Workshop on Search-Based and Fuzz Testing. IEEE, 61--64.

[6]

Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2023. Simulation-based test case generation for unmanned aerial vehicles in the neighborhood of real flights. In International Conference on Software Testing, Verification and Validation.

[7]

Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2024. Simulation-based Testing of Unmanned Aerial Vehicles with Aerialist. In International Conference on Software Engineering (ICSE).

[8]

Sajad Khatiri, Prasun Saurabh, Timothy Zimmermann, Charith Munasinghe, Christian Birchler, and Sebastiano Panichella. 2024. SBFT 2024 UAV Test Case Generation Competition: Evaluation Artifacts".

[9]

Marco De Liso and Zhi Wen Soi. 2024. CAMBA CPS-UAV at the SBFT Tool Competition 2024. In IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2024.

[10]

Lorenz Meier, Dominik Honegger, and Marc Pollefeys. 2015. PX4: A node-based multithreaded open source robotics framework for deeply embedded platforms. In international conference on robotics and automation. IEEE, 6235--6240.

[11]

Sebastiano Panichella, Alessio Gambi, Fiorella Zampetti, and Vincenzo Riccio. 2021. SBST Tool Competition 2021. In International Workshop on Search-Based Software Testing. IEEE, 20--27.

[12]

Shuncheng Tang, Zhenya Zhang, Ahmet Cetinkaya, and Paolo Arcaini. 2024. TUMB at the SBFT 2024 Tool Competition - CPS-UAV Test Case Generation Track. In IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2024.

Digital Library

[13]

Christopher Steven Timperley, Afsoon Afzal, Deborah S Katz, Jam Marcos Hernandez, and Claire Le Goues. 2018. Crashing simulated planes is cheap: Can simulation detect robotics bugs early?. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 331--342.

[14]

Jesper Winsten, Valentin Soloviev, Jarkko Peltomäki, and Ivan Porres. 2024. Adaptive test generation for unmanned aerial vehicles using WOGAN-UAV. In IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2024.

Digital Library

[15]

Fiorella Zampetti, Ritu Kapur, Massimiliano Di Penta, and Sebastiano Panichella. 2022. An empirical characterization of software bugs in open-source Cyber-Physical Systems. Journal of Systems and Software 192 (2022), 111425.

Digital Library

[16]

Taohong Zhu, William Newton, Suzanne Embury, and Youcheng Sun. 2024. TAI-iST CPS-UAV at the SBFT Tool Competition 2024. In IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2024.

[17]

Tahereh Zohdinasab and Andrea Doreste. 2024. DeepHyperion-UAV at the SBFT 2024 Tool Competition - CPS-UAV Test Case Generation Track. In IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2024.

Digital Library

Cited By

Birchler CKhatiri SRani PKehrer TPanichella S(2025)A Roadmap for Simulation-Based Testing of Autonomous Cyber-Physical Systems: Challenges and Future DirectionACM Transactions on Software Engineering and Methodology10.1145/3711906Online publication date: 16-Jan-2025
https://doi.org/10.1145/3711906
Humeniuk DBen Braiek HReid TKhomh FFilkov VRay BZhou M(2024)In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic ManipulatorsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695281(2187-2198)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695281
Khatiri SDi Sorbo AZampetti FVisaggio CDi Penta MPanichella S(2024)Identifying safety–critical concerns in unmanned aerial vehicle software platforms with SALIENTSoftwareX10.1016/j.softx.2024.10174827(101748)Online publication date: Sep-2024
https://doi.org/10.1016/j.softx.2024.101748
Show More Cited By

Index Terms

SBFT Tool Competition 2024 - CPS-UAV Test Case Generation Track

Index terms have been assigned to the content through auto-classification.

Recommendations

SBFT Tool Competition 2024 - Python Test Case Generation Track
SBFT '24: Proceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing

Test case generation (TCG) for Python poses distinctive challenges due to the language's dynamic nature and the absence of strict type information. Previous research has successfully explored automated unit TCG for Python, with solutions outperforming ...
DeepHyperion-UAV at the SBFT Tool Competition 2024 - CPS-UAV Test Case Generation Track
SBFT '24: Proceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing

Unmanned Aerial Vehicles (UAV) have showcased the potential for autonomous flights in real-world settings, sparking considerable interest across various application domains. Ensuring the robustness and safety of UAV operations in real-world scenarios ...
SBST tool competition 2022
SBST '22: Proceedings of the 15th Workshop on Search-Based Software Testing

We report on the organization, challenges, and results of the tenth edition of the Java Unit Testing Competition as well as the second edition of the Cyber-Physical Systems (CPS) Testing Competition.

Java Unit Testing Competition. Seven tools, i.e., BBC, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SBFT '24: Proceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing

April 2024

84 pages

ISBN:9798400705625

DOI:10.1145/3643659

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

In-Cooperation

Faculty of Engineering of University of Porto

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 September 2024

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

European Commission

Conference

SBFT '24

Sponsor:

SIGSOFT

SBFT '24: 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing

April 14, 2024

Lisbon, Portugal

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
210
Total Downloads

Downloads (Last 12 months)210
Downloads (Last 6 weeks)40

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Birchler CKhatiri SRani PKehrer TPanichella S(2025)A Roadmap for Simulation-Based Testing of Autonomous Cyber-Physical Systems: Challenges and Future DirectionACM Transactions on Software Engineering and Methodology10.1145/3711906Online publication date: 16-Jan-2025
https://doi.org/10.1145/3711906
Humeniuk DBen Braiek HReid TKhomh FFilkov VRay BZhou M(2024)In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic ManipulatorsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695281(2187-2198)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695281
Khatiri SDi Sorbo AZampetti FVisaggio CDi Penta MPanichella S(2024)Identifying safety–critical concerns in unmanned aerial vehicle software platforms with SALIENTSoftwareX10.1016/j.softx.2024.10174827(101748)Online publication date: Sep-2024
https://doi.org/10.1016/j.softx.2024.101748
Panichella AOlsthoorn M(2024)Higher Fault Detection Through Novel Density Estimators in Unit Test GenerationSearch-Based Software Engineering10.1007/978-3-031-64573-0_2(18-32)Online publication date: 2-Jul-2024
https://doi.org/10.1007/978-3-031-64573-0_2

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten