article

AmbieGen: : A search-based framework for autonomous systems testing▪

Authors:

Dmytro Humeniuk,

Giuliano AntoniolAuthors Info & Claims

Volume 230, Issue C

https://doi.org/10.1016/j.scico.2023.102990

Published: 01 August 2023 Publication History

Abstract

Thorough testing of safety-critical autonomous systems, such as self-driving cars, autonomous robots, and drones, is essential for detecting potential failures before deployment. One crucial testing stage is model-in-the-loop testing, where the system model is evaluated by executing various scenarios in a simulator. However, the search space of possible parameters defining these test scenarios is vast, and simulating all combinations is computationally infeasible. To address this challenge, we introduce AmbieGen, a search-based test case generation framework for autonomous systems. AmbieGen uses evolutionary search to identify the most critical scenarios for a given system, and has a modular architecture that allows for the addition of new systems under test, algorithms, and search operators. Currently, AmbieGen supports test case generation for autonomous robots and autonomous car lane keeping assist systems. In this paper, we provide a high-level overview of the framework's architecture and demonstrate its practical use cases.

Highlights

•

AmbieGen is an evolutionary algorithm based test scenario generation tool.

•

The search algorithm maximizes the difficulty of test scenarios as well as their diversity.

•

The tool is customizable and can be used to test different robotic systems.

•

Current tool version includes test scenario generation for autonomous vehicles and mobile robots.

•

The tool can be accessed at: https://github.com/swat-lab-optimization/AmbieGen-tool.

References

[1]

D. Bruggner, A. Hegde, F.S. Acerbo, D. Gulati, T.D. Son, Model in the loop testing and validation of embedded autonomous driving algorithms, in: 2021 IEEE Intelligent Vehicles Symposium (IV), IEEE, 2021, pp. 136–141.

[2]

Y. Annpureddy, C. Liu, G. Fainekos, S. Sankaranarayanan, S-taliro: a tool for temporal logic falsification for hybrid systems, in: International Conference on Tools and Algorithms for the Construction and Analysis of Systems, Springer, 2011, pp. 254–257.

[3]

A. Donzé, Breach, a toolbox for verification and parameter synthesis of hybrid systems, in: International Conference on Computer Aided Verification, Springer, 2010, pp. 167–170.

[4]

C. Menghi, S. Nejati, L. Briand, Y.I. Parache, Approximation-refinement testing of compute-intensive cyber-physical models: an approach based on system identification, in: 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), IEEE, 2020, pp. 372–384.

[5]

A. Gambi, M. Mueller, G. Fraser, Automatically testing self-driving cars with search-based procedural content generation, in: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, IEEE, 2019, pp. 318–328.

[6]

E. Castellano, A. Cetinkaya, C.H. Thanh, S. Klikovits, X. Zhang, P. Arcaini, Frenetic at the sbst 2021 tool competition, in: 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST), IEEE, 2021, pp. 36–37.

[7]

V. Riccio, P. Tonella, Model-based exploration of the frontier of behaviours for deep learning system testing, in: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 876–888.

[8]

T. Zohdinasab, V. Riccio, A. Gambi, P. Tonella, Deephyperion: exploring the feature space of deep learning-based systems through illumination search, in: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021, pp. 79–90.

[9]

S. Panichella, A. Gambi, F. Zampetti, V. Riccio, Sbst tool competition 2021, in: 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST), IEEE, 2021, pp. 20–27.

[10]

A. Gambi, G. Jahangirova, V. Riccio, F. Zampetti, Sbst tool competition 2022, in: 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), IEEE, 2022, pp. 25–32.

[11]

D. Humeniuk, F. Khomh, G. Antoniol, A search-based framework for automatic generation of testing environments for cyber–physical systems, Inf. Softw. Technol. 149 (2022),.

Digital Library

[12]

J. Blank, K. Deb, pymoo: multi-objective optimization in python, IEEE Access 8 (2020) 89497–89509.

[13]

D. Humeniuk, G. Antoniol, F. Khomh, Ambiegen tool at the sbst 2022 tool competition, in: 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), IEEE, 2022, pp. 43–46.

[14]

J. Minguez, L. Montano, Nearness diagram (nd) navigation: collision avoidance in troublesome scenarios, IEEE Trans. Robot. Autom. 20 (1) (2004) 45–59.

[15]

M. Kranz, R.B. Rusu, A. Maldonado, M. Beetz, A. Schmidt, A player/stage system for context-aware intelligent environments, Proc. UbiSys 6 (8) (2006) 17–21.

[16]

BeamNG.tech (2021): Beamng gmbh. https://www.beamng.gmbh/research.

[17]

Ding, W.; Xu, C.; Arief, M.; Lin, H.; Li, B.; Zhao, D. (2022): A survey on safety-critical driving scenario generation – a methodological perspective. https://doi.org/10.48550/ARXIV.2202.02215 : A survey on safety-critical driving scenario generation – a methodological perspective. https://arxiv.org/abs/2202.02215.

[18]

Klikovits, S.; Riccio, V.; Castellano, E.; Cetinkaya, A.; Gambi, A.; Arcaini, P. (2022): Does road diversity really matter in testing automated driving systems?–a registered report. arXiv preprint arXiv:2209.05947.

[19]

R.B. Abdessalem, S. Nejati, L.C. Briand, T. Stifter, Testing vision-based control systems using learnable evolutionary algorithms, in: 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), IEEE, 2018, pp. 1016–1026.

[20]

A. Stocco, B. Pulfer, P. Tonella, Mind the gap! A study on the transferability of virtual vs physical-world testing of autonomous driving systems, IEEE Trans. Softw. Eng. (2022).

[21]

A. Stocco, P. Tonella, Confidence-driven weighted retraining for predicting safety-critical failures in autonomous driving systems, J. Softw. Evol. Process (2021).

[22]

A. Arrieta, P. Valle, J.A. Agirre, G. Sagardui, Some seeds are strong: seeding strategies for search-based test case selection, ACM Trans. Softw. Eng. Methodol. (2022).

[23]

J.K. Pugh, L.B. Soros, K.O. Stanley, Quality diversity: a new frontier for evolutionary computation, in: Frontiers in Robotics and AI, 2016, p. 40.

[24]

S. Ramakrishna, B. Luo, Y. Barve, G. Karsai, A. Dubey, Risk-aware scene sampling for dynamic assurance of autonomous systems, in: 2022 IEEE International Conference on Assured Autonomy (ICAA), IEEE, 2022, pp. 107–116.

[25]

K. Pei, Y. Cao, J. Yang, S. Jana, Deepxplore: automated whitebox testing of deep learning systems, in: Proceedings of the 26th Symposium on Operating Systems Principles, 2017, pp. 1–18.

Digital Library

[26]

J. Kim, R. Feldt, S. Yoo, Guiding deep learning system testing using surprise adequacy, in: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), IEEE, 2019, pp. 1039–1049.

[27]

D.J. Fremont, E. Kim, T. Dreossi, S. Ghosh, X. Yue, A.L. Sangiovanni-Vincentelli, S.A. Seshia, Scenic: a language for scenario specification and data generation, Mach. Learn. (2022) 1–45.

[28]

A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, V. Koltun, Carla: an open urban driving simulator, in: Conference on Robot Learning, PMLR, 2017, pp. 1–16.

[29]

G. Rong, B.H. Shin, H. Tabatabaee, Q. Lu, S. Lemke, M. Možeiko, E. Boise, G. Uhm, M. Gerow, S. Mehta, et al., Lgsvl simulator: a high fidelity simulator for autonomous driving, in: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2020, pp. 1–6.

Cited By

Humeniuk DKhomh FAntoniol G(2024)Reinforcement Learning Informed Evolutionary Search for Autonomous Systems TestingACM Transactions on Software Engineering and Methodology10.1145/368046833:8(1-45)Online publication date: 27-Jul-2024
https://dl.acm.org/doi/10.1145/3680468
Humeniuk DKhomh F(2024)AmbieGen at the SBFT 2024 Tool Competition - CPS-UAV TrackProceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing10.1145/3643659.3648552(69-70)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643659.3648552
Sorokin LSafin DNejati S(2024)Can search-based testing with pareto optimization effectively cover failure-revealing test inputs?Empirical Software Engineering10.1007/s10664-024-10564-330:1Online publication date: 16-Nov-2024
https://dl.acm.org/doi/10.1007/s10664-024-10564-3

Recommendations

Rescue Missions Bots using Active SLAM and Map Feature Extraction
ICCMA '16: Proceedings of the 4th International Conference on Control, Mechatronics and Automation

The main objective of our study is to implement a heterogeneous multi-robot system for mapping and exploration. We present a novel approach where the SLAM map is used to speed up the exploration process by extracting points of interest from the map and ...
Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing
Evolutionary search (ES)-based techniques are commonly used for testing autonomous robotic systems. However, these approaches often rely on computationally expensive simulator-based models for test scenario evaluation. To improve the computational ...
A mobile robot with autonomous climbing and descending of stairs

Mobile robots are used to operate in urban environments, for surveillance, reconnaissance, and inspection, as well as for military operations and in hazardous environments. Some are intended for exploration of only natural terrains, but others also for ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Science of Computer Programming

Science of Computer Programming Volume 230, Issue C

Aug 2023

340 pages

ISSN:0167-6423

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier North-Holland, Inc.

United States

Publication History

Published: 01 August 2023

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Humeniuk DKhomh FAntoniol G(2024)Reinforcement Learning Informed Evolutionary Search for Autonomous Systems TestingACM Transactions on Software Engineering and Methodology10.1145/368046833:8(1-45)Online publication date: 27-Jul-2024
https://dl.acm.org/doi/10.1145/3680468
Humeniuk DKhomh F(2024)AmbieGen at the SBFT 2024 Tool Competition - CPS-UAV TrackProceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing10.1145/3643659.3648552(69-70)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643659.3648552
Sorokin LSafin DNejati S(2024)Can search-based testing with pareto optimization effectively cover failure-revealing test inputs?Empirical Software Engineering10.1007/s10664-024-10564-330:1Online publication date: 16-Nov-2024
https://dl.acm.org/doi/10.1007/s10664-024-10564-3

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents