article

Frenetic-lib: : An extensible framework for search-based generation of road structures for ADS testing▪

Authors:

Stefan Klikovits,

Ezequiel Castellano,

Ahmet Cetinkaya,

Paolo ArcainiAuthors Info & Claims

Volume 230, Issue C

https://doi.org/10.1016/j.scico.2023.102996

Published: 01 August 2023 Publication History

Abstract

Being capable of identifying significant safety shortcomings, search-based methods are a core tool for testing automated driving system (ADS) technologies. In this domain, Frenetic has proven to be a popular and very effective tool, searching and identifying diverse sets of roads that point out potentially faulty ADS behaviour. This paper presents Frenetic-lib, a Python library that captures Frenetic's novel combination of road representation and genetic algorithm, and makes it generally available in a customisable way. Next to the capacity to integrate additional ADS simulators, Frenetic-lib further creates new research opportunities on search-based road testing, novel road representations and mutation operators.

Highlights

•

Frenetic-lib is a Python library for the objective-based search of ADS road scenarios.

•

It implements Frenetic (a combination of road representation and genetic algorithm).

•

But allows dynamic alteration of almost all parts of the approach.

•

Road representation, genetic algorithm and ADS simulator can be customised.

•

Practitioners will benefit from using Frenetic-lib for scenario search for their ADSs.

References

[1]

Zhao, D.; Peng, H. (2017): From the lab to the street: solving the challenge of accelerating automated vehicle testing. CoRR arXiv:1707.04792.

[2]

A. Gambi, M. Mueller, G. Fraser, Automatically testing self-driving cars with search-based procedural content generation, in: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Association for Computing Machinery, New York, NY, USA, 2019, pp. 318–328,.

Digital Library

[3]

A. Gambi, M. Müller, G. Fraser, AsFault: testing self-driving car software using search-based procedural content generation, in: Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings, ICSE '19, IEEE Press, 2019, pp. 27–30,.

Digital Library

[4]

D. Humeniuk, G. Antoniol, F. Khomh, AmbieGen tool at the SBST 2022 tool competition, in: 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), 2022, pp. 43–46,.

Digital Library

[5]

D. Humeniuk, G. Antoniol, F. Khomh, SWAT tool at the SBST 2021 tool competition, in: 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST), 2021, pp. 42–43,.

[6]

J. Peltomäki, F. Spencer, I. Porres, WOGAN at the SBST 2022 CPS tool competition, in: 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), 2022, pp. 53–54,.

Digital Library

[7]

R. Ferdous, C. Hung, F. Kifetew, D. Prandi, A. Susi, EvoMBT at the SBST 2022 tool competition, in: 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), 2022, pp. 51–52,.

Digital Library

[8]

L.L.L. Starace, A. Romdhana, S. Di Martino, GenRL at the SBST 2022 tool competition, in: 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), 2022, pp. 49–50,.

Digital Library

[9]

M.H. Moghadam, M. Borg, S.J. Mousavirad, Deeper at the SBST 2021 tool competition: ADAS testing using multi-objective search, in: 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST), 2021, pp. 40–41,.

[10]

S. Panichella, A. Gambi, F. Zampetti, V. Riccio, SBST tool competition 2021, in: 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST), 2021, pp. 20–27,.

[11]

A. Gambi, G. Jahangirova, V. Riccio, F. Zampetti, SBST tool competition 2022, in: 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), 2022, pp. 25–32,.

Digital Library

[12]

E. Castellano, A. Cetinkaya, P. Arcaini, Analysis of road representations in search-based testing of autonomous driving systems, in: 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS), 2021, pp. 167–178,.

[13]

E. Castellano, A. Cetinkaya, C. Ho Thanh, S. Klikovits, X. Zhang, P. Arcaini, Frenetic at the SBST 2021 tool competition, in: 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST), 2021, pp. 36–37,.

[14]

E. Castellano, S. Klikovits, A. Cetinkaya, P. Arcaini, FreneticV at the SBST 2022 tool competition, in: 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), 2022, pp. 47–48,.

Digital Library

[15]

M. Mitchell, An Introduction to Genetic Algorithms, MIT Press, 1998.

[16]

S. Kobayashi, Differential Geometry of Curves and Surfaces, Springer, 2019.

[17]

S. Yan, M. Fan, AdaFrenetic at the SBST 2022 tool competition, in: 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), 2022, pp. 41–42,.

Digital Library

[18]

L. Klampfl, F. Klück, F. Wotawa, Using genetic algorithms for automating automated lane-keeping system testing, J. Softw. Evol. Process (2022),.

Digital Library

[19]

D. Humeniuk, F. Khomh, G. Antoniol, A search-based framework for automatic generation of testing environments for cyber–physical systems, Inf. Softw. Technol. 149 (2022),.

Digital Library

[20]

C. Birchler, N. Ganz, S. Khatiri, A. Gambi, S. Panichella, Cost-effective simulation-based test selection in self-driving cars software with SDC-scissor, in: 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2022, pp. 164–168,.

[21]

Moghadam, M.H.; Borg, M.; Saadatmand, M.; Mousavirad, S.J.; Bohlin, M.; Lisper, B. (2022): Machine learning testing in an ADAS case study using simulation-integrated bio-inspired search-based testing. CoRR arXiv:2203.12026.

[22]

J. Gallier, Curves and Surfaces in Geometric Modeling: Theory and Algorithms, Morgan Kaufmann, 2000.

[23]

E. Catmull, R. Rom, A class of local interpolating splines, in: Computer Aided Geometric Design, Academic Press, 1974, pp. 317–326,.

[24]

P. Arcaini, A. Cetinkaya, CRAG at the SBFT 2023 tool competition – cyber-physical systems track, in: 16th IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT 2023, Melbourne, Australia, May 14, 2023, 2023, pp. 41–42,.

Cited By

Humeniuk DBen Braiek HReid TKhomh FFilkov VRay BZhou M(2024)In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic ManipulatorsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695281(2187-2198)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695281
Arcaini PCetinkaya A(2024)CRAG at the SBFT 2024 Tool Competition - Cyber-Physical Systems TrackProceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing10.1145/3643659.3648559(71-72)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643659.3648559
Blattner TBirchler CKehrer TPanichella S(2024)Diversity-guided Search Exploration for Self-driving Cars Test Generation through Frenet Space EncodingProceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing10.1145/3643659.3643926(9-12)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643659.3643926
Show More Cited By

Recommendations

AsFault: testing self-driving car software using search-based procedural content generation
ICSE '19: Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings

Ensuring the safety of self-driving cars is important, but neither industry nor authorities have settled on a standard way to test them. Deploying self-driving cars for testing in regular traffic is a common, but costly and risky method, which has ...
An empirical comparison of combinatorial testing and search-based testing in the context of automated and autonomous driving systems
Abstract Context:
More automated and autonomous systems are becoming daily use that implements safety–critical functions, e.g., autonomous driving or mobile robots. Testing such systems people depend on is challenging because some ...
Highlights
- For testing safety–critical systems, the identification of critical scenarios is essential.
Automatically testing self-driving cars with search-based procedural content generation
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

Self-driving cars rely on software which needs to be thoroughly tested. Testing self-driving car software in real traffic is not only expensive but also dangerous, and has already caused fatalities. Virtual tests, in which self-driving car software is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Science of Computer Programming

Science of Computer Programming Volume 230, Issue C

Aug 2023

340 pages

ISSN:0167-6423

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier North-Holland, Inc.

United States

Publication History

Published: 01 August 2023

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Humeniuk DBen Braiek HReid TKhomh FFilkov VRay BZhou M(2024)In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic ManipulatorsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695281(2187-2198)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695281
Arcaini PCetinkaya A(2024)CRAG at the SBFT 2024 Tool Competition - Cyber-Physical Systems TrackProceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing10.1145/3643659.3648559(71-72)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643659.3648559
Blattner TBirchler CKehrer TPanichella S(2024)Diversity-guided Search Exploration for Self-driving Cars Test Generation through Frenet Space EncodingProceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing10.1145/3643659.3643926(9-12)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643659.3643926
Arcaini PCetinkaya A(2024)CRAG – a combinatorial testing-based generator of road geometries for ADS testingScience of Computer Programming10.1016/j.scico.2024.103171238:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.scico.2024.103171

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents