Just Accepted
A Roadmap for Simulation-Based Testing of Autonomous Cyber-Physical Systems: Challenges and Future Direction
Authors: 
Christian Birchler, 
Sajad Khatiri, Pooja Rani, Timo Kehrer, 
Sebastiano PanichellaAuthors Info & Claims
Accepted on 20 November 2024
Abstract
As the era of autonomous cyber-physical systems (ACPSs), such as unmanned aerial vehicles and self-driving cars, unfolds, the demand for robust testing methodologies is key to realizing the adoption of such systems in real-world scenarios. However, traditional software testing paradigms face unprecedented challenges in ensuring the safety and reliability of these systems. In response, this paper pioneers a strategic roadmap for simulation-based system-level testing of ACPSs, specifically focusing on autonomous systems. Our paper discusses the relevant challenges and obstacles of ACPSs, focusing on test automation and quality assurance, hence advocating for tailored solutions to address the unique demands of autonomous systems. While providing concrete definitions of test cases within simulation environments, we also accentuate the need to create new benchmark assets and the development of automated tools tailored explicitly for autonomous systems in the software engineering community. This paper not only highlights the relevant, pressing issues the software engineering community should focus on (in terms of practices, expected automation, and paradigms), but it also outlines ways to tackle them. By outlining the various domains and challenges of simulation-based testing/development for ACPSs, we provide directions for future research efforts.
References
[1]
Afsoon Afzal, Deborah S Katz, Claire Le Goues, and Christopher S Timperley. 2021. Simulation for robotics test automation: Developer perspectives. In Conference on Software Testing, Verification and Validation. IEEE, 263–274.
[2]
Afsoon Afzal, Claire Le Goues, Michael Hilton, and Christopher Steven Timperley. 2020. A study on challenges of testing robotic systems. In Intl. Conference on Software Testing, Validation and Verification. IEEE, 96–107.
[3]
Anders SG Andrae and Tomas Edler. 2015. On global electricity usage of communication technology: trends to 2030. Challenges 6, 1 (2015), 117–157.
[4]
Aitor Arrieta, Maialen Otaegi, Liping Han, Goiuria Sagardui, Shaukat Ali, and Maite Arratibel. 2022. Automating Test Oracle Generation in DevOps for Industrial Elevators. In Intl. Conference on Software Analysis, Evolution and Reengineering. IEEE, 284–288.
[5]
Aitor Arrieta, Shuai Wang, Goiuria Sagardui, and Leire Etxeberria. 2016. Search-based test case selection of cyber-physical system product lines for simulation-based validation. In Intl. Systems and Software Product Line Conference, Hong Mei (Ed.). ACM, 297–306.
[6]
Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Trans. Software Eng. 41, 5 (2015), 507–525.
[7]
Matteo Biagiola, Stefan Klikovits, Jarkko Peltomäki, and Vincenzo Riccio. 2023. SBFT Tool Competition 2023 - Cyber-Physical Systems Track. In Intl. Workshop on Search-Based and Fuzz Testing. IEEE, 45–48.
[8]
Christian Birchler, Sajad Khatiri, Bill Bosshard, Alessio Gambi, and Sebastiano Panichella. 2023. Machine learning-based test selection for simulation-based testing of self-driving cars software. Empir. Softw. Eng. 28, 3 (2023), 71.
[9]
Christian Birchler, Sajad Khatiri, Pouria Derakhshanfar, Sebastiano Panichella, and Annibale Panichella. 2023. Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments. ACM Trans. Softw. Eng. Methodol. 32, 2 (2023), 28:1–28:30.
[10]
Christian Birchler, Tanzil Kombarabettu Mohammed, Pooja Rani, Teodora Nechita, Timo Kehrer, and Sebastiano Panichella. 2024. How does Simulation-based Testing for Self-driving Cars match Human Perception?. In Intl. Conference on the Foundations of Software Engineering.
[11]
Christian Birchler, Cyrill Rohrbach, Timo Kehrer, and Sebastiano Panichella. 2024. SensoDat: Simulation-based Sensor Dataset of Self-driving Cars. In Intl. Conference on Mining Software Repositories.
[12]
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, et al. 2018. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In Intl. Conference on Robotics and Automation. IEEE, 4243–4250.
[13]
Jack Collins, Ross Brown, Jurgen Leitner, and David Howard. 2020. Traversing the reality gap via simulator tuning. arXiv preprint arXiv:2003.01369 (2020).
[14]
Konstantinos Dimitropoulos, Ioannis Hatzilygeroudis, and Konstantinos Chatzilygeroudis. 2022. A Brief Survey of Sim2Real Methods for Robot Learning. In Intl. Conference on Robotics in Alpe-Adria Danube Region. Springer, 133–140.
[15]
Federico Formica, Tony Fan, Akshay Rajhans, Vera Pantelic, Mark Lawford, and Claudio Menghi. 2024. Simulation-Based Testing of Simulink Models With Test Sequence and Test Assessment Blocks. IEEE Trans. Software Eng. 50, 2 (2024), 239–257.
[16]
Sylvain Frey, Awais Rashid, Pauline Anthonysamy, Maria Pinto-Albuquerque, and Syed Asad Naqvi. 2019. The Good, the Bad and the Ugly: A Study of Security Decisions in a Cyber-Physical Systems Game. IEEE Trans. Software Eng. 45, 5 (2019), 521–536.
[17]
Alessio Gambi, Gunel Jahangirova, Vincenzo Riccio, and Fiorella Zampetti. 2022. SBST Tool Competition 2022. In Intl. Workshop on Search-Based Software Testing. IEEE, 25–32.
[18]
Stefanos Georgiou, Stamatia Rizou, and Diomidis Spinellis. 2019. Software development lifecycle for energy efficiency: techniques and tools. ACM Comput. Surv. 52, 4 (2019), 1–33.
[19]
Cláudio Gomes, Casper Thule, David Broman, Peter Gorm Larsen, and Hans Vangheluwe. 2018. Co-Simulation: A Survey. ACM Comput. Surv. 51, 3, Article 49 (may 2018), 33 pages.
[20]
Giovanni Grano, Christoph Laaber, Annibale Panichella, and Sebastiano Panichella. 2021. Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation. IEEE Trans. Software Eng. 47, 11 (2021), 2332–2347.
[21]
Carl Hildebrandt and Sebastian Elbaum. 2021. World-in-the-loop simulation for autonomous systems validation. In Intl. Conference on Robotics and Automation. IEEE, 10912–10919.
[22]
Sebastian Höfer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Melissa Mozifian, Florian Golemo, Chris Atkeson, Dieter Fox, Ken Goldberg, John Leonard, et al. 2021. Sim2Real in robotics and automation: Applications and challenges. IEEE Transactions on Automation Science and Engineering 18, 2 (2021), 398–400.
[23]
William E. Howden. 1982. Weak Mutation Testing and Completeness of Test Sets. IEEE Trans. Software Eng. 8, 4 (1982), 371–379.
[24]
Gunel Jahangirova, David Clark, Mark Harman, and Paolo Tonella. 2016. Test oracle assessment and improvement. In Intl. Symposium on Software Testing and Analysis. ACM, 247–258.
[25]
Gunel Jahangirova, Andrea Stocco, and Paolo Tonella. 2021. Quality Metrics and Oracles for Autonomous Vehicles Testing. In Conference on Software Testing, Verification and Validation. IEEE, 194–204.
[26]
Stephen James, Andrew J Davison, and Edward Johns. 2017. Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task. In Conference on Robot Learning. PMLR, 334–343.
[27]
Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2023. Simulation-based Test Case Generation for Unmanned Aerial Vehicles in the Neighborhood of Real Flights. In IEEE Conference on Software Testing, Verification and Validation. IEEE, 281–292.
[28]
Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2023. Simulation-based Test Case Generation for Unmanned Aerial Vehicles in the Neighborhood of Real Flights. In Intl. Conference on Software Testing, Verification and Validation. IEEE, 281–292.
[29]
Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2024. Simulation-based Testing of Unmanned Aerial Vehicles with Aerialist. In Intl. Conference on Software Engineering.
[30]
Sajad Khatiri, Prasun Saurabh, Timothy Zimmermann, Charith Munasinghe, Christian Birchler, and Sebastiano Panichella. 2024. SBFT Tool Competition 2024: CPS-UAV Test Case Generation Track. In Intl. Workshop on Search-Based and Fuzz Testing.
[31]
Timothy E Lee, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Oliver Kroemer, Dieter Fox, and Stan Birchfield. 2020. Camera-to-robot pose estimation from a single image. In Intl. Conference on Robotics and Automation. IEEE, 9426–9432.
[32]
Chengjie Lu, Tao Yue, and Shaukat Ali. 2023. DeepScenario: An Open Driving Scenario Dataset for Autonomous Driving System Testing. In Intl. Conference on Mining Software Repositories. IEEE, 52–56.
[33]
Toni Mancini, Igor Melatti, and Enrico Tronci. 2023. Optimizing Highly-Parallel Simulation-Based Verification of Cyber-Physical Systems. IEEE Trans. Software Eng. 49, 9 (2023), 4443–4455.
[34]
Irene Manotas, Christian Bird, Rui Zhang, David Shepherd, Ciera Jaspan, Caitlin Sadowski, Lori Pollock, and James Clause. 2016. An empirical study of practitioners’ perspectives on green software engineering. In Intl. Conference on Software Engineering. 237–248.
[35]
Anthony Ngo, Max Paul Bauer, and Michael Resch. 2021. A Multi-Layered Approach for Measuring the Simulation-to-Reality Gap of Radar Perception for Autonomous Driving. In Intl. Intelligent Transportation Systems Conference. IEEE, 4008–4014.
[36]
Sebastiano Panichella, Alessio Gambi, Fiorella Zampetti, and Vincenzo Riccio. 2021. SBST Tool Competition 2021. In Intl. Workshop on Search-Based Software Testing. IEEE, 20–27.
[37]
Samuel Parra, Argentina Ortega, Sven Schneider, and Nico Hochgeschwender. 2023. A Thousand Worlds: Scenery Specification and Generation for Simulation-Based Testing of Mobile Robot Navigation Stacks. In IROS. 5537–5544.
[38]
Pooja Rani, Jonas Zellweger, Veronika Kousadianos, Luis Cruz, Timo Kehrer, and Alberto Bacchelli. 2024. Energy Patterns for Web: An Exploratory Study. arXiv preprint arXiv:2401.06482 (2024).
[39]
Fabio Reway, Abdul Hoffmann, Diogo Wachtel, Werner Huber, Alois C. Knoll, and Eduardo Parente Ribeiro. 2020. Test Method for Measuring the Simulation-to-Reality Gap of Camera-based Object Detection Algorithms for Autonomous Driving. In IEEE Intelligent Vehicles Symposium. IEEE, 1249–1256.
[40]
Erica Salvato, Gianfranco Fenu, Eric Medvet, and Felice Andrea Pellegrino. 2021. Crossing the Reality Gap: a Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning. IEEE Access (2021).
[41]
Andrea Di Sorbo, Fiorella Zampetti, Aaron Visaggio, Massimiliano Di Penta, and Sebastiano Panichella. 2023. Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software Platforms. ACM Trans. Softw. Eng. Methodol. 32, 3 (2023), 67:1–67:37.
[42]
Dinghua Wang, Shuqing Li, Guanping Xiao, Yepang Liu, and Yulei Sui. 2021. An exploratory study of autopilot software bugs in unmanned aerial vehicles. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 20–31.
[43]
Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verification Reliab. 22, 2 (2012), 67–120.
[44]
Andy Zaidman. 2024. An Inconvenient Truth in Software Engineering? The Environmental Impact of Testing Open Source Java Projects. (2024).
[45]
Fiorella Zampetti, Ritu Kapur, Massimiliano Di Penta, and Sebastiano Panichella. 2022. An empirical characterization of software bugs in open-source Cyber–Physical Systems. J. Syst. Softw. 192 (2022), 111425.
[46]
Fangyi Zhang, Jürgen Leitner, Zongyuan Ge, Michael Milford, and Peter Corke. 2019. Adversarial discriminative sim-to-real transfer of visuo-motor policies. The International Journal of Robotics Research 38, 10-11 (2019), 1229–1245.
[47]
Man Zhang, Shaukat Ali, and Tao Yue. 2019. Uncertainty-wise test case generation and minimization for Cyber-Physical Systems. J. Syst. Softw. 153 (2019), 1–21.
Index Terms
- A Roadmap for Simulation-Based Testing of Autonomous Cyber-Physical Systems: Challenges and Future Direction
Recommendations
Digital-twin-based testing for cyber–physical systems: A systematic literature review
Abstract Context:Cyber–physical systems present a challenge to testers, bringing complexity and scale to safety-critical and collaborative environments. Digital twins enhance these systems through data-driven and simulation based ...
TinyML: Current Progress, Research Challenges, and Future Roadmap
2021 58th ACM/IEEE Design Automation Conference (DAC)TinyML: tiny in size, BIG in impact!This paper highlights the current progress, challenges and open research opportunities in the domain of tinyML, benchmarking, and emerging applications for Edge-AI.
Design Automation of Cyber-Physical Systems: Challenges, Advances, and Opportunities
A cyber-physical system (CPS) is an integration of computation with physical processes whose behavior is defined by both computational and physical parts of the system. In this paper, we present a view of the challenges and opportunities for design ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
EISSN:1557-7392
Table of ContentsCopyright © 2025 Copyright held by the owner/author(s).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Online AM: 16 January 2025
Accepted: 20 November 2024
Revised: 01 October 2024
Received: 01 October 2024
Check for updates
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 100Total Downloads
- Downloads (Last 12 months)100
- Downloads (Last 6 weeks)91
Reflects downloads up to 28 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in