Exploration-Driven Reinforcement Learning for Avionic System Fault Detection (Experience Paper)
Authors: 
Paul-Antoine Le Tolguenec, Emmanuel Rachelson, Yann Besse, Florent Teichteil-Koenigsbuch, Nicolas Schneider, Hélène Waeselynck, Dennis WilsonAuthors Info & Claims
Pages 920 - 931
Abstract
Critical software systems require stringent testing to identify possible failure cases, which can be difficult to find using manual testing. In this study, we report our industrial experience in testing a realistic R&D flight control system using a heuristic based testing method. Our approach utilizes evolutionary strategies augmented with intrinsic motivation to yield a diverse range of test cases, each revealing different potential failure scenarios within the system. This diversity allows for a more comprehensive identification and understanding of the system’s vulnerabilities. We analyze the test cases found by evolution to identify the system’s weaknesses. The results of our study show that our approach can be used to improve the reliability and robustness of avionics systems by providing high-quality test cases in an efficient and cost-effective manner.
References
[1]
2022. ODI RESUME. https://static.nhtsa.gov/odi/inv/2022/INOA-PE22002-4385.PDF Accessed: 23/03/2024
[2]
Yasasa Abeysirigoonawardena, Florian Shkurti, and Gregory Dudek. 2019. Generating adversarial driving scenarios in high-fidelity simulators. In 2019 International Conference on Robotics and Automation (ICRA). 8271–8277.
[3]
Andrea Arcuri. 2010. It does matter how you normalise the branch distance in search based software testing. In 2010 Third International Conference on Software Testing, Verification and Validation. 205–214.
[4]
Aitor Arrieta, Shuai Wang, Goiuria Sagardui, and Leire Etxeberria. 2016. Test case prioritization of configurable cyber-physical systems with weight-based search algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference 2016. 1053–1060.
[5]
Arthur Aubret, Laetitia Matignon, and Salima Hassas. 2019. A survey on intrinsic motivation in reinforcement learning. arXiv e-prints, arXiv–1908.
[6]
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, and Andew Bolt. 2020. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038.
[7]
Richard Bellman. 1957. A Markovian Decision Process. Indiana University Mathematics Journal, 6, 4 (1957), 679–684. issn:0022-2518
[8]
Jon Berndt. 2004. JSBSim: An open source flight dynamics model in C++. In AIAA Modeling and Simulation Technologies Conference and Exhibit. 4923.
[9]
Thomas Bochot, Pierre Virelizier, Hélene Waeselynck, and Virginie Wiels. 2009. Model checking flight control systems: The Airbus experience. In 2009 31st International Conference on Software Engineering-Companion Volume. 18–27.
[10]
Mohamed Boussaa, Olivier Barais, Gerson Sunyé, and Benoit Baudry. 2015. A novelty search approach for automatic test data generation. In 2015 IEEE/ACM 8th International Workshop on Search-Based Software Testing. 40–43.
[11]
Oliver Bühler and Joachim Wegener. 2008. Evolutionary functional testing. Computers & Operations Research, 35, 10 (2008), 3144–3160.
[12]
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. 2018. Exploration by random network distillation. In International Conference on Learning Representations.
[13]
Ying Chen, Shaowei Huang, Feng Liu, Zhisheng Wang, and Xinwei Sun. 2018. Evaluation of reinforcement learning-based false data injection attack to automatic voltage control. IEEE Transactions on Smart Grid, 10, 2 (2018), 2158–2169.
[14]
Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. 2018. Back to basics: benchmarking canonical evolution strategies for playing Atari. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 1419–1426.
[15]
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. 2018. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. Advances in neural information processing systems, 31 (2018).
[16]
Antoine Cully. [n. d.]. Autonomous Skill Discovery with Quality-Diversity and Unsupervised Descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ’19). Association for Computing Machinery, 81–89. isbn:978-1-4503-6111-8 https://doi.org/10.1145/3321707.3321804
[17]
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. [n. d.]. Robots That Can Adapt like Animals. 521, 7553 ([n. d.]), 503–507. issn:1476-4687 https://doi.org/10.1038/nature14422
[18]
Minghui Dai, Zhou Su, Qichao Xu, and Weiwei Chen. 2019. A Q-learning based scheme to securely cache content in edge-enabled heterogeneous networks. IEEE Access, 7 (2019), 163898–163911.
[19]
Didier Essame, Jean Arlat, and David Powell. 1999. Padre: A protocol for asymmetric duplex redundancy. In Dependable Computing for Critical Applications 7. 229–248.
[20]
Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. 2023. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615, 7953 (2023), 620–627.
[21]
Federico Formica, Nicholas Petrunti, Lucas Bruck, Vera Pantelic, Mark Lawford, and Claudio Menghi. 2023. Test case generation for drivability requirements of an automotive cruise controller: An experience with an industrial simulator. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1949–1960.
[22]
Alessio Gambi, Marc Mueller, and Gordon Fraser. 2019. Automatically testing self-driving cars with search-based procedural content generation. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 318–328.
[23]
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. 1861–1870.
[24]
Nikolaus Hansen and Andreas Ostermeier. 2001. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9, 2 (2001), 159–195.
[25]
Junda He, Zhou Yang, Jieke Shi, Chengran Yang, Kisub Kim, Bowen Xu, Xin Zhou, and David Lo. 2024. Curiosity-Driven Testing for Sequential Decision-Making Process. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). 949–949.
[26]
Gunel Jahangirova, Andrea Stocco, and Paolo Tonella. 2021. Quality metrics and oracles for autonomous vehicles testing. In 2021 14th IEEE conference on software testing, verification and validation (ICST). 194–204.
[27]
Mark Koren, Saud Alsaif, Ritchie Lee, and Mykel J Kochenderfer. 2018. Adaptive stress testing for autonomous vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV). 1–7.
[28]
Mark Koren and Mykel J Kochenderfer. 2019. Efficient autonomy validation in simulation with adaptive stress testing. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC). 4178–4183.
[29]
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. 2016. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29 (2016).
[30]
Alexandr Kuznetsov, Yehor Yeromin, Oleksiy Shapoval, Kyrylo Chernov, Mariia Popova, and Kostyantyn Serdukov. 2019. Automated software vulnerability testing using deep learning methods. In 2019 IEEE 2nd Ukraine Conference on Electrical and Computer Engineering (UKRCON). 837–841.
[31]
Paul-Antoine Le Tolguenec, Emmanuel Rachelson, Yann Besse, and Dennis G. Wilson. 2023. Curiosity Creates Diversity in Policy Search. ACM Trans. Evol. Learn. Optim., jun, https://doi.org/10.1145/3605782
[32]
Ritchie Lee, Mykel J Kochenderfer, Ole J Mengshoel, Guillaume P Brat, and Michael P Owen. 2015. Adaptive stress testing of airborne collision avoidance systems. In 2015 IEEE/AIAA 34th Digital Avionics Systems Conference (DASC). 6C2–1.
[33]
Joel Lehman and Kenneth O Stanley. [n. d.]. Exploiting Open-Endedness to Solve Problems through the Search for Novelty. In ALIFE. 329–336.
[34]
Joel Lehman and Kenneth O Stanley. 2011. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19, 2 (2011), 189–223.
[35]
Joel Lehman and Kenneth O Stanley. 2011. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation. 211–218.
[36]
Chao Li, Wen Zhou, Kai Yu, Liseng Fan, and Junjuan Xia. 2019. Enhanced secure transmission against intelligent attacks. IEEE Access, 7 (2019), 53596–53602.
[37]
Bing Liu, Shiva Nejati, Lucia, and Lionel C Briand. 2019. Effective fault localization of automotive Simulink models: achieving the trade-off between test oracle effort and fault localization accuracy. Empirical Software Engineering, 24 (2019), 444–490.
[38]
Xing Liu, Hansong Xu, Weixian Liao, and Wei Yu. 2019. Reinforcement learning for cyber-physical systems. In 2019 IEEE International Conference on Industrial Internet (ICII). 318–327.
[39]
Zengguang Liu, Xiaochun Yin, and Yuemei Hu. 2020. CPSS LR-DDoS detection and defense in edge computing utilizing DCNN Q-learning. IEEE Access, 8 (2020), 42120–42130.
[40]
Reza Matinnejad, Shiva Nejati, and Lionel C Briand. 2017. Automated testing of hybrid simulink/stateflow controllers: industrial case studies. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 938–943.
[41]
Reza Matinnejad, Shiva Nejati, Lionel C Briand, and Thomas Bruckmann. 2016. Automated test suite generation for time-continuous simulink models. In proceedings of the 38th International Conference on Software Engineering. 595–606.
[42]
Phil McMinn. 2011. Search-based software testing: Past, present and future. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops. 153–163.
[43]
Paulina Stevia Nouwou Mindom, Amin Nikanjam, and Foutse Khomh. 2022. A comparison of reinforcement learning frameworks for software testing tasks. arXiv preprint arXiv:2208.12136.
[44]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.
[45]
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, and Michael I Jordan. 2018. Ray: A distributed framework for emerging $AI$ applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 561–577.
[46]
Justin Norden, Matthew O’Kelly, and Aman Sinha. 2019. Efficient Black-box Assessment of Autonomous Vehicle Safety. arXiv e-prints, arXiv–1912.
[47]
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning. 2778–2787.
[48]
Peter Puschner and Roman Nossal. 1998. Testing the results of static worst-case execution-time analysis. In Proceedings 19th IEEE Real-Time Systems Symposium (Cat. No. 98CB36279). 134–143.
[49]
Martin L Puterman. 2014. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.
[50]
Ingo Rechenberg. 1978. Evolutionsstrategien. In Simulationsmethoden in der Medizin und Biologie. Springer, 83–114.
[51]
Raymond Ros and Nikolaus Hansen. 2008. A Simple Modification in CMA-ES Achieving Linear Time and Space Complexity. In Proceedings of the 10th International Conference on Parallel Problem Solving from Nature—PPSN X-Volume 5199. 296–305.
[52]
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv e-prints, arXiv–1703.
[53]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
[54]
Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2022. Mind the gap! a study on the transferability of virtual vs physical-world testing of autonomous driving systems. IEEE Transactions on Software Engineering.
[55]
Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2023. Model vs system level testing of autonomous driving systems: a replication and extension study. Empirical Software Engineering, 28, 3 (2023), 73.
[56]
Jörg Stork, Martin Zaefferer, Nils Eisler, Patrick Tichelmann, Thomas Bartz-Beielstein, and AE Eiben. 2021. Behavior-based neuroevolutionary training in reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 1753–1761.
[57]
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
[58]
Nigel Tracey, John Clark, Keith Mander, and John McDermid. 1998. An automated framework for structural test-data generation. In Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No. 98EX239). 285–288.
[59]
Pascal Traverse, Isabelle Lacaze, and Jean Souyris. 2004. Airbus fly-by-wire: A total approach to dependability. In Building the Information Society: IFIP 18th World Computer Congress Topical Sessions 22–27 August 2004 Toulouse, France. 191–212.
[60]
Joachim Wegener, André Baresel, and Harmen Sthamer. 2001. Evolutionary test environment for automatic structural testing. Information and software technology, 43, 14 (2001), 841–854.
[61]
Joachim Wegener and Matthias Grochtmann. 1998. Verifying timing constraints of real-time systems by means of evolutionary testing. Real-Time Systems, 15 (1998), 275–298.
[62]
Joachim Wegener and Frank Mueller. 2001. A comparison of static analysis and evolutionary testing for the verification of timing constraints. Real-time systems, 21 (2001), 241–268.
[63]
Joachim Wegener, Harmen Sthamer, Bryan F Jones, and David E Eyres. 1997. Testing real-time systems using genetic algorithms. Software Quality Journal, 6 (1997), 127–135.
[64]
Junhui Zhang and Jitao Sun. 2019. A game theoretic approach to multi-channel transmission scheduling for multiple linear systems under DoS attacks. Systems & Control Letters, 133 (2019), 104546.
[65]
Tao Zhang, Xiaohui Kuang, Zan Zhou, Hongquan Gao, and Changqiao Xu. 2019. An intelligent route mutation mechanism against mixed attack based on security awareness. In 2019 IEEE Global Communications Conference (GLOBECOM). 1–6.
[66]
Yan Zheng, Yi Liu, Xiaofei Xie, Yepang Liu, Lei Ma, Jianye Hao, and Yang Liu. 2021. Automatic web testing using curiosity-driven reinforcement learning. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 423–435.
Index Terms
- Exploration-Driven Reinforcement Learning for Avionic System Fault Detection (Experience Paper)
Recommendations
Directed test generation for effective fault localization
ISSTA '10: Proceedings of the 19th international symposium on Software testing and analysisFault-localization techniques that apply statistical analyses to execution data gathered from multiple tests are quite effective when a large test suite is available. However, if no test suite is available, what is the best approach to generate one? ...
Testing for Fault Diversity in Reinforcement Learning
AST '24: Proceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)Reinforcement Learning is the premier technique to approach sequential decision problems, including complex tasks such as driving cars and landing spacecraft. Among the software validation and verification practices, testing for functional fault ...
An Efficient Adaptive Method of Software-Based Self Test Generation for RISC Processors
ECBS-EERC '15: Proceedings of the 2015 4th Eastern European Regional Conference on the Engineering of Computer Based SystemsThe contribution presents automatic software-based self (SBST) test generation for processors as a basic blocks of current complex systems on chip and embedded systems. Processors testing can be extended by functional tests or using various application ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
September 2024
1928 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 September 2024
Check for updates
Badges
Distinguished Paper
Author Tags
Qualifiers
- Research-article
Conference
ISSTA '24
Sponsor:
ISSTA '24: 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis
September 16 - 20, 2024
Vienna, Austria
Acceptance Rates
Overall Acceptance Rate 58 of 213 submissions, 27%
Upcoming Conference
ISSTA '25
- Sponsor:
- sigsoft
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 165Total Downloads
- Downloads (Last 12 months)165
- Downloads (Last 6 weeks)88
Reflects downloads up to 26 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in