research-article

Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components

Authors:

Cumhur Erkan Tuncali,

Georgios Fainekos,

James KapinskiAuthors Info & Claims

2018 IEEE Intelligent Vehicles Symposium (IV)

Pages 1555 - 1562

https://doi.org/10.1109/IVS.2018.8500421

Published: 26 June 2018 Publication History

Abstract

Many organizations are developing autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning (ML) components, such as deep neural networks, for which formal properties are difficult to characterize. We present a testing framework that is compatible with test case generation and automatic falsification methods, which are used to evaluate cyber-physical systems. We demonstrate how the framework can be used to evaluate closed-loop properties of an autonomous driving system model that includes the ML components, all within a virtual environment. We demonstrate how to use test case generation methods, such as covering arrays, as well as requirement falsification methods to automatically identify problematic test scenarios. The resulting framework can be used to increase the reliability of autonomous driving systems.

References

[1]

A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012.

[2]

D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in Advances in neural information processing systems, 1989, pp. 305–313.

[3]

M. Strickland, G. Fainekos, and H. B. Amor, “Deep predictive models for collision risk assessment in autonomous driving,” arXiv preprint arXiv:1711.10453, 2017.

[4]

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.

Digital Library

[5]

S. Dutta, S. Jha, S. Sanakaranarayanan, and A. Tiwari, “Output range analysis for deep neural networks,” arXiv:1709.09130, 2017.

[6]

X. Huang, M. Kwiatkowska, S. Wang, and M. Wu, “Safety verification of deep neural networks,” in 29th International Conference on Computer Aided Verification (CAV), ser. LNCS, vol. 10426. Springer, 2017, pp. 3–29.

[7]

G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer, “Reluplex: An efficient SMT solver for verifying deep neural networks,” in International Conference on Computer Aided Verification (CAV). Springer, 2017.

[8]

L. Pulina and A. Tacchella, An Abstraction-Refinement Approach to Verification of Artificial Neural Networks, ser. LNCS. Berlin, Heidelberg: Springer Berlin Heidelberg 2010, vol. 6174, pp. 243–257.

[9]

K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” in Symposium on Operating Systems Principles (SOSP), 2017.

[10]

Y. Tian, K. Pei, S. Jana, and B. Ray, “Deeptest: Automated testing of deep-neural-network-driven autonomous cars,” arXiv:1708.08559v1, 2017.

[11]

N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2016.

[12]

N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in ACM Asia Conference on Computer and Communications Security. ACM, 2017.

[13]

I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in 3rd International Conference on Learning Representations (ICLR), 2015.

[14]

P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, “ZOO: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” arXiv:1708.03999v2, 2017.

[15]

I. Goodfellow, “NIPS 2016 tutorial: Generative adversarial networks,” arXiv:1701.00160v3, 2017.

[16]

T. Dreossi, S. Ghosh, A. Sangiovanni-Vincentelli, and S. A. Seshia, “Systematic testing of convolutional neural networks for autonomous driving,” in Reliable Machine Learning in the Wild (RMLW), 2017.

[17]

T. Dreossi, A. Donze, and S. A. Seshia, “Compositional falsication of cyber-physical systems with machine learning components,” in NASA Formal Methods (NFM), ser. LNCS, vol. 10227. Springer, 2017, pp. 357–372.

[18]

M. O'Kelly, H. Abbas, and R. Mangharam, “Computer-aided design for safe autonomous vehicles,” in 2017 Resilience Week (RWS), 2017.

[19]

C. E. Tuncali, T. P. Pavlic, and G. Fainekos, “Utilizing S-TaLiRo as an automatic test generation framework for autonomous vehicles,” in IEEE Intelligent Transportation Systems Conference, 2016.

[20]

B. Kim, Y. Kashiba, S. Dai, and S. Shiraishi, “Testing autonomous vehicle software in the virtual prototyping environment,” Embedded Systems Letters, vol. 9, no. 1, pp. 5–8, 2017.

Digital Library

[21]

B. Kim, A. Jarandikar, J. Shum, S. Shiraishi, and M. Yamaura, “The SMT-based automatic road network generation in vehicle simulation environment,” in International Conference on Embedded Software (EMSOFT). ACM, 2016, pp. 18:1–18:10.

[22]

E. Bartocci, J. Deshmukh, A. Donzé, G. Fainekos, O. Maler, D. Nickovic, S. Sankaranarayanan, “Specification-based monitoring of cyber-physical systems: A survey on theory, tools and applications,” in Lectures on Runtime Verification - Introductory and Advanced Topics, ser. LNCS. Springer, 2018, vol. 10457, pp. 128–168.

[23]

G. E. Fainekos and G. J. Pappas, “Robustness of temporal logic specifications,” in Formal Approaches to Testing and Runtime Verification, ser. LNCS, vol. 4262. Springer, 2006, pp. 178–192.

Digital Library

[24]

G. Fainekos, S. Sankaranarayanan, K. Ueda, and H. Yazarel, “Verification of automotive control applications using s-taliro,” in Proceedings of the American Control Conference, 2012.

[25]

A. Donze and O. Maler, “Robust satisfaction of temporal logic over real-valued signals,” in Formal Modelling and Analysis of Timed Systems, ser. LNCS, vol. 6246. Springer, 2010.

[26]

S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a formal model of safe and scalable self-driving cars,” arXiv:1708.06374, 2017.

[27]

H. Abbas, G. E. Fainekos, S. Sankaranarayanan, F. Ivancic, and A. Gupta, “Probabilistic temporal logic falsification of cyber-physical systems,” ACM Transactions on Embedded Computing Systems, vol. 12, no. s2, May 2013.

[28]

B. Hoxha, H. Bach, H. Abbas, A. Dokhanchi, Y. Kobayashi, and G. Fainekos, “Towards formal specification visualization for testing and monitoring of cyber-physical systems,” in International Workshop on Design and Implementation of Formal Tools and Systems, 2014.

[29]

J. Kapinski, J. V. Deshmukh, X. Jin, H. Ito, and K. Butts, “Simulationbased approaches for verification of embedded control systems: An overview of traditional and advanced modeling, testing, and verification techniques,” IEEE Control Systems Magazine, vol. 36, no. 6, pp. 45–64, 2016.

[30]

A. Hartman, “Software and hardware testing using combinatorial covering suites,” Graph theory, combinatorics and algorithms, vol. 34, pp. 237–266, 2005.

[31]

D. R. Kuhn, R. N. Kacker, and Y. Lei, Introduction to combinatorial testing. CRC press, 2013.

Digital Library

[32]

B. Wu, F. Iandola, P. H. Jin, and K. Keutzer, “Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving,” arXiv preprint arXiv:1612.01051, 2016.

[33]

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al., “Tensorflow: Largescale machine learning on heterogeneous distributed systems,” arXiv preprint:1603.04467, 2016.

[34]

Z. Kalal, K. Mikolajczyk, and J. Matas, “Forward-backward error: Automatic detection of tracking failures,” in Pattern recognition (ICPR), 20th international conference on. IEEE, 2010.

[35]

O. Michel, “Cyberbotics ltd. Webots: professional mobile robot simulation,” International Journal of Advanced Robotic Systems, vol. 1, no. 1, p. 5, 2004.

Cited By

Humeniuk DKhomh FAntoniol G(2024)Reinforcement Learning Informed Evolutionary Search for Autonomous Systems TestingACM Transactions on Software Engineering and Methodology10.1145/368046833:8(1-45)Online publication date: 27-Jul-2024
https://dl.acm.org/doi/10.1145/3680468
Humeniuk DKhomh F(2024)AmbieGenVAE at the SBFT 2024 Tool Competition - Cyber-Physical Systems TrackProceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing10.1145/3643659.3648560(45-46)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643659.3648560
Arcaini PCetinkaya A(2024)CRAG – a combinatorial testing-based generator of road geometries for ADS testingScience of Computer Programming10.1016/j.scico.2024.103171238:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.scico.2024.103171
Show More Cited By

Index Terms

Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components

Index terms have been assigned to the content through auto-classification.

Recommendations

Achieving scalable mutation-based generation of whole test suites

Without complete formal specification, automatically generated software tests need to be manually checked in order to detect faults. This makes it desirable to produce the strongest possible test set while keeping the number of tests as small as ...
Automatic Unit Test Generation for Machine Learning Libraries: How Far Are We?
ICSE '21: Proceedings of the 43rd International Conference on Software Engineering

Automatic unit test generation that explores the input space and produces effective test cases for given programs have been studied for decades. Many unit test generation tools that can help generate unit test cases with high structural coverage over a ...
Fault-oriented automated test data generation

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

2018 IEEE Intelligent Vehicles Symposium (IV)

Jun 2018

2094 pages

Copyright © 2018.

Publisher

IEEE Press

Publication History

Published: 26 June 2018

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Humeniuk DKhomh FAntoniol G(2024)Reinforcement Learning Informed Evolutionary Search for Autonomous Systems TestingACM Transactions on Software Engineering and Methodology10.1145/368046833:8(1-45)Online publication date: 27-Jul-2024
https://dl.acm.org/doi/10.1145/3680468
Humeniuk DKhomh F(2024)AmbieGenVAE at the SBFT 2024 Tool Competition - Cyber-Physical Systems TrackProceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing10.1145/3643659.3648560(45-46)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643659.3648560
Arcaini PCetinkaya A(2024)CRAG – a combinatorial testing-based generator of road geometries for ADS testingScience of Computer Programming10.1016/j.scico.2024.103171238:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.scico.2024.103171
Yamaguchi THoxha BNičković D(2024)RTAMT – Runtime Robustness Monitors with Application to CPS and RoboticsInternational Journal on Software Tools for Technology Transfer (STTT)10.1007/s10009-023-00720-326:1(79-99)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s10009-023-00720-3
Li QPeng ZFeng LLiu ZDuan CMo WZhou BOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)ScenarioNetProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666294(3894-3920)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666294
Song ROzmen MKim HMuller RCelik ZBianchi ACalandrino JTroncoso C(2023)Discovering adversarial driving maneuvers against autonomous vehiclesProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620403(2957-2974)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620403
Qin XArechiga NDeshmukh JBest AZhu Qvon Hanxleden RStephen EBrandt J(2023)Robust Testing for Cyber-Physical Systems using Reinforcement LearningProceedings of the 21st ACM-IEEE International Conference on Formal Methods and Models for System Design10.1145/3610579.3611087(36-46)Online publication date: 21-Sep-2023
https://dl.acm.org/doi/10.1145/3610579.3611087
Cheng MZhou YXie XJust RFraser G(2023)BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving SystemsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598072(488-500)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598072
Hildebrandt Cvon Stein MElbaum SJust RFraser G(2023)PhysCov: Physical Test Coverage for Autonomous VehiclesProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598069(449-461)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598069
Tang SZhang ZZhang YZhou JGuo YLiu SGuo SLi YMa LXue YLiu Y(2023)A Survey on Automated Driving System Testing: Landscapes and TrendsACM Transactions on Software Engineering and Methodology10.1145/357964232:5(1-62)Online publication date: 24-Jul-2023
https://dl.acm.org/doi/10.1145/3579642
Show More Cited By

View Options

View options

Figures

Tables

Media

View Table of Conten