Abstract
Highly automated and autonomous driving systems are usually tested for their safe behavior using a so-called scenario-based testing approach. A common practice is to let experts create parameterized scenarios by selecting and varying parameters of a given scenario type, e.g., the initial speed of the participating vehicles. By assigning concrete values to the selected parameters, scenario instances are generated, which may be used as test scenarios for the driving system under test (SUT). For the generation of test cases, parameterized scenarios typically serve as input. Most works assume parameterized scenarios to be given without evaluating their quality. However, a parameterized scenario may be insufficient, leading to inadequately and incomplete generated test cases, unreliable test results, and even incorrect conclusions about the safety of the SUT. As contribution of this work, we present a quality criterion and a novel data-driven assurance approach to assess parameterized scenarios. We consider the quality of a parameterized scenario to be acceptable if it contains at least all scenario instances collected in real traffic for the studied scenario type. For this containment check, search-based techniques are used. We show experiments for a parameterized lane change scenario using 6736 lane change recordings from real traffic for the assessment. The experiment results show that in addition to shortcomings of a parameterized scenario, those of the simulation setup can be revealed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abdessalem, R.B., Nejati, S., Briand, L., Stifter, T.: Testing vision-based control systems using learnable evolutionary algorithms. In: Proceedings of the 40th International Conference on Software Engineering (ICSE), pp. 1016–1026. ACM (2018)
Abdessalem, R.B., Nejati, S., Briand, L.C., Stifter, T.: Testing advanced driver assistance systems using multi-objective search and neural networks. In: 31st IEEE/ACM International Conference on Automated Software Engineering, pp. 63–74 (2016)
Abdessalem, R.B., Panichella, A., Nejati, S., Briand, L.C., Stifter, T.: Testing autonomous cars for feature interaction failures using many-objective search. In: 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 143–154 (2018)
Bagschik, G., Menzel, T., Maurer, M.: Ontology based scene creation for the development of automated vehicles. In: IEEE Intelligent Vehicles Symposium (IV), pp. 1813–1820. IEEE (2018)
Barber, C.B., Dobkin, D.P., Huhdanpaa, H.: The quickhull algorithm for convex hulls. ACM Trans. Math. Softw. (TOMS) 22(4), 469–483 (1996)
Calò, A., Arcaini, P., Ali, S., Hauer, F., Ishikawa, F.: Generating avoidable collision scenarios for testing autonomous driving systems. In: 13th International Conference on Software Testing, Validation and Verification (ICST), pp. 375–386. IEEE (2020)
Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)
Gambi, A., Huynh, T., Fraser, G.: Generating effective test cases for self-driving cars from police reports. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 257–267 (2019)
Gambi, A., Mueller, M., Fraser, G.: Automatically testing self-driving cars with search-based procedural content generation. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 318–328 (2019)
Gauerhof, L., Munk, P., Burton, S.: Structuring validation targets of a machine learning function applied to automated driving. In: Gallina, B., Skavhaug, A., Bitsch, F. (eds.) SAFECOMP 2018. LNCS, vol. 11093, pp. 45–58. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99130-6_4
de Gelder, E., Paardekooper, J., Op den Camp, O., De Schutter, B.: Safety assessment of automated vehicles: how to determine whether we have collected enough field data? Traffic Inj. Prev. 20(sup1), S162–S170 (2019)
de Gelder, E., et al.: Ontology for scenarios for the assessment of automated vehicles. arXiv preprint arXiv:2001.11507 (2020)
Gladisch, C., Heinz, T., Heinzemann, C., Oehlerking, J., von Vietinghoff, A., Pfitzer, T.: Experience paper: search-based testing in automated driving control applications. In: 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 26–37. IEEE (2019)
Hauer, F., Gerostathopoulos, I., Schmidt, T., Pretschner, A.: Clustering traffic scenarios using mental models as little as possible. In: IEEE Intelligent Vehicles Symposium (IV), pp. 1007–1012. IEEE (2020)
Hauer, F., Pretschner, A., Holzmüller, B.: Fitness functions for testing automated and autonomous driving systems. In: Romanovsky, A., Troubitsyna, E., Bitsch, F. (eds.) SAFECOMP 2019. LNCS, vol. 11698, pp. 69–84. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-26601-1_5
Hauer, F., Schmidt, T., Holzmüller, B., Pretschner, A.: Did we test all scenarios for automated and autonomous driving systems? In: IEEE Intelligent Transportation Systems Conference (ITSC), pp. 2950–2955. IEEE (2019)
Huynh, T., Gambi, A., Fraser, G.: AC3R: automatically reconstructing car crashes from police reports. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 31–34. IEEE (2019)
Kalra, N., Paddock, S.M.: Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability? Transp. Res. Part A Policy Pract. 94, 182–193 (2016)
Koopman, P., Fratrik, F.: How many operational design domains, objects, and events? In: AAAI Workshop on Artificial Intelligence Safety (2019)
Koopman, P., Kane, A., Black, J.: Credible autonomy safety argumentation. In: 27th Safety-Critical Systems Symposium (2019)
Koopman, P., Wagner, M.: Challenges in autonomous vehicle testing and validation. SAE Int. J. Transp. Saf. 4(1), 15–24 (2016)
Krajewski, R., Bock, J., Kloeker, L., Eckstein, L.: The highD dataset: a drone dataset of naturalistic vehicle trajectories on German highways for validation of highly automated driving systems. In: IEEE Intelligent Transportation Systems Conference (ITSC), pp. 2118–2125 (2018)
Li, G., et al.: AV-FUZZER: finding safety violations in autonomous driving systems. In: 31st International Symposium on Software Reliability Engineering (ISSRE), pp. 25–36. IEEE (2020)
Menzel, T., Bagschik, G., Isensee, L., Schomburg, A., Maurer, M.: From functional to logical scenarios: detailing a keyword-based scenario description for execution in a simulation environment. In: IEEE Intelligent Vehicles Symposium (IV), pp. 2383–2390 (2019)
Menzel, T., Bagschik, G., Maurer, M.: Scenarios for development, test and validation of automated vehicles. In: IEEE Intelligent Vehicles Symposium (IV), pp. 1821–1827 (2018)
Mullins, G.E., Stankiewicz, P.G., Gupta, S.K.: Automated generation of diverse and challenging scenarios for test and evaluation of autonomous vehicles. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 1443–1450 (2017)
Nilsson, J., Silvlin, J., Brannstrom, M., Coelingh, E., Fredriksson, J.: If, when, and how to perform lane change maneuvers on highways. IEEE Intell. Transp. Syst. Mag. 8(4), 68–78 (2016)
Riedmaier, S., Ponn, T., Ludwig, D., Schick, B., Diermeyer, F.: Survey on scenario-based safety assessment of automated vehicles. IEEE Access 8, 87456–87477 (2020)
Roesener, C., et al.: A comprehensive evaluation approach for highly automated driving. In: 25th International Technical Conference on the Enhanced Safety of Vehicles (ESV) National Highway Traffic Safety Administration (2017)
Satopaa, V., Albrecht, J., Irwin, D., Raghavan, B.: Finding a “kneedle” in a haystack: detecting knee points in system behavior. In: IEEE International Conference on Distributed Computing Systems Workshops, pp. 166–171 (2011)
Tkachenko, P., Zhou, J., del Re, L.: Unsupervised clustering of highway motion patterns. In: IEEE Intelligent Transportation Systems Conference, pp. 2337–2342 (2019)
Wachenfeld, W., Winner, H.: The release of autonomous vehicles. In: Maurer, M., Gerdes, J.C., Lenz, B., Winner, H. (eds.) Autonomous Driving, pp. 425–449. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-662-48847-8_21
Zhou, J., del Re, L.: Identification of critical cases of ADAS safety by FOT based parameterization of a catalogue. In: IEEE Asian Control Conference, pp. 453–458 (2017)
Zhou, J., del Re, L.: Reduced complexity safety testing for ADAS & ADF. IFAC-PapersOnLine 50(1), 5985–5990 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kolb, N., Hauer, F., Golagha, M., Pretschner, A. (2022). Data-Driven Assessment of Parameterized Scenarios for Autonomous Vehicles. In: Trapp, M., Saglietti, F., Spisländer, M., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2022. Lecture Notes in Computer Science, vol 13414. Springer, Cham. https://doi.org/10.1007/978-3-031-14835-4_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-14835-4_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14834-7
Online ISBN: 978-3-031-14835-4
eBook Packages: Computer ScienceComputer Science (R0)