Cited By
View all- Li XLi JGao K(2024)An Ethical Behavioral Decision Algorithm Awaring of VRUs for Autonomous Vehicle Trajectory PlanningIEEE Access10.1109/ACCESS.2024.342210712(91541-91550)Online publication date: 2024
SCTrans: Constructing a Large Public Scenario Dataset for Simulation Testing of Autonomous Driving Systems
Authors: 
Jiarun Dai, Bufan Gao, Mingyuan Luo, Zongan Huang, Zhongrui Li, Yuan Zhang, Min YangAuthors Info & Claims
Article No.: 50, Pages 1 - 13
Abstract
For the safety assessment of autonomous driving systems (ADS), simulation testing has become an important complementary technique to physical road testing. In essence, simulation testing is a scenario-driven approach, whose effectiveness is highly dependent on the quality of given simulation scenarios. Moreover, simulation scenarios should be encoded into well-formatted files, otherwise, ADS simulation platforms cannot take them as inputs. Without large public datasets of simulation scenario files, both industry and academic applications of ADS simulation testing are hindered.
To fill this gap, we propose a transformation-based approach SCTrans to construct simulation scenario files, utilizing existing traffic scenario datasets (i.e., naturalistic movement of road users recorded on public roads) as data sources. Specifically, we try to transform existing traffic scenario recording files into simulation scenario files that are compatible with the most advanced ADS simulation platforms, and this task is formalized as a Model Transformation Problem. Following this idea, we construct a dataset consisting of over 1,900 diverse simulation scenarios, each of which can be directly used to test the state-of-the-art ADSs (i.e., Apollo and Autoware) via high-fidelity simulators (i.e., Carla and LGSVL). To further demonstrate the utility of our dataset, we showcase that it can boost the collision-finding capability of existing simulation-based ADS fuzzers, helping identify about seven times more unique ADS-involved collisions within the same time period. By analyzing these collisions at the code level, we identify nine safety-critical bugs of Apollo and Autoware, each of which can be stably exploited to cause vehicle crashes. Till now, four of them have been confirmed.
References
[1]
ASAM OpenDRIVE. https://www.asam.net/standards/detail/opendrive, 2023.
[2]
ASAM OpenSCENARIO. https://www.asam.net/standards/detail/openscenario, 2023.
[3]
Autonomous Vehicle Collision Reports. https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-collision-reports/, 2023.
[4]
Autoware-AI. https://github.com/autowarefoundation/autoware, 2023.
[5]
Autoware Vector Map. https://github.com/autowarefoundation/autoware_common/blob/main/tmp/lanelet2_extension/docs/lanelet2_format_extension.md, 2023.
[6]
Carla Map Converter. https://gitlab.com/autowarefoundation/autoware.ai/utilities/-/tree/master/lanelet_aisan_converter, 2023.
[7]
Carla Map Meshes. https://carla.readthedocs.io/en/0.9.7/how_to_make_a_new_map/, 2023.
[8]
Carla Scenario Runner. https://github.com/carla-simulator/scenario_runner, 2023.
[9]
CarMaker. https://ipg-automotive.com/en/products-solutions/software/carmaker/, 2023.
[10]
Commonroad XML. https://gitlab.lrz.de/tum-cps/commonroad-scenarios/-/blob/master/documentation/XML_commonRoad_2020a.pdf, 2023.
[11]
Companies that Develop Autonomous Driving. https://aimagazine.com/technology/top-10-companies-developing-autonomous-vehicle-technology, 2023.
[12]
Convert XML Schema into Ecore Meta-Model. https://www.eclipse.org/modeling/emf/docs/1.x/tutorials/xlibmod/xlibmod_emf1.1.html, 2023.
[13]
Cyber-RT. https://cyber-rt.readthedocs.io/en/latest/, 2023.
[14]
Dataset Converter. https://commonroad.in.tum.de/tools/dataset-converters, 2023.
[15]
EMF Validation. https://www.eclipse.org/emf-validation, 2023.
[16]
Esmini. https://github.com/esmini/esmini, 2023.
[17]
HighD CSV. https://www.highd-dataset.com/format, 2023.
[18]
HighD Map. https://gitlab.lrz.de/tum-cps/dataset-converters/-/blob/master/src/highD/map_utils.py, 2023.
[19]
InD CSV. https://www.ind-dataset.com/format, 2023.
[20]
ISO 21448. https://www.iso.org/standard/77490.html, 2023.
[21]
Levelxdata. https://levelxdata.com/, 2023.
[22]
LGSVL Map AssetBundle. https://www.svlsimulator.com/docs/user-interface/web/library/#maps, 2023.
[23]
LGSVL VSE Scenario. https://www.svlsimulator.com/docs/visual-scenario-editor/vse-inspector/, 2023.
[24]
Map Asset Generation Tool. https://www.svlsimulator.com/docs/archive/2020.06/unity-help/, 2023.
[25]
Open-sourced Version of Baidu Apollo. https://github.com/ApolloAuto/apollo, 2023.
[26]
OpenPilot. https://comma.ai/openpilot/, 2023.
[27]
OSC-ASKS. https://github.com/asam-oss/OSC-ALKS-scenarios, 2023.
[28]
OSM. :https://www.openstreetmap.org, 2023.
[29]
PreScan. https://m.tass.plm.automation.siemens.com/cn/prescan-2, 2023.
[30]
ROS. https://www.ros.org/, 2023.
[31]
Safety Pool. https://www.safetypool.ai/, 2023.
[32]
Uber. https://www.uber.com/, 2023.
[33]
Unity Engine. https://unity.com/, 2023.
[34]
Unreal Engine. https://www.unrealengine.com/, 2023.
[35]
Waymo. https://waymo.com/, 2023.
[36]
XdoTool. https://github.com/jordansissel/xdotool, 2023.
[37]
M. Althoff, M. Koschi, and S. Manzinger. CommonRoad: Composable Benchmarks for Motion Planning on Roads. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), 2017.
[38]
M. Althoff, S. Urban, and M. Koschi. Automatic Conversion of Road Networks from Opendrive to Lanelets. In Proceedings of the IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI), 2018.
[39]
S. Baltodano, S. Sibi, N. Martelaro, N. Gowda, and W. Ju. The RRADS Platform: A Real Road Autonomous Driving Simulator. In Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI), 2015.
[40]
P. Bender, J. Ziegler, and C. Stiller. Lanelets: Efficient Map Representation for Autonomous Driving. In Proceedings of the IEEE Intelligent Vehicles Symposium Proceedings, 2014.
[41]
J. Bock, R. Krajewski, T. Moers, S. Runde, L. Vater, and L. Eckstein. The Ind Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), 2020.
[42]
G. J. Brostow, J. Fauqueur, and R. Cipolla. Semantic Object Classes in Video: A High-definition Ground Truth Database. Pattern Recognition Letters, 2009.
[43]
H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. Nuscenes: A Multimodal Dataset for Autonomous Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[44]
Y. Cao, N. Wang, C. Xiao, D. Yang, J. Fang, R. Yang, Q. A. Chen, M. Liu, and B. Li. Invisible for Both Camera and Lidar: Security of Multi-sensor Fusion Based Perception in Autonomous Driving under Physical-world Attacks. In Proceedings of the 42nd IEEE Symposium on Security and Privacy (SP), 2021.
[45]
M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d Tracking And Forecasting with Rich Maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2019.
[46]
H. Chen, H. Ren, R. Li, G. Yang, and S. Ma. Generating Autonomous Driving Test Scenarios Based on OpenSCENARIO. In Proceedings of the 9th International Conference on Dependable Systems and Their Applications (DSA), 2022.
[47]
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2016.
[48]
E. De Gelder, J.-P. Paardekooper, A. K. Saberi, H. Elrofai, O. O. den Camp, S. Kraines, J. Ploeg, and B. De Schutter. Towards An Ontology for Scenario Definition for The Assessment of Automated Vehicles: An Object-Oriented Framework. IEEE Transactions on Intelligent Vehicles, 2022.
[49]
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An Open Urban Driving Simulator. In Conference on Robot Learning, 2017.
[50]
D. Friedman and A. B. Dieng. The Vendi Score: A Diversity Evaluation Metric for Machine Learning. arXiv preprint arXiv:2210.02410, 2022.
[51]
C. W. Gran. HD-maps in Autonomous Driving. M.S. thesis, 2019.
[52]
J. C. Han and Z. Q. Zhou. Metamorphic Fuzz Testing of Autonomous Vehicles. In Proceedings of the 42nd IEEE/ACM International Conference on Software Engineering Workshops, 2020.
[53]
Z. Hu, S. Guo, Z. Zhong, and K. Li. Coverage-based Scene Fuzzing for Virtual Autonomous Driving Testing. arXiv preprint arXiv:2106.00873, 2021.
[54]
X. Huang, X. Cheng, Q. Geng, B. Cao, D. Zhou, P. Wang, Y. Lin, and R. Yang. The ApolloScape Dataset for Autonomous Driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018.
[55]
T. Huynh, A. Gambi, and G. Fraser. AC3R: Automatically Reconstructing Car Crashes from Police Reports. In Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 2019.
[56]
D. Karunakaran, J. S. Berrio, S. Worrall, and E. Nebot. Automatic Lane Change Scenario Extraction and Generation of Scenarios in OpenX Format from Realworld Data. arXiv preprint arXiv:2203.07521, 2022.
[57]
P. Kaur, S. Taghavi, Z. Tian, and W. Shi. A Survey on Simulators for Testing Self-driving Cars. In Proceedings of the 4th International Conference on Connected and Autonomous Driving (MetroCAD), 2021.
[58]
S. Kim, M. Liu, J. J. Rhee, Y. Jeon, Y. Kwon, and C. H. Kim. DriveFuzz: Discovering Autonomous Driving Bugs through Driving Quality-Guided Fuzzing. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2022.
[59]
R. Krajewski, J. Bock, L. Kloeker, and L. Eckstein. The Highd Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems. In Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC), 2018.
[60]
R. Krajewski, T. Moers, J. Bock, L. Vater, and L. Eckstein. The RounD Dataset: A Drone Dataset of Road User Trajectories at Roundabouts in Germany. In Proceedings of the 23rd IEEE International Conference on Intelligent Transportation Systems (ITSC), 2020.
[61]
D. Krajzewicz, G. Hertkorn, C. Rössel, and P. Wagner. SUMO (Simulation of Urban MObility)-An Open-source Traffic Simulation. In Proceedings of the 4th Middle East Symposium on Simulation and Modelling (MESM), 2002.
[62]
G. Li, Y. Li, S. Jha, T. Tsai, M. Sullivan, S. K. S. Hari, Z. Kalbarczyk, and R. Iyer. AV-FUZZER: Finding Safety Violations in Autonomous Driving Systems. In Proceedings of the 31st IEEE International Symposium on Software Reliability Engineering (ISSRE), 2020.
[63]
G. Li, Y. Li, S. Jha, T. Tsai, M. Sullivan, S. K. S. Hari, Z. Kalbarczyk, and R. Iyer. AV-FUZZER: Finding Safety Violations in Autonomous Driving Systems. In Proceedings of the 31st IEEE International Symposium on Software Reliability Engineering (ISSRE), 2020.
[64]
G. Lou, Y. Deng, X. Zheng, M. Zhang, and T. Zhang. Testing of Autonomous Driving Systems: Where Are We and Where Should We Go? In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2022.
[65]
J. Ma, X. Che, Y. Li, and E. M.-K. Lai. Traffic Scenarios for Automated Vehicle Testing: A Review of Description Languages And Systems. Machines, 2021.
[66]
S. Maierhofer, M. Klischat, and M. Althoff. Commonroad Scenario Designer: An Open-source Toolbox for Map Conversion and Scenario Creation for Autonomous Vehicles. In Proceedings of the 24th IEEE International Intelligent Transportation Systems Conference (ITSC), 2021.
[67]
T. Mens and P. Van Gorp. A Taxonomy of Model Transformation. Electronic Notes in Theoretical Computer Science, 2006.
[68]
T. Moers, L. Vater, R. Krajewski, J. Bock, A. Zlocki, and L. Eckstein. The ExiD Dataset: A Real-World Trajectory Dataset of Highly Interactive Highway Scenarios in Germany. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), 2022.
[69]
F. Montanari, C. Stadler, J. Sichermann, R. German, and A. Djanatliev. Maneuver-based Resimulation of Driving Scenarios Based on Real Driving Data. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), 2021.
[70]
R. Muller, Y. Man, Z. B. Celik, M. Li, and R. Gerdes. Drivetruth: Automated Autonomous Driving Dataset Generation for Security Applications. In Proceedings of the 4th International Workshop on Automotive and Autonomous Vehicle Security (AutoSec), 2022.
[71]
S. of Automotive Engineers (SAE). J3016 - Taxonomy and Definitions for Terms Related to On-road Motor Vehicle Automated Driving Systems, 2016.
[72]
A. Ouaknine, A. Newson, J. Rebut, F. Tupin, and P. Pérez. Carrada Dataset: Camera and Automotive Radar with Range-Angle-Doppler Annotations. In Proceedings of the 25th International Conference on Pattern Recognition (ICPR), 2021.
[73]
S. Pathrudkar, S. Venkataraman, D. Kanade, A. Ajayan, P. Gupta, S. Khatib, V. S. Indla, and S. Mukherjee. SAFR-AV: Safety Analysis of Autonomous Vehicles Using Real World Data-An End-to-End Solution for Real World Data Driven Scenario-based Testing for Pre-certification of AV Stacks. arXiv preprint arXiv:2302.14601, 2023.
[74]
F. Poggenhans, J.-H. Pauls, J. Janosovits, S. Orf, M. Naumann, F. Kuhnt, and M. Mayr. Lanelet2: A High-Definition Map Framework for The Future of Automated Driving. In Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC), 2018.
[75]
V. Punzo, M. T. Borzacchiello, and B. Ciuffo. On The Assessment of Vehicle Trajectory Data Accuracy and Application to The Next Generation SIMulation (NGSIM) Program Data. Transportation Research Part C: Emerging Technologies, 2011.
[76]
R. Queiroz, T. Berger, and K. Czarnecki. GeoScenario: An Open DSL for Autonomous Driving Scenario Representation. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), 2019.
[77]
R. Rajamani. Vehicle Dynamics and Control. 2011.
[78]
A. Rebert, S. K. Cha, T. Avgerinos, J. Foote, D. Warren, G. Grieco, and D. Brumley. Optimizing Seed Selection for Fuzzing. In Proceedings of the 23rd USENIX Security Symposium (USENIX Security), 2014.
[79]
G. Rong, B. H. Shin, H. Tabatabaee, Q. Lu, S. Lemke, M. Možeiko, E. Boise, G. Uhm, M. Gerow, S. Mehta, et al. Lgsvl Simulator: A High Fidelity Simulator for Autonomous Driving. In Proceedings of the 23rd IEEE International Conference on Intelligent Transportation Systems (ITSC), 2020.
[80]
T. D. Son, A. Bhave, and H. Van der Auweraer. Simulation-based Testing Framework for Autonomous Driving Development. In Proceedings of the IEEE International Conference on Mechatronics (ICM), 2019.
[81]
D. Sportillo, A. Paljic, and L. Ojeda. On-road Evaluation of Autonomous Driving Training. In Proceedings of the 14th ACM/IEEE International Conference on HumanRobot Interaction (HRI), 2019.
[82]
D. Steinberg, F. Budinsky, E. Merks, and M. Paternostro. EMF: Eclipse Modeling Framework. 2008.
[83]
J. S. Sun, Y. C. Cao, Q. A. Chen, and Z. M. Mao. Towards Robust Lidar-based Perception in Autonomous Driving: General Black-Box Adversarial Sensor Attack and Countermeasures. In Proceedings of the 29th USENIX Security Symposium (Usenix Security), 2020.
[84]
P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[85]
A. Tenbrock, A. König, T. Keutgens, and H. Weber. The ConScenD Dataset: Concrete Scenarios from The HighD Dataset According to ALKS Regulation UNECE R157 in OpenX. In Proceedings of the IEEE Intelligent Vehicles Symposium Workshops (IV Workshops), 2021.
[86]
X. Wang, A.-K. Rettinger, M. T. B. Waez, and M. Althoff. Coupling Apollo with The CommonRoad Motion Planning Framework. In Proceedings of the FISITA Web Congress, 2020.
[87]
B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, et al. Argoverse 2: Next Generation Datasets for Self-driving Perception and Forecasting. arXiv preprint arXiv:2301.00493, 2023.
[88]
P. Xiao, Z. Shao, S. Hao, Z. Zhang, X. Chai, J. Jiao, Z. Li, J. Wu, K. Sun, K. Jiang, et al. Pandaset: Advanced Sensor Suite Dataset for Autonomous Driving. In Proceedings of the 24th IEEE International Intelligent Transportation Systems Conference (ITSC), 2021.
[89]
H. Yin and C. Berger. When to Use What Data Set for Your Self-driving Car Algorithm: An Overview of Publicly Available Driving Datasets. In Proceedings of the 20th IEEE International Conference on Intelligent Transportation Systems (ITSC), 2017.
[90]
F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell. Bdd100k: A Diverse Driving Dataset for Heterogeneous Multitask Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[91]
E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda. A Survey of Autonomous Driving: Common Practices and Emerging Technologies. IEEE Access, 2020.
[92]
X. Zhao, V. Robu, D. Flynn, K. Salako, and L. Strigini. Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing. In Proceedings of the 30th IEEE International Symposium on Software Reliability Engineering (ISSRE), 2019.
[93]
Z. Zhong, G. Kaiser, and B. Ray. Neural Network Guided Evolutionary Fuzzing for Finding Traffic Violations of Autonomous Vehicles. IEEE Transactions on Software Engineering (TSE), 2022.
[94]
J. Zhou, Y. Zhang, S. Guo, and Y. Guo. A Survey on Autonomous Driving System Simulators. In Proceedings of the IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), 2022.
[95]
X. Zhu, S. Wen, S. Camtepe, and Y. Xiang. Fuzzing: A Survey for Roadmap. ACM Computing Surveys (CSUR), 2022.
[96]
A. Zyner, S. Worrall, and E. M. Nebot. Acfr Five Roundabouts Dataset: Naturalistic Driving at Unsignalized Intersections. IEEE Intelligent Transportation Systems Magazine, 2019.
Index Terms
- SCTrans: Constructing a Large Public Scenario Dataset for Simulation Testing of Autonomous Driving Systems
Recommendations
Numerical Analysis of Tractor Accidents using Driving Simulator for Autonomous Driving Tractor
ICMRE'19: Proceedings of the 5th International Conference on Mechatronics and Robotics EngineeringAutonomous driving of automobiles is a hot research topic in recent years. The autonomous driving tractor also has been studied in the agricultural field as well as an autonomous driving automobile. On the other hand, tractor accidents frequently occur ...
Critical scenario identification for realistic testing of autonomous driving systems
AbstractAutonomous driving has become an important research area for road traffic, whereas testing of autonomous driving systems to ensure a safe and reliable operation remains an open challenge. Substantial real-world testing or massive driving data ...
Autonomous Driving: Investigating the Feasibility of Bimodal Take-Over Requests
Autonomous vehicles will need de-escalation strategies to compensate when reaching system limitations. Car-driver handovers can be considered one possible method to deal with system boundaries. The authors suggest a bimodal auditory and visual handover ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2024
2942 pages
ISBN:9798400702174
DOI:10.1145/3597503
- Co-chairs:
- Ana Paiva,
- Rui Abreu,
- Program Co-chairs:
- Abhik Roychoudhury,
- Margaret Storey
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
- Faculty of Engineering of University of Porto
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 06 February 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- Shanghai Rising-Star Program
- Shanghai Pilot Program for Basic Research
- Ministry of Industry and Information Technology of the People's Republic of China
Conference
ICSE '24
Sponsor:
ICSE '24: IEEE/ACM 46th International Conference on Software Engineering
April 14 - 20, 2024
Lisbon, Portugal
Acceptance Rates
Overall Acceptance Rate 276 of 1,856 submissions, 15%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 310Total Downloads
- Downloads (Last 12 months)310
- Downloads (Last 6 weeks)37
Reflects downloads up to 24 Sep 2024
Other Metrics
Citations
Cited By
View all- Li XLi JGao K(2024)An Ethical Behavioral Decision Algorithm Awaring of VRUs for Autonomous Vehicle Trajectory PlanningIEEE Access10.1109/ACCESS.2024.342210712(91541-91550)Online publication date: 2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in