research-article

Improving Movement Predictions of Traffic Actors in Bird's-Eye View Models using GANs and Differentiable Trajectory Rasterization

Authors:

Sai Yalamanchi,

Mohana Moorthy,

Nemanja DjuricAuthors Info & Claims

KDD '20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Pages 2340 - 2348

https://doi.org/10.1145/3394486.3403283

Published: 20 August 2020 Publication History

Abstract

One of the most critical pieces of the self-driving puzzle is the task of predicting future movement of surrounding traffic actors, which allows the autonomous vehicle to safely and effectively plan its future route in a complex world. Recently, a number of algorithms have been proposed to address this important problem, spurred by a growing interest of researchers from both industry and academia. Methods based on top-down scene rasterization on one side and Generative Adversarial Networks (GANs) on the other have shown to be particularly successful, obtaining state-of-the-art accuracies on the task of traffic movement prediction. In this paper we build upon these two directions and propose a raster-based conditional GAN architecture, powered by a novel differentiable rasterizer module at the input of the conditional discriminator that maps generated trajectories into the raster space in a differentiable manner. This simplifies the task for the discriminator as trajectories that are not scene-compliant are easier to discern, and allows the gradients to flow back forcing the generator to output better, more realistic trajectories. We evaluated the proposed method on a large-scale, real-world data set, showing that it outperforms state-of-the-art GAN-based baselines.

References

[1]

D. Silver, A. Huang, et al., "Mastering the game of go with deep neural networks and tree search," Nature, vol. 529, no. 7587, pp. 484--489, 2016.

[2]

D. Silver, T. Hubert, et al., "Mastering chess and shogi by self-play with a general reinforcement learning algorithm," arXiv preprint arXiv:1712.01815, 2017.

[3]

E. J. Topol, The patient will see you now: the future of medicine is in your hands. Tantor Media, 2015.

[4]

O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: Lessons learned from the 2015 mscoco image captioning challenge," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 4, pp. 652--663, 2017.

Digital Library

[5]

A. Z. Broder, P. Ciccolo, et al., "Search advertising using web relevance feedback," in Proceedings of the 17th ACM Conference on Information and Knowledge Management. ACM, 2008, pp. 1013--1022.

[6]

G. Nuti, M. Mirghaemi, P. Treleaven, and C. Yingsaeree, "Algorithmic trading," Computer, vol. 44, no. 11, pp. 61--69, 2011.

Digital Library

[7]

Y. N. Harari, "Reboot for the ai revolution," Nature News, vol. 550, no. 7676, p. 324, 2017.

[8]

O. Gusikhin, N. Rychtyckyj, and D. Filev, "Intelligent systems in the automotive industry: applications and trends," Knowledge and Information Systems, vol. 12, no. 2, pp. 147--168, 2007.

Digital Library

[9]

M. Hofmann, F. Neukart, and T. B"ack, "Artificial intelligence and data science in the automotive industry," arXiv preprint arXiv:1709.01989, 2017.

[10]

D. Singh and M. Singh, "Internet of vehicles for smart and safe driving," in 2015 international conference on connected vehicles and expo (ICCVE). IEEE, 2015, pp. 328--329.

[11]

K. N. Qureshi and A. H. Abdullah, "A survey on intelligent transportation systems," Middle-East Journal of Scientific Research, vol. 15, no. 5, pp. 629--642, 2013.

[12]

A. Shaout, D. Colella, and S. Awad, "Advanced driver assistance systems-past, present and future," in 2011 Seventh International Computer Engineering Conference (ICENCO'2011). IEEE, 2011, pp. 72--82.

[13]

S. Singh, "Critical reasons for crashes investigated in the national motor vehicle crash causation survey," National Highway Traffic Safety Administration, Tech. Rep. DOT HS 812 506, March 2018.

[14]

NHTSA, "2017 fatal motor vehicle crashes: Overview," National Highway Traffic Safety Administration, Tech. Rep. DOT HS 812 603, October 2018.

[15]

D. A. Pomerleau, "Alvinn: An autonomous land vehicle in a neural network," in Advances in neural information processing systems, 1989, pp. 305--313.

[16]

----, "Neural network perception for mobile robot guidance," Carnegie-Mellon Univ Pittsburgh PA Dept. of Computer Science, Tech. Rep., 1992.

[17]

C. Urmson and W. Whittaker, "Self-driving cars and the urban challenge," IEEE Intelligent Systems, vol. 23, no. 2, pp. 66--68, 2008.

Digital Library

[18]

J. Ziegler, P. Bender, M. Schreiber, et al., "Making bertha drive: An autonomous journey on a historic route," IEEE Intelligent Transportation Systems Magazine, vol. 6, pp. 8--20, 10 2015.

[19]

C. Urmson, J. A. Bagnell, C. Baker, M. Hebert, A. Kelly, R. Rajkumar, P. E. Rybski, S. Scherer, R. Simmons, S. Singh, et al., "Tartan racing: A multi-modal approach to the DARPA urban challenge," 2007.

[20]

C. Reinholtz, D. Hong, A. Wicks, A. Bacha, C. Bauman, R. Faruque, M. Fleming, C. Terwelp, T. Alberi, D. Anderson, et al., "Odin: Team VictorTango's entry in the DARPA urban challenge," in The DARPA Urban Challenge. Springer, 2009, pp. 125--162.

[21]

N. Djuric, V. Radosavljevic, H. Cui, T. Nguyen, F.-C. Chou, T.-H. Lin, N. Singh, and J. Schneider, "Uncertainty-aware short-term motion prediction of traffic actors for autonomous driving," arXiv preprint arXiv:1808.05819, 2018.

[22]

W. Luo, B. Yang, and R. Urtasun, "Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net," in Proceedings of the IEEE CVPR, 2018, pp. 3569--3577.

[23]

H. Cui, V. Radosavljevic, F.-C. Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, and N. Djuric, "Multimodal trajectory predictions for autonomous driving using deep convolutional networks," in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 2090--2096.

[24]

A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, "Social gan: Socially acceptable trajectories with generative adversarial networks," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2255--2264.

[25]

A. Sadeghian, V. Kosaraju, A. Sadeghian, N. Hirose, H. Rezatofighi, and S. Savarese, "Sophie: An attentive gan for predicting paths compliant to social and physical constraints," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 1349--1358.

[26]

V. Kosaraju, A. Sadeghian, R. Martín-Martín, I. Reid, H. Rezatofighi, and S. Savarese, "Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks," in Advances in Neural Information Processing Systems, 2019, pp. 137--146.

[27]

A. Cosgun, L. Ma, et al., "Towards full automated drive in urban environments: A demonstration in gomentum station, california," in IEEE Intelligent Vehicles Symposium, 2017, pp. 1811--1818. [Online]. Available: https://doi.org/10.1109/IVS.2017.7995969

Digital Library

[28]

M. Bansal, A. Krizhevsky, and A. Ogale, "Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst," arXiv preprint arXiv:1812.03079, 2018.

[29]

F.-C. Chou, T.-H. Lin, H. Cui, V. Radosavljevic, T. Nguyen, T.-K. Huang, M. Niedoba, J. Schneider, and N. Djuric, "Predicting motion of vulnerable road users using high-definition maps and efficient convnets," in Workshop on 'Machine Learning for Intelligent Transportation Systems' at Conference on Neural Information Processing Systems (MLITS), 2018.

[30]

Y. Chai, B. Sapp, M. Bansal, and D. Anguelov, "Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction," arXiv preprint arXiv:1910.05449, 2019.

[31]

J. Hong, B. Sapp, and J. Philbin, "Rules of the road: Predicting driving behavior with a convolutional model of semantic interactions," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 8454--8462.

[32]

S. Yalamanchi, T.-K. Huang, G. C. Haynes, and N. Djuric, "Long-term prediction of vehicle behavior using short-term uncertainty-aware trajectories and high-definition maps," in IEEE International Conference on Intelligent Transportation Systems (ITSC), 2020.

[33]

A. Jain, S. Casas, R. Liao, Y. Xiong, S. Feng, S. Segal, and R. Urtasun, "Discrete residual flow for probabilistic pedestrian behavior prediction," arXiv preprint arXiv:1910.08041, 2019.

[34]

D. Ridel, N. Deo, D. Wolf, and M. Trivedi, "Scene compliant trajectory forecast with agent-centric spatio-temporal grids," arXiv preprint arXiv:1909.07507, 2019.

[35]

M. Niedoba, H. Cui, K. Luo, D. Hegde, F.-C. Chou, and N. Djuric, "Improving movement prediction of traffic actors using off-road loss and bias mitigation," in Workshop on 'Machine Learning for Autonomous Driving' at Conference on Neural Information Processing Systems, 2019.

[36]

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in Advances in neural information processing systems, 2014, pp. 2672--2680.

Digital Library

[37]

B. Dai, S. Fidler, R. Urtasun, and D. Lin, "Towards diverse and natural image descriptions via a conditional gan," in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2970--2979.

[38]

Y. Zhang, Z. Gan, K. Fan, Z. Chen, R. Henao, D. Shen, and L. Carin, "Adversarial feature matching for text generation," in Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017, pp. 4006--4015.

[39]

T. Zhao, Y. Xu, M. Monfort, W. Choi, C. Baker, Y. Zhao, Y. Wang, and Y. N. Wu, "Multi-agent tensor fusion for contextual trajectory prediction," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 12, 126--12,134.

[40]

H. Cui, T. Nguyen, F.-C. Chou, T.-H. Lin, J. Schneider, D. Bradley, and N. Djuric, "Deep kinematic models for physically realistic prediction of vehicle trajectories," arXiv preprint arXiv:1908.00219, 2019.

[41]

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation," arXiv preprint arXiv:1801.04381, 2018.

[42]

A. Radford, L. Metz, and S. Chintala, "Unsupervised representation learning with deep convolutional generative adversarial networks," arXiv preprint arXiv:1511.06434, 2015.

[43]

M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein gan," arXiv preprint arXiv:1701.07875, 2017.

[44]

I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, "Improved training of wasserstein gans," CoRR, vol. abs/1704.00028, 2017. [Online]. Available: http://arxiv.org/abs/1704.00028

[45]

A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese, "Social lstm: Human trajectory prediction in crowded spaces," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 961--971.

[46]

J. Amirian, J.-B. Hayet, and J. Pettré, "Social ways: Learning multi-modal distributions of pedestrian trajectories with gans," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 0--0.

[47]

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, et al., "TensorFlow: Large-scale machine learning on heterogeneous systems," 2015. [Online]. Available: https://www.tensorflow.org/BIBentrySTDinterwordspacing

[48]

D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.

[49]

G. P. Meyer, A. Laddha, E. Kee, C. Vallespi-Gonzalez, and C. K. Wellington, "Lasernet: An efficient probabilistic 3d object detector for autonomous driving," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 12,677--12,686.

[50]

G. Máttyus and R. Urtasun, "Matching adversarial networks," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8024--8032.

Cited By

Sikora TPapić V(2024)Survey of Path Planning for Aerial Drone Inspection of Multiple Moving ObjectsDrones10.3390/drones81207058:12(705)Online publication date: 26-Nov-2024
https://doi.org/10.3390/drones8120705
Pazho ANoghre GKatariya VTabkhi H(2024)VT-Former: An Exploratory Study on Vehicle Trajectory Prediction for Highway Surveillance through Graph Isomorphism and Transformer2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00574(5651-5662)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00574
Park DJeong JYoon SJeong JYoon K(2024)T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-Specific Token Memory2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01427(15065-15076)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01427
Show More Cited By

Index Terms

Improving Movement Predictions of Traffic Actors in Bird's-Eye View Models using GANs and Differentiable Trajectory Rasterization
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object detection
      2. Computer vision tasks
        Scene understanding
        Vision for robotics
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Vehicle trajectory prediction on highways using bird eye view representations and deep learning
Abstract
This work presents a novel method for predicting vehicle trajectories in highway scenarios using efficient bird’s eye view representations and convolutional neural networks. Vehicle positions, motion histories, road configuration, and vehicle ...
Numerical Analysis of Tractor Accidents using Driving Simulator for Autonomous Driving Tractor
ICMRE'19: Proceedings of the 5th International Conference on Mechatronics and Robotics Engineering

Autonomous driving of automobiles is a hot research topic in recent years. The autonomous driving tractor also has been studied in the agricultural field as well as an autonomous driving automobile. On the other hand, tractor accidents frequently occur ...
Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions

Generative Adversarial Networks (GANs) is a novel class of deep generative models that has recently gained significant attention. GANs learn complex and high-dimensional distributions implicitly over images, audio, and data. However, there exist major ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

August 2020

3664 pages

ISBN:9781450379984

DOI:10.1145/3394486

General Chairs:
Rajesh Gupta
UC San Diego, USA
,
Yan Liu
USC, USA
,
Program Chairs:
Mohak Shah
LG Electronics, USA
,
Suju Rajan
Linkedin, USA
,
Publications Chairs:
Jiliang Tang
Michigan State, USA
,
B. Aditya Prakash
Georgia Tech, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 August 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '20

Sponsor:

KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

July 6 - 10, 2020

CA, Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
463
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)4

Reflects downloads up to 08 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sikora TPapić V(2024)Survey of Path Planning for Aerial Drone Inspection of Multiple Moving ObjectsDrones10.3390/drones81207058:12(705)Online publication date: 26-Nov-2024
https://doi.org/10.3390/drones8120705
Pazho ANoghre GKatariya VTabkhi H(2024)VT-Former: An Exploratory Study on Vehicle Trajectory Prediction for Highway Surveillance through Graph Isomorphism and Transformer2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00574(5651-5662)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00574
Park DJeong JYoon SJeong JYoon K(2024)T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-Specific Token Memory2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01427(15065-15076)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01427
Zerouali BBailek NTariq AKuriqi AGuermoui MAlharbi AKhafaga DEl-kenawy E(2024)Enhancing deep learning-based slope stability classification using a novel metaheuristic optimization algorithm for feature selectionScientific Reports10.1038/s41598-024-72588-514:1Online publication date: 18-Sep-2024
https://doi.org/10.1038/s41598-024-72588-5
Ngo HFang HWang H(2023)Cooperative Perception With V2V Communication for Autonomous VehiclesIEEE Transactions on Vehicular Technology10.1109/TVT.2023.326402072:9(11122-11131)Online publication date: Sep-2023
https://doi.org/10.1109/TVT.2023.3264020
Lin HLiu YLi SQu X(2023)How Generative Adversarial Networks Promote the Development of Intelligent Transportation Systems: A SurveyIEEE/CAA Journal of Automatica Sinica10.1109/JAS.2023.12374410:9(1781-1796)Online publication date: Sep-2023
https://doi.org/10.1109/JAS.2023.123744
Pourkeshavarz MChen CRasouli A(2023)Learn TAROT with MENTOR: A Meta-Learned Self-supervised Approach for Trajectory Prediction2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00770(8350-8359)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00770
Yadav MShahi SAhuja HBatra M(2023)Technological Reforms of Global Projects Using Artificial IntelligenceComputational Intelligence for Modern Business Systems10.1007/978-981-99-5354-7_20(391-405)Online publication date: 4-Nov-2023
https://doi.org/10.1007/978-981-99-5354-7_20
Schlauch CWirth CKlein N(2023)Informed Priors for Knowledge Integration in Trajectory PredictionMachine Learning and Knowledge Discovery in Databases: Research Track10.1007/978-3-031-43424-2_24(392-407)Online publication date: 18-Sep-2023
https://doi.org/10.1007/978-3-031-43424-2_24
Liu PWang XZhang C(2022)Data-Driven Human-Like Path Planning for Autonomous Driving Based on Imitation Learning2022 5th International Conference on Intelligent Autonomous Systems (ICoIAS)10.1109/ICoIAS56028.2022.9931262(119-124)Online publication date: 23-Sep-2022
https://doi.org/10.1109/ICoIAS56028.2022.9931262
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents