research-article

Trajectory Prediction in Heterogeneous Environment via Attended Ecology Embedding

Authors:

Hong-Han Shuai,

Wen-Huang ChengAuthors Info & Claims

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Pages 202 - 210

https://doi.org/10.1145/3394171.3413602

Published: 12 October 2020 Publication History

Abstract

Trajectory prediction is a highly desirable feature for safe navigation or autonomous vehicle in complex traffic. In this paper, we consider the practical environment of predicting trajectory in the heterogeneous traffic ecology. The proposed method has various applications in trajectory prediction problems and also in applied fields beyond tracking. One challenge stands out of the trajectory prediction-heterogeneous environment. Particularly, many factors should be considered in the environments, i.e., multiple types of road-agents, social interactions and terrains. The information is complicated and large that may result in inaccurate trajectory prediction. We propose two social and visual enforced attention modules to circumvent the problem and a variant of an Info-GAN structure to predict the trajectory with multi-modal behaviors. Experimental results show that the proposed method significantly outperforms state-of-the-art methods in both heterogeneous and homogeneous real environments.

Supplementary Material

ZIP File (mmfp1196aux.zip)

zip file include a Supplemental_Material.pdf that present more results from our model.

Download
20.69 MB

MP4 File (3394171.3413602.mp4)

This is a presentation video of our work "Trajectory Prediction in Heterogeneous Environment via Attended Ecology Embedding". Only includes the high level description in this video, for more technique detail please refers the full paper.

Download
36.73 MB

References

[1]

2019. Waymo Open Dataset: An autonomous driving dataset.

[2]

Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. 2016. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition. 961--971.

[3]

Alexandre Alahi, Vignesh Ramanathan, and Li Fei-Fei. 2014. Socially-aware large-scale crowd forecasting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2203--2210.

Digital Library

[4]

Javad Amirian, Jean-Bernard Hayet, and Julien Pettré. 2019. Social ways: Learning multi-modal distributions of pedestrian trajectories with GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 0--0.

[5]

Graeme Best and Robert Fitch. 2015. Bayesian intention inference for trajectory prediction with an unknown goal destination. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 5817--5823.

[6]

Rohan Chandra, Uttaran Bhattacharya, Aniket Bera, and Dinesh Manocha. 2019 a. Traphic: Trajectory prediction in dense and heterogeneous traffic using weighted interactions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8483--8492.

[7]

Rohan Chandra, Uttaran Bhattacharya, Aniket Bera, and Dinesh Manocha. 2019 b. TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[8]

Rohan Chandra, Uttaran Bhattacharya, Christian Roncal, Aniket Bera, and Dinesh Manocha. 2019 c. RobustTP: End-to-End Trajectory Prediction for Heterogeneous Road-Agents in Dense Traffic with Noisy Sensor Inputs. In ACM Computer Science in Cars Symposium. 1--9.

[9]

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3213--3223.

[10]

Nachiket Deo and Mohan M Trivedi. 2018. Convolutional social pooling for vehicle trajectory prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1468--1476.

[11]

Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. 2018. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2255--2264.

[12]

Dirk Helbing and Peter Molnar. 1995. Social force model for pedestrian dynamics. Physical review E, Vol. 51, 5 (1995), 4282.

[13]

Xinyu Huang, Xinjing Cheng, Qichuan Geng, Binbin Cao, Dingfu Zhou, Peng Wang, Yuanqing Lin, and Ruigang Yang. 2018. The apolloscape dataset for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 954--960.

[14]

Rudolph Emil Kalman. 1960. A new approach to linear filtering and prediction problems. Journal of basic Engineering, Vol. 82, 1 (1960), 35--45.

[15]

Vasiliy Karasev, Alper Ayvaci, Bernd Heisele, and Stefano Soatto. 2016. Intent-aware long-term prediction of pedestrian motion. In 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2543--2549.

[16]

Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. 2012. Activity forecasting. In European Conference on Computer Vision. Springer, 201--214.

Digital Library

[17]

Vineet Kosaraju, Amir Sadeghian, Roberto Mart'in-Mart'in, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. 2019. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. In Advances in Neural Information Processing Systems. 137--146.

[18]

Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. 2017. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 336--345.

[19]

Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. 2007. Crowds by example. In Computer graphics forum, Vol. 26. Wiley Online Library, 655--664.

[20]

Jian Li, Yabiao Wang, Changan Wang, Ying Tai, Jianjun Qian, Jian Yang, Chengjie Wang, Jilin Li, and Feiyue Huang. 2019. DSFD: Dual Shot Face Detector. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[21]

Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander G Hauptmann, and Li Fei-Fei. 2019. Peeking into the future: Predicting future person activities and locations in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5725--5734.

[22]

Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3431--3440.

[23]

Yuexin Ma, Xinge Zhu, Sibo Zhang, Ruigang Yang, Wenping Wang, and Dinesh Manocha. 2019. Trafficpredict: Trajectory prediction for heterogeneous traffic-agents. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 6120--6127.

Digital Library

[24]

Vijay Mahadevan, Weixin Li, Viral Bhalodia, and Nuno Vasconcelos. 2010. Anomaly detection in crowded scenes. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 1975--1981.

[25]

Andreas Møgelmose, Mohan M Trivedi, and Thomas B Moeslund. 2015. Trajectory analysis and prediction for improved pedestrian safety: Integrated framework and evaluations. In 2015 IEEE Intelligent Vehicles Symposium (IV). IEEE, 330--335.

[26]

Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. 2020. Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction. CVPR (2020).

[27]

Stefano Pellegrini, Andreas Ess, and Luc Van Gool. 2010. Improving data association by joint modeling of pedestrian trajectories and groupings. In European conference on computer vision. Springer, 452--465.

[28]

Mark Pfeiffer, Ulrich Schwesinger, Hannes Sommer, Enric Galceran, and Roland Siegwart. 2016. Predicting actions to act predictably: Cooperative partial motion planning with maximum entropy models. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2096--2101.

Digital Library

[29]

Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. 2016. Learning social etiquette: Human trajectory understanding in crowded scenes. In European conference on computer vision. Springer, 549--565.

[30]

Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. 2019. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1349--1358.

[31]

Amir Sadeghian, Ferdinand Legros, Maxime Voisin, Ricky Vesel, Alexandre Alahi, and Silvio Savarese. 2018. Car-net: Clairvoyant attentive recurrent network. In Proceedings of the European Conference on Computer Vision (ECCV). 151--167.

[32]

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. 2017. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV). 618--626. https://doi.org/10.1109/ICCV.2017.74

[33]

Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv e-prints, Article arXiv:1409.1556 (Sep 2014), arXiv:1409.1556 pages. arxiv: 1409.1556 [cs.CV]

[34]

Anirudh Vemula, Katharina Muelling, and Jean Oh. 2018. Social attention: Modeling attention in human crowds. In 2018 IEEE international Conference on Robotics and Automation (ICRA). IEEE, 1--7.

Digital Library

[35]

Jacob Walker, Abhinav Gupta, and Martial Hebert. 2014. Patch to the future: Unsupervised visual prediction. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 3302--3309.

Digital Library

[36]

Kota Yamaguchi, Alexander C Berg, Luis E Ortiz, and Tamara L Berg. 2011. Who are you with and where are you going?. In CVPR 2011. IEEE, 1345--1352.

Digital Library

[37]

Shuai Yi, Hongsheng Li, and Xiaogang Wang. 2016. Pedestrian behavior modeling from stationary crowds with applications to intelligent surveillance. IEEE transactions on image processing, Vol. 25, 9 (2016), 4354--4368.

[38]

Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2018. Self-attention generative adversarial networks. arXiv preprint arXiv:1805.08318 (2018).

[39]

Pu Zhang, Wanli Ouyang, Pengfei Zhang, Jianru Xue, and Nanning Zheng. 2019. Sr-lstm: State refinement for lstm towards pedestrian trajectory prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12085--12094.

[40]

Bolei Zhou, Xiaoou Tang, and Xiaogang Wang. 2015. Learning collective crowd behaviors with dynamic pedestrian-agents. International Journal of Computer Vision, Vol. 111, 1 (2015), 50--68.

Digital Library

[41]

Tao Zhuo, Zhiyong Cheng, Peng Zhang, Yongkang Wong, and Mohan Kankanhalli. 2019. Explainable Video Action Reasoning via Prior Knowledge and State Transitions. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM '19). Association for Computing Machinery, New York, NY, USA, 521--529. https://doi.org/10.1145/3343031.3351040

Digital Library

[42]

Brian D Ziebart, Nathan Ratliff, Garratt Gallagher, Christoph Mertz, Kevin Peterson, J Andrew Bagnell, Martial Hebert, Anind K Dey, and Siddhartha Srinivasa. 2009. Planning-based prediction for pedestrians. In 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 3931--3936.

Cited By

Dou WLu L(2024)SISGAN: A Generative Adversarial Network Pedestrian Trajectory Prediction Model Combining Interaction Information and Scene InformationApplied Sciences10.3390/app1420953714:20(9537)Online publication date: 18-Oct-2024
https://doi.org/10.3390/app14209537
Monjurul Karim MQin RWang Y(2024)Fusion-GRU: A Deep Learning Model for Future Bounding Box Prediction of Traffic Agents in Risky Driving VideosTransportation Research Record: Journal of the Transportation Research Board10.1177/036119812412305402678:9(699-709)Online publication date: 28-Feb-2024
https://doi.org/10.1177/03611981241230540
Bhaskara RViswanath HBera A(2024)Trajectory Prediction for Robot Navigation using Flow-Guided Markov Neural Operator2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10611154(15209-15216)Online publication date: 13-May-2024
https://doi.org/10.1109/ICRA57147.2024.10611154
Show More Cited By

Index Terms

Trajectory Prediction in Heterogeneous Environment via Attended Ecology Embedding
1. Computer systems organization
  1. Embedded and cyber-physical systems
    1. Robotics
      1. Robotic autonomy
2. Computing methodologies
  1. Artificial intelligence
    1. Computer vision

Recommendations

Freeway Trajectory Prediction via SpatiotemporalTransformers
CACML '24: Proceedings of the 2024 3rd Asia Conference on Algorithms, Computing and Machine Learning

For autonomous driving, accurate trajectory prediction is paramount, necessitating effective harnessing of spatiotemporal data. This study proposes an innovative Spatiotemporal Transformer-based model, enhancing trajectory prediction precision by ...
Data-Driven Vehicle Trajectory Prediction
SIGSIM-PADS '16: Proceedings of the 2016 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation

Vehicle trajectory or route prediction is useful in online, data-driven transportation simulation to predict future traffic patterns and congestion, among other uses. The various approaches to route prediction have varying degrees of data required to ...
Spatial-Temporal Attention Networks for Vehicle Trajectory Prediction
ICCAI '22: Proceedings of the 8th International Conference on Computing and Artificial Intelligence

Predicting the future trajectory of vehicles is essential to the safety of autonomous driving. However, due to the uncertainty of the future behavior of vehicles and the complexity of interactions between vehicles, reasonable and accurate trajectory ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

October 2020

4889 pages

ISBN:9781450379885

DOI:10.1145/3394171

General Chairs:
Chang Wen Chen
Chinese University of Hong Kong, Shenzhen, China
,
Rita Cucchiara
UNIMORE, Italy
,
Xian-Sheng Hua
Alibaba Group, China
,
Program Chairs:
Guo-Jun Qi
Futurewei Technologies, USA
,
Elisa Ricci
UNITN & Fondazione Bruno Kessler, Italy
,
Zhengyou Zhang
Tencent, China
,
Roger Zimmermann
National University of Singapore, Singapore

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Ministry of Science and Technology, Taiwan

Conference

MM '20

Sponsor:

SIGMM

MM '20: The 28th ACM International Conference on Multimedia

October 12 - 16, 2020

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
385
Total Downloads

Downloads (Last 12 months)44
Downloads (Last 6 weeks)3

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dou WLu L(2024)SISGAN: A Generative Adversarial Network Pedestrian Trajectory Prediction Model Combining Interaction Information and Scene InformationApplied Sciences10.3390/app1420953714:20(9537)Online publication date: 18-Oct-2024
https://doi.org/10.3390/app14209537
Monjurul Karim MQin RWang Y(2024)Fusion-GRU: A Deep Learning Model for Future Bounding Box Prediction of Traffic Agents in Risky Driving VideosTransportation Research Record: Journal of the Transportation Research Board10.1177/036119812412305402678:9(699-709)Online publication date: 28-Feb-2024
https://doi.org/10.1177/03611981241230540
Bhaskara RViswanath HBera A(2024)Trajectory Prediction for Robot Navigation using Flow-Guided Markov Neural Operator2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10611154(15209-15216)Online publication date: 13-May-2024
https://doi.org/10.1109/ICRA57147.2024.10611154
Patachi ALeon F(2023)Multiagent Multimodal Trajectory Prediction in Urban Traffic Scenarios Using a Neural Network-Based SolutionMathematics10.3390/math1108192311:8(1923)Online publication date: 19-Apr-2023
https://doi.org/10.3390/math11081923
Cai YMa ZLu CWang CHe G(2023)Global Representation Guided Adaptive Fusion Network for Stable Video Crowd CountingIEEE Transactions on Multimedia10.1109/TMM.2022.318924625(5222-5233)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TMM.2022.3189246
Golchoubian MGhafurian MDautenhahn KAzad N(2023)Pedestrian Trajectory Prediction in Pedestrian-Vehicle Mixed Environments: A Systematic ReviewIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.329119624:11(11544-11567)Online publication date: Nov-2023
https://doi.org/10.1109/TITS.2023.3291196
Zhou ZHuang GSu ZLi YHua W(2023)Dynamic Attention-Based CVAE-GAN for Pedestrian Trajectory PredictionIEEE Robotics and Automation Letters10.1109/LRA.2022.32315318:2(704-711)Online publication date: Feb-2023
https://doi.org/10.1109/LRA.2022.3231531
Dang HKorbmacher RTordeux AGaudou BVerstaevel N(2023)TTC-SLSTM: Human Trajectory Prediction Using Time-to-Collision Interaction Energy2023 15th International Conference on Knowledge and Systems Engineering (KSE)10.1109/KSE59128.2023.10299443(1-6)Online publication date: 18-Oct-2023
https://doi.org/10.1109/KSE59128.2023.10299443
Golchoubian MGhafurian MDautenhahn KAzad N(2023)Polar Collision Grids: Effective Interaction Modelling for Pedestrian Trajectory Prediction in Shared Space Using Collision Checks2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC)10.1109/ITSC57777.2023.10422509(791-798)Online publication date: 24-Sep-2023
https://doi.org/10.1109/ITSC57777.2023.10422509
Korbmacher RDang HTordeux A(2023)Predicting pedestrian trajectories at different densities: A multi-criteria empirical analysisPhysica A: Statistical Mechanics and its Applications10.1016/j.physa.2023.129440(129440)Online publication date: Dec-2023
https://doi.org/10.1016/j.physa.2023.129440
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents