research-article

Open access

TrajFormer: Efficient Trajectory Classification with Transformers

Authors:

Roger ZimmermannAuthors Info & Claims

CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

Pages 1229 - 1237

https://doi.org/10.1145/3511808.3557481

Published: 17 October 2022 Publication History

Abstract

Transformers have been an efficient alternative to recurrent neural networks in many sequential learning tasks. When adapting transformers to modeling trajectories, we encounter two major issues. First, being originally designed for language modeling, transformers assume regular intervals between input tokens, which contradicts the irregularity of trajectories. Second, transformers often suffer high computational costs, especially for long trajectories. In this paper, we address these challenges by presenting a novel transformer architecture entitled TrajFormer. Our model first generates continuous point embeddings by jointly considering the input features and the information of spatio-temporal intervals, and then adopts a squeeze function to speed up the representation learning. Moreover, we introduce an auxiliary loss to ease the training of transformers using the supervision signals provided by all output tokens. Extensive experiments verify that our TrajFormer achieves a preferable speed-accuracy balance compared to existing approaches.

References

[1]

Zain Ul Abideen, Heli Sun, Zhou Yang, Rana Zeeshan Ahmad, Adnan Iftekhar, and Amir Ali. 2021. Deep Wide Spatial-Temporal Based Transformer Networks Modeling for the Next Destination According to the Taxi Driver Behavior Prediction. Applied Sciences, Vol. 11, 1 (2021), 17.

[2]

Xin Cao, Gao Cong, and Christian S Jensen. 2010. Mining significant semantic locations from GPS data. Proceedings of the VLDB Endowment, Vol. 3, 1--2 (2010), 1009--1020.

Digital Library

[3]

Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. 2018. Recurrent neural networks for multivariate time series with missing values. Scientific reports, Vol. 8, 1 (2018), 1--12.

[4]

Zaiben Chen, Heng Tao Shen, and Xiaofang Zhou. 2011. Discovering popular routes from trajectories. In 2011 IEEE 27th International Conference on Data Engineering. IEEE, 900--911.

Digital Library

[5]

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).

[6]

Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. 2021. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882 (2021).

[7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).

[8]

Xin Ding, Lu Chen, Yunjun Gao, Christian S Jensen, and Hujun Bao. 2018. UlTraMan: A unified platform for big trajectory data management and analytics. Proceedings of the VLDB Endowment, Vol. 11, 7 (2018), 787--799.

Digital Library

[9]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).

[10]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, Vol. 9, 8 (1997), 1735--1780.

[11]

Xiaocheng Huang, Yifang Yin, Simon Lim, Guanfeng Wang, Bo Hu, Jagannadan Varadarajan, Shaolin Zheng, Ajay Bulusu, and Roger Zimmermann. 2019. Grab-posisi: An extensive real-life gps trajectory dataset in southeast asia. In Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Prediction of Human Mobility. 1--10.

Digital Library

[12]

Md Amirul Islam, Sen Jia, and Neil DB Bruce. 2020. How much position information do convolutional neural networks encode? arXiv preprint arXiv:2001.08248 (2020).

[13]

Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Xiaojie Jin, Anran Wang, and Jiashi Feng. 2021. Token labeling: Training a 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet. arXiv preprint arXiv:2104.10858 (2021).

[14]

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2019. Reformer: The Efficient Transformer. In ICLR.

[15]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE, Vol. 86, 11 (1998), 2278--2324.

[16]

Xutao Li, Gao Cong, Xiao-Li Li, Tuan-Anh Nguyen Pham, and Shonali Krishnaswamy. 2015. Rank-geofm: A ranking based geographical factorization method for point of interest recommendation. In SIGIR. 433--442.

[17]

Xiucheng Li, Kaiqi Zhao, Gao Cong, Christian S Jensen, and Wei Wei. 2018. Deep representation learning for trajectory similarity computation. In 2018 IEEE 34th international conference on data engineering (ICDE). IEEE, 617--628.

[18]

Yuxuan Liang, Kun Ouyang, Hanshu Yan, Yiwei Wang, Zekun Tong, and Roger Zimmermann. 2021. Modeling Trajectories with Neural Ordinary Differential Equations. In IJCAI. 1498--1504.

[19]

Hongbin Liu, Hao Wu, Weiwei Sun, and Ickjai Lee. 2019. Spatio-temporal GRU for trajectory classification. In ICDM. IEEE, 1228--1233.

[20]

Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by Summarizing Long Sequences. In ICLR.

[21]

Xuanqing Liu, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. 2020. Learning to encode position for transformer with continuous dynamical model. In ICML. 6327--6335.

[22]

Cheng Long, Raymond Chi-Wing Wong, and HV Jagadish. 2014. Trajectory simplification: on minimizing the direction-based error. Proceedings of the VLDB Endowment, Vol. 8, 1 (2014), 49--60.

Digital Library

[23]

Sijie Ruan, Cheng Long, Jie Bao, Chunyang Li, Zisheng Yu, Ruiyuan Li, Yuxuan Liang, Tianfu He, and Yu Zheng. 2020. Learning to generate maps from trajectories. In AAAI, Vol. 34. 890--897.

[24]

Simonas vS altenis, Christian S Jensen, Scott T Leutenegger, and Mario A Lopez. 2000. Indexing the positions of continuously moving objects. In SIGMOD. 331--342.

[25]

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. In NAACL (Short Papers). 464--468.

[26]

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In ICML. 10347--10357.

[27]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.

[28]

Dong Wang, Junbo Zhang, Wei Cao, Jian Li, and Yu Zheng. 2018. When will you arrive? estimating travel time based on deep neural networks. In AAAI, Vol. 32.

[29]

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 (2020).

[30]

Hao Wu, Ziyang Chen, Weiwei Sun, Baihua Zheng, and Wei Wang. 2017. Modeling Trajectories with Recurrent Neural Networks. In IJCAI. 3083--3090.

[31]

Hao Xue, Flora Salim, Yongli Ren, and Nuria Oliver. 2021. MobTCast: Leveraging Auxiliary Trajectory Forecasting for Human Mobility Prediction. Advances in Neural Information Processing Systems, Vol. 34 (2021).

[32]

Quan Yuan, Gao Cong, Zongyang Ma, Aixin Sun, and Nadia Magnenat Thalmann. 2013. Time-aware point-of-interest recommendation. In SIGIR. 363--372.

[33]

Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. 2021. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In CVPR. 2340--2350.

[34]

Pengpeng Zhao, Haifeng Zhu, Yanchi Liu, Jiajie Xu, Zhixu Li, Fuzhen Zhuang, Victor S Sheng, and Xiaofang Zhou. 2019. Where to Go Next: A Spatio-Temporal Gated Network for Next POI Recommendation. In AAAI, Vol. 33. 5877--5884.

Digital Library

[35]

Yu Zheng, Quannan Li, Yukun Chen, Xing Xie, and Wei-Ying Ma. 2008a. Understanding mobility based on GPS data. In Proceedings of the 10th international conference on Ubiquitous computing. 312--321.

Digital Library

[36]

Yu Zheng, Like Liu, Longhao Wang, and Xing Xie. 2008b. Learning transportation mode from raw gps data for geographic applications on the web. In WWW. 247--256.

[37]

Yu Zheng, Xing Xie, Wei-Ying Ma, et al. 2010. GeoLife: A collaborative social networking service among user, location and trajectory. IEEE Data Eng. Bull., Vol. 33, 2 (2010), 32--39.

[38]

Yu Zhu, Hao Li, Yikang Liao, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. 2017. What to do next: modeling user behaviors by time-LSTM. In IJCAI. 3602--3608.

Cited By

Zhou SShang SChen LJensen CKalnis P(2025)RED: Effective Trajectory Representation Learning with Comprehensive InformationProceedings of the VLDB Endowment10.14778/3705829.370583018:2(80-92)Online publication date: 28-Feb-2025
https://doi.org/10.14778/3705829.3705830
Zhang ZYuan WFan ZSong XShibasaki R(2025)AISFuser: Encoding Maritime Graphical Representations With Temporal Attribute Modeling for Vessel Trajectory PredictionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2025.353177037:4(1571-1584)Online publication date: Apr-2025
https://doi.org/10.1109/TKDE.2025.3531770
Lin YZhou ZLiu YLv HWen HLi TLi YJensen CGuo SLin YWan H(2025)UniTE: A Survey and Unified Pipeline for Pre-Training Spatiotemporal Trajectory EmbeddingsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.352399637:3(1475-1494)Online publication date: Mar-2025
https://doi.org/10.1109/TKDE.2024.3523996
Show More Cited By

Index Terms

TrajFormer: Efficient Trajectory Classification with Transformers
1. Information systems
  1. Information systems applications
    1. Spatial-temporal systems

Recommendations

Trajectory Data Classification: A Review
Survey Papers and Regular Papers

This article comprehensively surveys the development of trajectory data classification. Considering the critical role of trajectory data classification in modern intelligent systems for surveillance security, abnormal behavior detection, crowd behavior ...
Trajectory Data Mining: An Overview
Survey Paper, Regular Papers and Special Section on Participatory Sensing and Crowd Intelligence

The advances in location-acquisition and mobile computing techniques have generated massive spatial trajectory data, which represent the mobility of a diversity of moving objects, such as people, vehicles, and animals. Many techniques have been proposed ...
Private Trajectory Data Publication for Trajectory Classification
Web Information Systems and Applications
Abstract
Trajectory classification (TC), i.e., predicting the class labels of moving objects based on their trajectories and other features, has many important real-world applications. Private trajectory data publication is to anonymize trajectory data, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

October 2022

5274 pages

ISBN:9781450392365

DOI:10.1145/3511808

General Chairs:
Mohammad Al Hasan
Indiana University Purdue University, Indianapolis, USA
,
Li Xiong
Emory University, Atlanta, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Singapore Ministry of Education Academic Research Fund Tier 1
key research project of Zhejiang Lab

Conference

CIKM '22

Sponsor:

CIKM '22: The 31st ACM International Conference on Information and Knowledge Management

October 17 - 21, 2022

GA, Atlanta, USA

Acceptance Rates

CIKM '22 Paper Acceptance Rate 621 of 2,257 submissions, 28%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
4,316
Total Downloads

Downloads (Last 12 months)1,858
Downloads (Last 6 weeks)188

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou SShang SChen LJensen CKalnis P(2025)RED: Effective Trajectory Representation Learning with Comprehensive InformationProceedings of the VLDB Endowment10.14778/3705829.370583018:2(80-92)Online publication date: 28-Feb-2025
https://doi.org/10.14778/3705829.3705830
Zhang ZYuan WFan ZSong XShibasaki R(2025)AISFuser: Encoding Maritime Graphical Representations With Temporal Attribute Modeling for Vessel Trajectory PredictionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2025.353177037:4(1571-1584)Online publication date: Apr-2025
https://doi.org/10.1109/TKDE.2025.3531770
Lin YZhou ZLiu YLv HWen HLi TLi YJensen CGuo SLin YWan H(2025)UniTE: A Survey and Unified Pipeline for Pre-Training Spatiotemporal Trajectory EmbeddingsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.352399637:3(1475-1494)Online publication date: Mar-2025
https://doi.org/10.1109/TKDE.2024.3523996
Huang HXie HXu ZLiu MXu YZhu T(2024)GenTrajRec: A Graph-Enhanced Trajectory Recovery Model Based on Signaling DataApplied Sciences10.3390/app1413593414:13(5934)Online publication date: 8-Jul-2024
https://doi.org/10.3390/app14135934
Luo KZhu YChen WWang KZhou ZRuan SLiang YLarson K(2024)Towards robust trajectory representationsProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/248(2243-2251)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/248
Song YDing JYuan JLiao QLi Y(2024)Controllable Human Trajectory Generation Using Profile-Guided Latent DiffusionACM Transactions on Knowledge Discovery from Data10.1145/370173619:1(1-25)Online publication date: 25-Oct-2024
https://dl.acm.org/doi/10.1145/3701736
Xie YZhang YYin YZhang SZhang YShah RZimmermann RXiao GCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Traj2Former: A Local Context-aware Snapshot and Sequential Dual Fusion Transformer for Trajectory ClassificationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681340(8053-8061)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681340
Xu XZhong TYu HZhou FTrajcevski G(2024)Overcoming Catastrophic Forgetting in Continual Fine-Grained Urban Flow InferenceACM Transactions on Spatial Algorithms and Systems10.1145/366052310:4(1-26)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3660523
Hong ZWang HDing YWang GHe TZhang D(2024)SmallMap: Low-cost Community Road Map Sensing with Uncertain Delivery BehaviorProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595968:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659596
Musleh MMokbel M(2024)Let's Speak Trajectories: A Vision to Use NLP Models for Trajectory Analysis TasksACM Transactions on Spatial Algorithms and Systems10.1145/365647010:2(1-25)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3656470
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten