research-article

Fredformer: Frequency Debiased Transformer for Time Series Forecasting

Authors:

Taichi Murayama,

Yasuko Matsubara,

Yasushi SakuraiAuthors Info & Claims

KDD '24: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 2400 - 2410

https://doi.org/10.1145/3637528.3671928

Published: 24 August 2024 Publication History

Abstract

The Transformer model has shown leading performance in time series forecasting. Nevertheless, in some complex scenarios, it tends to learn low-frequency features in the data and overlook high-frequency features, showing a frequency bias. This bias prevents the model from accurately capturing important high-frequency data features. In this paper, we undertake empirical analyses to understand this bias and discover that frequency bias results from the model disproportionately focusing on frequency features with higher energy. Based on our analysis, we formulate this bias and propose Fredformer, a Transformer-based framework designed to mitigate frequency bias by learning features equally across different frequency bands. This approach prevents the model from overlooking lower amplitude features important for accurate forecasting. Extensive experiments show the effectiveness of our proposed approach, which can outperform other baselines in different real-world time-series datasets. Furthermore, we introduce a lightweight variant of the Fredformer with an attention matrix approximation, which achieves comparable performance but with much fewer parameters and lower computation costs. The code is available at: https://github.com/chenzRG/Fredformer

Supplemental Material

MP4 File - Promo video of paper "Fredformer: Frequency Debiased Transformer for Time Series Forecasting"

A 2-minute promo video for the paper "Fredformer: Frequency Debiased Transformer for Time Series Forecasting." KDD2024, rtp1555.

Download
4.96 MB

References

[1]

Rob J. Hyndman Alysha M. De Livera and Ralph D. Snyder. 2011. Forecasting Time Series With Complex Seasonal Patterns Using Exponential Smoothing. J. Amer. Statist. Assoc. (2011), 1513--1527.

[2]

Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. 2018. Convolutional Sequence Modeling Revisited. (2018).

[3]

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. (2020). arxiv: 2004.05150 [cs.CL]

[4]

S.A. Broughton and K. Bryan. 2011. Discrete Fourier Analysis and Wavelets: Applications to Signal and Image Processing. (2011).

[5]

Daniela Calvetti. 1991. A Stochastic Roundoff Error Analysis for the Fast Fourier Transform. (1991).

[6]

Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Conguri Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. 2021. Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting. (2021).

[7]

Yen-Yu Chang, Fan-Yun Sun, Yueh-Hua Wu, and Shou-De Lin. 2018. A Memory-Network Based Solution for Multivariate Time-Series Forecasting. (2018). arxiv: 1809.02105 [cs.LG]

[8]

Zheng Chen, Ziwei Yang, Lingwei Zhu, Wei Chen, Toshiyo Tamura, Naoaki Ono, Md Altaf-Ul-Amin, Shigehiko Kanaya, and Ming Huang. 2023. Automated Sleep Staging via Parallel Frequency-Cut Attention. IEEE Transactions on Neural Systems and Rehabilitation Engineering (2023), 1974--1985.

[9]

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. (2014). arxiv: 1412.3555 [cs.NE]

[10]

Jesus Crespo Cuaresma, Jaroslava Hlouskova, Stephan Kossmeier, and Michael Obersteiner. 2004. Forecasting Electricity Spot-Prices Using Linear Univariate Time-Series Models. Applied Energy, Vol. 77 (2004), 87--106.

[11]

Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan K Mathur, Rajat Sen, and Rose Yu. 2023. Long-term Forecasting with TiDE: Time-series Dense Encoder. Transactions on Machine Learning Research (2023).

[12]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations.

[13]

Filip Elvander and Andreas Jakobsson. 2020. Defining Fundamental Frequency for Almost Harmonic Signals. IEEE TRANSACTIONS ON SIGNAL PROCESSING (2020).

Digital Library

[14]

Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. 2023. ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond. (2023).

[15]

James D Hamilton. 2020. Time series analysis. (2020).

[16]

Nicholas W. Hammond, François Birgand, Cayelan C. Carey, Bethany Bookout, Adrienne Breef-Pilz, and Madeline E. Schreiber. 2023. High-frequency Sensor Data Capture Short-term Variability In Fe and Mn Concentrations Due to Hypolimnetic Oxygenation and Seasonal Dynamics in a Drinking Water Reservoir. Water Research, Vol. 240 (2023).

[17]

Long Steven R. Wu Manli C. Shih Hsing H. Zheng Quanan Yen Nai-Chyuan Tung Chi Chao Huang Norden E. Shen Zheng and Liu Henry H. 1998. The Empirical Mode Decomposition and the Hilbert Spectrum for Nonlinear and Non-stationary Time Series Analysis. Proceedings of the Royal Society of London. Series A: mathematical, physical, and engineering sciences (1998), 903--995.

[18]

Jiawei Jiang, Chengkai Han, Wayne Xin Zhao, and Jingyuan Wang. 2023. PDFormer: Propagation Delay-aware Dynamic Long-range Transformer for Traffic Flow Prediction. (2023), 4365--4373.

[19]

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks. (2018), 95--104.

[20]

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. (2019).

[21]

Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. 2023. Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping. (2023).

[22]

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. 2022. Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting. (2022).

[23]

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. 2024. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.

[24]

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. 2022. Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting. (2022).

[25]

Liu M., Zeng A., Chen M., Xu Z., Lai Q., Ma L., and Q. Xu. 2022. SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction. (2022), 5816--5828.

[26]

Sobhan Moosavi, Mohammad Hossein Samavatian, Arnab Nandi, Srinivasan Parthasarathy, and Rajiv Ramnath. 2019. Short and Long-Term Pattern Discovery Over Large-Scale Geo-Spatiotemporal Data. (2019), 2905--2913.

[27]

Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. 2016. Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences. (2016). arxiv: 1610.09513 [cs.LG]

[28]

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. (2023).

[29]

Namuk Park and Songkuk Kim. 2022. How Do Vision Transformers Work? (2022).

[30]

John G. Proakis and Dimitris G. Manolakis. 1996. Digital Signal Processing (3rd Ed.): Principles, Algorithms, and Applications. (1996).

[31]

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. 2019. On the Spectral Bias of Neural Networks., Vol. 97 (2019), 5301--5310.

[32]

Daniel Stoller, Mi Tian, Sebastian Ewert, and Simon Dixon. 2019. Seq-U-Net: A One-Dimensional Causal U-Net for Efficient Sequence Modelling. (2019). arxiv: 1911.06393 [cs.LG]

[33]

James R. Thompson and James R. Wilson. 2016. Multifractal Detrended Fluctuation Analysis: Practical Applications to Financial Time Series. Mathematics and Computers in Simulation, Vol. 126 (2016), 63--88.

Digital Library

[34]

Yuandong Tian, Yiping Wang, Beidi Chen, and Simon Du. 2023. Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer. (2023).

[35]

Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. 2022. Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice. (2022).

[36]

Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. 2022. Learning Latent Seasonal-Trend Representations for Time Series Forecasting. (2022).

[37]

Qingsong Wen, Zhe Zhang, Yan Li, and Liang Sun. 2020. Fast RobustSTL: Efficient and Robust Seasonal-Trend Decomposition for Time Series with Complex Patterns. (2020), 2203--2213.

[38]

Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. 2023. Transformers in Time Series: A Survey. (2023).

[39]

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. 2022. CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting. (2022).

[40]

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. 2022. ETSformer: Exponential Smoothing Transformers for Time-series Forecasting. (2022).

[41]

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2023. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. (2023).

[42]

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. (2021).

[43]

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. 2020. Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks. (2020).

[44]

Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. 2021. Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention. (2021).

[45]

Zhi-Qin John Xu. 2020. Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks. Communications in Computational Physics, Vol. 28 (2020), 1746--1767.

[46]

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are Transformers Effective for Time Series Forecasting? (2023).

Digital Library

[47]

Yunhao Zhang and Junchi Yan. 2023. Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting. (2023).

[48]

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. (2021).

[49]

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. (2022), 1--12.

[50]

Yunyue Zhu and Dennis Shasha. 2002. StatStream: Statistical Monitoring of Thousands of Data Streams in Real Time. (2002), 358--369.

Cited By

Li JJiang XWang K(2024)Dynamic Partitioning of Graphs Based on Multivariate Blood Glucose Data—A Graph Neural Network Model for Diabetes PredictionElectronics10.3390/electronics1318372713:18(3727)Online publication date: 20-Sep-2024
https://doi.org/10.3390/electronics13183727
Zhou SYang LLiu XWang L(2024)Learning Short-Term Spatial–Temporal Dependency for UAV 2-D Trajectory ForecastingIEEE Sensors Journal10.1109/JSEN.2024.346651624:22(38256-38269)Online publication date: 15-Nov-2024
https://doi.org/10.1109/JSEN.2024.3466516
Wang SYe XWang WJiang XYang Q(2024)Carbon Emission Factor Multi-Time Scale Prediction with Adaptive Graph Convolution Strategy2024 14th International Conference on Information Science and Technology (ICIST)10.1109/ICIST63249.2024.10805462(626-632)Online publication date: 6-Dec-2024
https://doi.org/10.1109/ICIST63249.2024.10805462
Show More Cited By

Recommendations

IFTNet: Interpolation Frequency- and Time-Domain Network for Long-Term Time Series Forecasting
Advanced Intelligent Computing Technology and Applications
Abstract
Long-term time series forecasting has widespread applications in multiple fields. Time series possess seasonality and trend, and existing models predict time series in either the time or frequency domain, with the former generally struggling to ...
Combining seasonal ARIMA models with computational intelligence techniques for time series forecasting

Seasonal autoregressive integrated moving average (SARIMA) models form one of the most popular and widely used seasonal time series models over the past three decades. However, in several researches it has been argued that they have two basic ...
Statistical and deep learning models for reference evapotranspiration time series forecasting: A comparison of accuracy, complexity, and data efficiency
Graphical abstract

Display Omitted
Highlights
- Ten statistical, machine learning, and deep learning models forecasted ETo.
- Data from 107 California weather stations were categorized based on their length.
- Most deep learning models suffer from low data efficiency.
- Simpler ...
Abstract
Reference evapotranspiration (ETo) is an essential variable in agricultural water resources management and irrigation scheduling. An accurate and reliable forecast of ETo facilitates effective decision-making in agriculture. Although numerous ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '24: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2024

6901 pages

ISBN:9798400704901

DOI:10.1145/3637528

General Chairs:
Ricardo Baeza-Yates
Northeastern University, USA
,
Francesco Bonchi
CENTAI / Eurecat, Italy

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NICT
JST-CREST
JSPS
JST-AIP
JST-RISTEX

Conference

KDD '24

Sponsor:

KDD '24: The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 25 - 29, 2024

Barcelona, Spain

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
530
Total Downloads

Downloads (Last 12 months)530
Downloads (Last 6 weeks)91

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li JJiang XWang K(2024)Dynamic Partitioning of Graphs Based on Multivariate Blood Glucose Data—A Graph Neural Network Model for Diabetes PredictionElectronics10.3390/electronics1318372713:18(3727)Online publication date: 20-Sep-2024
https://doi.org/10.3390/electronics13183727
Zhou SYang LLiu XWang L(2024)Learning Short-Term Spatial–Temporal Dependency for UAV 2-D Trajectory ForecastingIEEE Sensors Journal10.1109/JSEN.2024.346651624:22(38256-38269)Online publication date: 15-Nov-2024
https://doi.org/10.1109/JSEN.2024.3466516
Wang SYe XWang WJiang XYang Q(2024)Carbon Emission Factor Multi-Time Scale Prediction with Adaptive Graph Convolution Strategy2024 14th International Conference on Information Science and Technology (ICIST)10.1109/ICIST63249.2024.10805462(626-632)Online publication date: 6-Dec-2024
https://doi.org/10.1109/ICIST63249.2024.10805462
Zhu JLiu DChen HLiu JTao Z(2024)DTSFormer: Decoupled temporal-spatial diffusion transformer for enhanced long-term time series forecastingKnowledge-Based Systems10.1016/j.knosys.2024.112828(112828)Online publication date: Dec-2024
https://doi.org/10.1016/j.knosys.2024.112828

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten