Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

MIANet: : Multi-level temporal information aggregation in mixed-periodicity time series forecasting tasks

Published: 01 May 2023 Publication History

Abstract

Regular human activities generate a large number of time series with mixed periodicity that can reflect human behavior patterns and the societal working mechanism. When forecasting these time series, nonlinear neural networks often encounter some limitations, such as utilizing mixed-periodic patterns, balancing multi-level information, incorporating future vision, forecasting delays and scale insensitivity, which affect the forecasting accuracy. To address these problems, we propose the Multi-level Information Aggregation Network (MIANet), a novel neural network with four key characteristics: (i) a novel folded recurrent structure that dynamically updates the local and mini-local information at a global range in a compact manner; (ii) a new recurrent unit called Folded Convolution Aggregation Temporal Memory (FCATM) that extracts and aggregates neighbor-trends in local and mini-local data; (iii) a fusing decoder structure that promotes the sharing of forward–backward future information and adaptively adjusts relationships among adjacent points; and (iv) a new Skip-Autoregressive (SAR) linear strategy that addresses scale sensitivity issues. The SAR can be embedded as a plug-and-play component into other deep learning (DL) models. Compared with other baseline methods, MIANet obtains statistically significant improvements on six real-world datasets, as demonstrated by conducting two-sample t-tests, indicating that the MIANet can be applied to various predictive scenarios, such as road occupancy, electricity consumption, pedestrian flow and urban noise.

Graphical abstract

Display Omitted

Highlights

MIANet thoroughly integrates multi-level temporal information in a compact manner.
We design a folded recurrent unit, circularly updating local and mini-local features.
We develop a novel decoder structure that strengthens the interplay of future vision.
We devise an SAR linear strategy that can practically solve the scale insensitivity.
Compared to eight other methods, MIANet achieves the best results on six datasets.

References

[1]
Abdi J., Moshiri B., Abdulhai B., Sedigh A.K., Forecasting of short-term traffic-flow based on improved neurofuzzy models via emotional temporal difference learning algorithm, Eng. Appl. Artif. Intell. 25 (5) (2012) 1022–1042. URL: https://doi.org/10.1016/j.engappai.2011.09.011.
[2]
Abdollahi J., Irani A.J., Nouri-Moghaddam B., Modeling and forecasting spread of COVID-19 epidemic in Iran until Sep 22, 2021, based on deep learning, 2021, CoRR abs/2103.08178. URL: https://arxiv.org/abs/2103.08178, arXiv:2103.08178.
[3]
Bahdanau D., Cho K., Bengio Y., Neural machine translation by jointly learning to align and translate, in: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015, URL: http://arxiv.org/abs/1409.0473.
[4]
Bai S., Kolter J.Z., Koltun V., An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, 2018, CoRR abs/1803.01271. URL: http://arxiv.org/abs/1803.01271.
[5]
Beltagy I., Peters M.E., Cohan A., Longformer: The long-document transformer, 2020, CoRR abs/2004.05150. URL: https://arxiv.org/abs/2004.05150.
[6]
Box G.E., Jenkins G.M., Reinsel G.C., Ljung G.M., Time Series Analysis: Forecasting and Control, John Wiley & Sons, 2015.
[7]
Brownlee J., Introduction to Time Series Forecasting with Python: How to Prepare Data and Develop Models to Predict the Future, Machine Learning Mastery, 2017, URL: https://machinelearningmastery.com/introduction-to-time-series-forecasting-with-python/.
[8]
Cao L., Tay F.E.H., Support vector machine with adaptive parameters in financial time series forecasting, IEEE Trans. Neural Netw. 14 (6) (2003) 1506–1518. URL: https://doi.org/10.1109/TNN.2003.820556.
[9]
Cao D., Wang Y., Duan J., Zhang C., Zhu X., Huang C., Tong Y., Xu B., Bai J., Tong J., Zhang Q., Spectral temporal graph neural network for multivariate time-series forecasting, 2021, CoRR abs/2103.07719. URL: https://arxiv.org/abs/2103.07719.
[10]
Carion N., Massa F., Synnaeve G., Usunier N., Kirillov A., Zagoruyko S., End-to-end object detection with transformers, in: European Conference on Computer Vision, Springer, 2020, pp. 213–229. URL: https://doi.org/10.1007/978-3-030-58452-8_13.
[11]
Chang Y.-Y., Sun F.-Y., Wu Y.-H., Lin S.-D., A memory-network based solution for multivariate time-series forecasting, 2018, CoRR abs/1809.02105. URL: http://arxiv.org/abs/1809.02105.
[12]
Chen S., Wang X., Harris C.J., NARX-based nonlinear system identification using orthogonal least squares basis hunting, IEEE Trans. Control Syst. Technol. 16 (1) (2008) 78–84. URL: https://doi.org/10.1109/TCST.2007.899728.
[13]
Cheng J., Huang K., Zheng Z., Towards better forecasting by fusing near and distant future visions, in: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, AAAI Press, 2020, pp. 3593–3600. URL: https://aaai.org/ojs/index.php/AAAI/article/view/5766.
[14]
Cho K., van Merrienboer B., Gülçehre Ç., Bahdanau D., Bougares F., Schwenk H., Bengio Y., Learning phrase representations using RNN encoder-decoder for statistical machine translation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, a Meeting of SIGDAT, a Special Interest Group of the ACL, ACL, 2014, pp. 1724–1734. URL: https://doi.org/10.3115/v1/d14-1179.
[15]
Connor J.T., Martin R.D., Atlas L.E., Recurrent neural networks and robust time series prediction, IEEE Trans. Neural Netw. 5 (2) (1994) 240–254. URL: https://doi.org/10.1109/72.279188.
[16]
Cressie N., Whitford H., How to use the two sample t-test, Biom. J. 28 (2) (1986) 131–148. URL: https://doi.org/10.1002/bimj.4710280202.
[17]
Fan W., Zheng S., Yi X., Cao W., Fu Y., Bian J., Liu T., DEPTS: deep expansion learning for periodic time series forecasting, in: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022, URL: https://openreview.net/forum?id=AJAR-JgNw__.
[18]
Garasia K., Stock Prediction : GRU Model Predicting Same Given Values Instead of Future Stock Price, Website, 2018, https://stackoverflow.com/q/52778922.
[19]
Girard A., Rasmussen C., Candela J.Q., Murray-Smith R., Gaussian process priors with uncertain inputs application to multiple-step ahead time series forecasting, in: Becker S., Thrun S., Obermayer K. (Eds.), Advances in Neural Information Processing Systems, Vol. 15, MIT Press, 2002, URL: https://proceedings.neurips.cc/paper/2002/file/f3ac63c91272f19ce97c7397825cc15f-Paper.pdf.
[20]
Godahewa R., Bergmeir C., Webb G.I., Montero-Manso P., A strong baseline for weekly time series forecasting, 2020, CoRR abs/2010.08158. URL: https://arxiv.org/abs/2010.08158, arXiv:2010.08158.
[21]
Graves A., Schmidhuber J., Framewise phoneme classification with bidirectional LSTM and other neural network architectures, Neural Netw. 18 (5–6) (2005) 602–610. URL: https://doi.org/10.1016/j.neunet.2005.06.042.
[22]
He K., Zhang X., Ren S., Sun J., Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016, pp. 770–778,.
[23]
Hewamalage H., Bergmeir C., Bandara K., Recurrent neural networks for time series forecasting: Current status and future directions, 2019, CoRR abs/1909.00590. URL: http://arxiv.org/abs/1909.00590, arXiv:1909.00590.
[24]
Hochreiter S., Schmidhuber J., Long short-term memory, Neural Comput. 9 (8) (1997) 1735–1780. URL: https://doi.org/10.1162/neco.1997.9.8.1735.
[25]
Hu J., Shen L., Sun G., Squeeze-and-excitation networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2018, URL: https://arxiv.org/abs/1709.01507v4.
[26]
Huang S., Wang D., Wu X., Tang A., DSANet: Dual self-attention network for multivariate time series forecasting, in: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM ’19, Association for Computing Machinery, New York, NY, USA, 2019, pp. 2129–2132. URL: https://doi.org/10.1145/3357384.3358132.
[27]
Hyndman R.J., Athanasopoulos G., Forecasting: Principles and Practice, OTexts, 2018.
[28]
Kingma D.P., Ba J., Adam: A method for stochastic optimization, 2014, arXiv preprint arXiv:1412.6980. URL: http://arxiv.org/abs/1412.6980.
[29]
Lai G., Chang W.-C., Yang Y., Liu H., Modeling long-and short-term temporal patterns with deep neural networks, in: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018, pp. 95–104. URL: https://arxiv.org/abs/1703.07015.
[30]
Lara-Benítez P., Carranza-García M., Riquelme J.C., An experimental review on deep learning architectures for time series forecasting, Int. J. Neural Syst. 31 (3) (2021) 2130001:1–2130001:28. URL: https://doi.org/10.1142/S0129065721300011.
[31]
LeCun Y., Bengio Y., et al., Convolutional networks for images, speech, and time series, in: The Handbook of Brain Theory and Neural Networks, Vol. 3361, 1995, p. 1995. URL: http://yann.lecun.org/exdb/publis/psgz/lecun-bengio-95a.ps.gz.
[32]
Li S., Jin X., Xuan Y., Zhou X., Chen W., Wang Y., Yan X., Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, in: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019, pp. 5244–5254. URL: https://proceedings.neurips.cc/paper/2019/hash/6775a0635c302542da2c32aa19d86be0-Abstract.html.
[33]
Liberman N., Trope Y., The role of feasibility and desirability considerations in near and distant future decisions: A test of temporal construal theory, J. Personal. Soc. Psychol. 75 (1) (1998) 5. URL: https://psycnet.apa.org/record/1998-04530-001.
[34]
Lovrić M., Milanović M., Stamenković M., Algoritmic methods for segmentation of time series: An overview, J. Contemp. Econ. Bus. Issues 1 (1) (2014) 31–53. URL: https://www.econstor.eu/handle/10419/147468.
[35]
Luong T., Pham H., Manning C.D., Effective approaches to attention-based neural machine translation, in: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, The Association for Computational Linguistics, 2015, pp. 1412–1421. URL: https://doi.org/10.18653/v1/d15-1166.
[36]
Morales-Hernández A., Nápoles G., Jastrzebska A., Salgueiro Y., Vanhoof K., Online learning of windmill time series using long short-term cognitive networks, Expert Syst. Appl. 205 (2022),.
[37]
Nguyen H.-P., Baraldi P., Zio E., Ensemble empirical mode decomposition and long short-term memory neural network for multi-step predictions of time series signals in nuclear power plants, Appl. Energy 283 (2021) URL: https://doi.org/10.1016/j.apenergy.2020.116346.
[38]
Oreshkin B.N., Carpov D., Chapados N., Bengio Y., N-BEATS: Neural basis expansion analysis for interpretable time series forecasting, in: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net, 2020, URL: https://openreview.net/forum?id=r1ecqn4YwB.
[39]
OverLordGoldDragon B.N., See RNN, GitHub, 2019,. Note: https://github.com/OverLordGoldDragon/see-rnn/.
[40]
Prechelt L., Early stopping-but when?, in: Neural Networks: Tricks of the Trade, Springer, 1998, pp. 55–69. URL: https://doi.org/10.1007/978-3-642-35289-8_5.
[41]
Qin Y., Song D., Chen H., Cheng W., Jiang G., Cottrell G.W., A dual-stage attention-based recurrent neural network for time series prediction, in: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, ijcai.org, 2017, pp. 2627–2633. URL: https://doi.org/10.24963/ijcai.2017/366.
[42]
Quesada D., Valverde G., Larranaga P., Bielza C., Long-term forecasting of multivariate time series in industrial furnaces with dynamic Gaussian Bayesian networks, Eng. Appl. Artif. Intell. 103 (2021) URL: https://doi.org/10.1016/j.engappai.2021.104301.
[43]
Ribeiro G.T., Mariani V.C., dos Santos Coelho L., Enhanced ensemble structures using wavelet neural networks applied to short-term load forecasting, Eng. Appl. Artif. Intell. 82 (2019) 272–281. URL: https://doi.org/10.1016/j.engappai.2019.03.012.
[44]
Rick R., Berton L., Energy forecasting model based on CNN-LSTM-AE for many time series with unequal lengths, Eng. Appl. Artif. Intell. 113 (2022) URL: https://doi.org/10.1016/j.engappai.2022.104998.
[45]
Rodrigues Moreno S., Gomes da Silva R., Cocco Mariani V., dos Santos Coelho L., Multi-step wind speed forecasting based on hybrid multi-stage decomposition model and long short-term memory neural network, Energy Convers. Manage. 213 (2020),.
[46]
Salinas D., Flunkert V., Gasthaus J., Januschowski T., DeepAR: Probabilistic forecasting with autoregressive recurrent networks, Int. J. Forecast. 36 (3) (2020) 1181–1191. URL: https://doi.org/10.1016/j.ijforecast.2019.07.001.
[47]
Salloom T., Kaynak O., Yu X., He W., Proportional integral derivative booster for neural networks-based time-series prediction: Case of water demand prediction, Eng. Appl. Artif. Intell. 108 (2022) URL: https://doi.org/10.1016/j.engappai.2021.104570.
[48]
Sen R., Yu H., Dhillon I.S., Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting, in: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019, pp. 4838–4847. URL: https://proceedings.neurips.cc/paper/2019/hash/3a0844cee4fcf57de0c71e9ad3035478-Abstract.html.
[49]
Shi X., Chen Z., Wang H., Yeung D., Wong W., Woo W., Convolutional LSTM network: A machine learning approach for precipitation nowcasting, in: Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, 2015, pp. 802–810. URL: https://proceedings.neurips.cc/paper/2015/hash/07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html.
[50]
Shih S., Sun F., Lee H., Temporal pattern attention for multivariate time series forecasting, Mach. Learn. 108 (8–9) (2019) 1421–1441. URL: https://doi.org/10.1007/s10994-019-05815-0.
[51]
Siami-Namini S., Tavakoli N., Namin A.S., The performance of LSTM and BiLSTM in forecasting time series, in: 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, December 9-12, 2019, IEEE, 2019, pp. 3285–3292. URL: https://doi.org/10.1109/BigData47090.2019.9005997.
[52]
Sreeram T.P., How to Handle Shift in Forecasted Value, Website, 2018, https://stackoverflow.com/q/52252442.
[53]
Srivastava R.K., Greff K., Schmidhuber J., Highway networks, 2015, CoRR abs/1505.00387. URL: http://arxiv.org/abs/1505.00387.
[54]
Stefenon S.F., Ribeiro M.H.D.M., Nied A., Mariani V.C., dos Santos Coelho L., da Rocha D.F.M., Grebogi R.B., de Barros Ruano A.E., Wavelet group method of data handling for fault prediction in electrical power insulators, Int. J. Electr. Power Energy Syst. 123 (2020) URL: https://doi.org/10.1016/j.ijepes.2020.106269.
[55]
Sukhbaatar S., Weston J., Fergus R., et al., End-to-end memory networks, in: Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, 2015, pp. 2440–2448. URL: https://proceedings.neurips.cc/paper/2015/hash/8fb21ee7a2207526da55a679f0332de2-Abstract.html.
[56]
Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser L., Polosukhin I., Attention is all you need, in: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017, pp. 5998–6008. URL: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
[57]
Wen Q., Zhou T., Zhang C., Chen W., Ma Z., Yan J., Sun L., Transformers in time series: A survey, 2022, CoRR abs/2202.07125. URL: https://arxiv.org/abs/2202.07125.
[58]
Yang Y., Lu J., Foreformer: an enhanced transformer-based framework for multivariate time series forecasting, Appl. Intell. (2022) 1–20. URL: https://doi.org/10.1007/s10489-022-04100-3.
[59]
Yazici I., Beyca O.F., Delen D., Deep-learning-based short-term electricity load forecasting: A real case application, Eng. Appl. Artif. Intell. 109 (2022) URL: https://doi.org/10.1016/j.engappai.2021.104645.
[60]
Zhang G.P., Time series forecasting using a hybrid ARIMA and neural network model, Neurocomputing 50 (2003) 159–175. URL: https://doi.org/10.1016/S0925-2312(01)00702-0.
[61]
Zhang W., Lin Z., Liu X., Short-term offshore wind power forecasting-A hybrid model based on Discrete Wavelet Transform (DWT), Seasonal Autoregressive Integrated Moving Average (SARIMA), and deep-learning-based Long Short-Term Memory (LSTM), Renew. Energy 185 (2022) 611–628. URL: https://doi.org/10.1016/j.renene.2021.12.100.
[62]
Zhang X., Zhou X., Lin M., Sun J., Shufflenet: An extremely efficient convolutional neural network for mobile devices, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6848–6856. URL: http://arxiv.org/abs/1707.01083v2.
[63]
Zhou H., Zhang S., Peng J., Zhang S., Li J., Xiong H., Zhang W., Informer: Beyond efficient transformer for long sequence time-series forecasting, in: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Event, February 2-9, 2021, AAAI Press, 2021, pp. 11106–11115. URL: https://ojs.aaai.org/index.php/AAAI/article/view/17325.

Cited By

View all
  • (2024)SCCGCN: A Skip-Connection Coupled Graph Convolutional Network with Dynamic Fusion Attention Mechanism for Traffic Flow PredictionProceedings of the 2024 4th International Conference on Internet of Things and Machine Learning10.1145/3697467.3697586(19-23)Online publication date: 9-Aug-2024
  • (2024)Intersecting reinforcement learning and deep factor methods for optimizing locality and globality in forecastingEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108082133:PBOnline publication date: 1-Jul-2024

Index Terms

  1. MIANet: Multi-level temporal information aggregation in mixed-periodicity time series forecasting tasks
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Please enable JavaScript to view thecomments powered by Disqus.

        Information & Contributors

        Information

        Published In

        cover image Engineering Applications of Artificial Intelligence
        Engineering Applications of Artificial Intelligence  Volume 121, Issue C
        May 2023
        1624 pages

        Publisher

        Pergamon Press, Inc.

        United States

        Publication History

        Published: 01 May 2023

        Author Tags

        1. Deep learning
        2. Data mining
        3. Time series forecasting
        4. Sequence modeling
        5. Encoder–decoder network
        6. Time series with mixed periodicity

        Qualifiers

        • Research-article

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)0
        • Downloads (Last 6 weeks)0
        Reflects downloads up to 19 Dec 2024

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)SCCGCN: A Skip-Connection Coupled Graph Convolutional Network with Dynamic Fusion Attention Mechanism for Traffic Flow PredictionProceedings of the 2024 4th International Conference on Internet of Things and Machine Learning10.1145/3697467.3697586(19-23)Online publication date: 9-Aug-2024
        • (2024)Intersecting reinforcement learning and deep factor methods for optimizing locality and globality in forecastingEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108082133:PBOnline publication date: 1-Jul-2024

        View Options

        View options

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media