Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

AutoCTS++: zero-shot joint neural architecture and hyperparameter search for correlated time series forecasting

  • Regular Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

Sensors in cyber-physical systems often capture interconnected processes and thus emit correlated time series (CTS), the forecasting of which enables important applications. Recent deep learning based forecasting methods show strong capabilities at capturing both the temporal dynamics of time series and the spatial correlations among time series, thus achieving impressive accuracy. In particular, automated CTS forecasting, where a deep learning architecture is configured automatically, enables forecasting accuracy that surpasses what has been achieved by manual approaches. However, automated CTS forecasting remains in its infancy, as existing proposals are only able to find optimal architectures for predefined hyperparameters and for specific datasets and forecasting settings (e.g., short vs. long term forecasting). These limitations hinder real-world industrial application, where forecasting faces diverse datasets and forecasting settings. We propose AutoCTS++, a zero-shot, joint search framework, to efficiently configure effective CTS forecasting models (including both neural architectures and hyperparameters), even when facing unseen datasets and foreacsting settings. Specifically, we propose an architecture-hyperparameter joint search space by encoding candidate architecture and accompanying hyperparameters into a graph representation. We then introduce a zero-shot Task-aware Architecture-Hyperparameter Comparator (T-AHC) to rank architecture-hyperparameter pairs according to different tasks (i.e., datasets and forecasting settings). We propose zero-shot means to train T-AHC, enabling it to rank architecture-hyperparameter pairs given unseen datasets and forecasting settings. A final forecasting model is then selected from the top-ranked pairs. Extensive experiments involving multiple benchmark datasets and forecasting settings demonstrate that AutoCTS++ is able to efficiently devise forecasting models for unseen datasets and forecasting settings that are capable of outperforming existing manually designed and automated models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Algorithm 2
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C.C., Soatto, S., Perona, P.: Task2Vec: task embedding for meta-learning. In: International Conference on Computer Vision, pp. 6430–6439 (2019)

  2. Ansari, A.F., Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S.S., Arango, S.P., Kapoor, S., et al.: Chronos: learning the language of time series. arXiv preprint arXiv:2403.07815 (2024)

  3. Bai, L., Yao, L., Li, C., Wang, X., Wang, C.: Adaptive graph convolutional recurrent network for traffic forecasting. In: NeurIPS, vol. 33, pp. 17804–17815 (2020)

  4. Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: International Conference on Machine Learning, pp. 41–48 (2009)

  5. Campos, D., Kieu, T., Guo, C., Huang, F., Zheng, K., Yang, B., Jensen, C.S.: Unsupervised time series outlier detection with diversity-driven convolutional ensembles. Proc. VLDB Endow. 15(3), 611–623 (2022)

    Article  Google Scholar 

  6. Campos, D., Yang, B., Kieu, T., Zhang, M., Guo, C., Jensen, C.S.: QCore: Data-efficient, on-device continual calibration for quantized models. In: Proceedings of the VLDB Endowment (2024)

  7. Campos, D., Zhang, M., Yang, B., Kieu, T., Guo, C., Jensen, C.S.: LightTS: lightweight time series classification with adaptive ensemble distillation. Proc. ACM Manag. Data 1(2), 171:1-171:27 (2023)

    Article  Google Scholar 

  8. Cao, K., You, J., Liu, J., Leskovec, J.: Autotransfer: AutoML with knowledge transfer—an application to graph neural networks. In: International Conference on Learning Representations (2023)

  9. Chen, Y., Guo, Y., Chen, Q., Li, M., Zeng, W., Wang, Y., Tan, M.: Contrastive neural architecture search with neural architecture comparators. In: Conference on Computer Vision and Pattern Recognition pp. 9502–9511 (2021)

  10. Cheng, Y., Chen, P., Guo, C., Zhao, K., Wen, Q., Yang, B., Jensen, C.S.: Weakly guided adaptation for robust time series forecasting. In: Proceedings of the VLDB Endowment (2024)

  11. Cirstea, R., Guo, C., Yang, B., Kieu, T., Dong, X., Pan, S.: Triangular, variable-specific attentions for long sequence multivariate time series forecasting. In: IJCAI, pp. 1994–2001 (2022)

  12. Cirstea, R., Yang, B., Guo, C., Kieu, T., Pan, S.: Towards spatio-temporal aware traffic time series forecasting. In: ICDE, pp. 2900–2913 (2022)

  13. Cirstea, R.G., Kieu, T., Guo, C., Yang, B., Pan, S.J.: Enhancenet: plugin neural networks for enhancing correlated time series forecasting. In: ICDE, pp. 1739–1750 (2021)

  14. Cirstea, R.G., Yang, B., Guo, C.: Graph attention recurrent neural networks for correlated time series forecasting. In: MileTS19@KDD (2019)

  15. Dai, X., Wan, A., Zhang, P., Wu, B., He, Z., Wei, Z., Chen, K., Tian, Y., Yu, M., Vajda, P., et al.: Fbnetv3: joint architecture-recipe search using predictor pretraining. In: Conference on Computer Vision and Pattern Recognition, pp. 16276–16285 (2021)

  16. Das, A., Kong, W., Sen, R., Zhou, Y.: A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688 (2023)

  17. Deng, J., Chen, X., Jiang, R., Yin, D., Yang, Y., Song, X., Tsang, I.W.: Disentangling structured components: Towards adaptive, interpretable and scalable time series forecasting. IEEE Transactions on Knowledge and Data Engineering (2024)

  18. Dong, X., Tan, M., Yu, A.W., Peng, D., Gabrys, B., Le, Q.V.: Autohas: Efficient hyperparameter and architecture search. arXiv preprint arXiv:2006.03656 (2020)

  19. Dudziak, L., Chau, T., Abdelfattah, M., Lee, R., Kim, H., Lane, N.: Prediction-based NAS using GCNs. In: BRP-NAS: Advances in Neural Information Processing Systems, vol. 33, pp. 10480–10490 (2020)

  20. El, O.B., Milo, T., Somech, A.: Automatically generating data exploration sessions using deep reinforcement learning, pp. 1527–1537 (2020)

  21. Guo, C., Xu, R., Yang, B., Yuan, Y., Kieu, T., Zhao, Y., Jensen, C.S.: Efficient stochastic routing in path-centric uncertain road networks. In: Proceedings of the VLDB Endowment (2024)

  22. Guo, C., Yang, B., Hu, J., Jensen, C.S., Chen, L.: Context-aware, preference-based vehicle routing. VLDB J. 29(5), 1149–1170 (2020)

    Article  Google Scholar 

  23. Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., Sun, J.: Single path one-shot neural architecture search with uniform sampling. In: European Conference on Computer Vision, pp. 544–560 (2020)

  24. Jiang, J., Han, C., Zhao, W.X., Wang, J.: Pdformer: Propagation delay-aware dynamic long-range transformer for traffic flow prediction. arXiv preprint arXiv:2301.07945 (2023)

  25. Jin, G., Liang, Y., Fang, Y., Shao, Z., Huang, J., Zhang, J., Zheng, Y.: Spatio-temporal graph neural networks for predictive learning in urban computing: A survey. IEEE Trans. Knowl. Data Eng. (2023)

  26. Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J.Y., Shi, X., Chen, P.Y., Liang, Y., Li, Y.F., Pan, S., et al.: Time-llm: time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 (2023)

  27. Jin, M., Wen, Q., Liang, Y., Zhang, C., Xue, S., Wang, X., Zhang, J., Wang, Y., Chen, H., Li, X., et al.: Large models for time series and spatio-temporal data: a survey and outlook. arXiv preprint arXiv:2310.10196 (2023)

  28. Jin, M., Zheng, Y., Li, Y., Chen, S., Yang, B., Pan, S.: Multivariate time series forecasting with dynamic graph neural odes. IEEE Trans. Knowl. Data Eng. 35(9), 9168–9180 (2023)

    Article  Google Scholar 

  29. Jomaa, H.S., Schmidt-Thieme, L., Grabocka, J.: Dataset2vec: learning dataset meta-features. Data Min. Knowl. Discov. 35, 964–985 (2021)

    Article  MathSciNet  Google Scholar 

  30. Karakida, R., Akaho, S., Amari, S.i.: Universal statistics of fisher information in deep neural networks: Mean field approach. In: International Conference on Artificial Intelligence and Statistics, pp. 1032–1041 (2019)

  31. Ke, S., Pan, Z., He, T., Liang, Y., Zhang, J., Zheng, Y.: Autostg+: an automatic framework to discover the optimal network for spatio-temporal graph prediction. Artif. Intell. 318, 103899 (2023)

    Article  MathSciNet  Google Scholar 

  32. Kieu, T., Yang, B., Guo, C., Cirstea, R., Zhao, Y., Song, Y., Jensen, C.S.: Anomaly detection in time series with robust variational quasi-recurrent autoencoders. In: ICDE, pp. 1342–1354 (2022)

  33. Kieu, T., Yang, B., Guo, C., Jensen, C.S., Zhao, Y., Huang, F., Zheng, K.: Robust and explainable autoencoders for unsupervised time series outlier detection. In: ICDE, pp. 3038–3050 (2022)

  34. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  35. Lai, G., Chang, W.C., Yang, Y., Liu, H.: Modeling long-and short-term temporal patterns with deep neural networks. In: SIGIR, pp. 95–104 (2018)

  36. Le, C.P., Soltani, M., Dong, J., Tarokh, V.: Fisher task distance and its application in neural architecture search. IEEE Access 10, 47235–47249 (2022)

    Article  Google Scholar 

  37. Lee, H., Hyung, E., Hwang, S.J.: Rapid neural architecture search by learning to generate graphs from datasets. arXiv preprint arXiv:2107.00860 (2021)

  38. Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., Teh, Y.W.: Set transformer: a framework for attention-based permutation-invariant neural networks. In: International Conference on Machine Learning, pp. 3744–3753 (2019)

  39. Li, T., Zhang, J., Bao, K., Liang, Y., Li, Y., Zheng, Y.: Autost: efficient neural architecture search for spatio-temporal prediction. In: SIGKDD, pp. 794–802 (2020)

  40. Li, X., Li, Z., Xie, H., Li, Q.: Merging statistical feature via adaptive gate for improved text classification. In: AAAI, vol. 35, pp. 13288–13296 (2021)

  41. Li, Y., Chen, Z., Zha, D., Zhou, K., Jin, H., Chen, H., Hu, X.: AutoOD: Neural architecture search for outlier detection. In: ICDE, pp. 2117–2122. IEEE (2021)

  42. Li, Y., Shen, Y., Zhang, W., Jiang, J., Li, Y., Ding, B., Zhou, J., Yang, Z., Wu, W., Zhang, C., Cui, B.: VolcanoML: speeding up end-to-end autoML via scalable search space decomposition. Proc. VLDB Endow. 14(11), 2167–2176 (2021)

    Article  Google Scholar 

  43. Li, Y., Yu, R., Shahabi, C., Liu, Y.: Diffusion convolutional recurrent neural network: data-driven traffic forecasting. In: ICLR (2018)

  44. Liu, H., Simonyan, K., Yang, Y.: Darts: differentiable architecture search. In: ICLR (2018)

  45. Liu, Y., Zhang, H., Li, C., Huang, X., Wang, J., Long, M.: Timer: Transformers for time series analysis at scale. arXiv preprint arXiv:2402.02368 (2024)

  46. Miao, H., Zhao, Y., Guo, C., Yang, B., Kai, Z., Huang, F., Xie, J., Jensen, C.S.: A unified replay-based continuous learning framework for spatio-temporal prediction on streaming data. In: ICDE (2024)

  47. Nguyen, C., Hassner, T., Seeger, M., Archambeau, C.: Leep: A new measure to evaluate transferability of learned representations. In: International Conference on Machine Learning, pp. 7294–7305. PMLR (2020)

  48. Pan, Z., Ke, S., Yang, X., Liang, Y., Yu, Y., Zhang, J., Zheng, Y.: AutoSTG: neural architecture search for predictions of spatio-temporal graphs. In: WWW, pp. 1846–1855 (2021)

  49. Pedersen, S.A., Yang, B., Jensen, C.S.: Anytime stochastic routing with hybrid learning. Proc. VLDB Endow. 13(9), 1555–1567 (2020)

    Article  Google Scholar 

  50. Pedersen, S.A., Yang, B., Jensen, C.S.: Fast stochastic routing under time-varying uncertainty. VLDB J. 29(4), 819–839 (2020)

    Article  Google Scholar 

  51. Qiu, X., Hu, J., Zhou, L., Wu, X., Du, J., Zhang, B., Guo, C., Zhou, A., Jensen, C.S., Sheng, Z., Yang, B.: Tfb: towards comprehensive and fair benchmarking of time series forecasting methods. Proc. VLDB Endow. 17, 2363–2377 (2024)

    Article  Google Scholar 

  52. Rajkumar, R., Lee, I., Sha, L., Stankovic, J.: Cyber-physical systems: the next computing revolution. In: Design Automation Conference, pp. 731–736 (2010)

  53. Shah, S.Y., Patel, D., Vu, L., Dang, X., Chen, B., Kirchner, P., Samulowitz, H., Wood, D., Bramble, G., Gifford, W.M., Ganapavarapu, G., Vaculín, R., Zerfos, P.: AutoAI-TS: Autoai for time series forecasting. In: SIGMOD, pp. 2584–2596 (2021)

  54. Shala, G., Elsken, T., Hutter, F., Grabocka, J.: Transfer NAS with meta-learned bayesian surrogates. In: International Conference on Learning Representations (2023)

  55. Shih, S.Y., Sun, F.K., Lee, Hy.: Temporal pattern attention for multivariate time series forecasting. Mach. Learn. 108(8), 1421–1441 (2019)

    Article  MathSciNet  Google Scholar 

  56. Song, C., Lin, Y., Guo, S., Wan, H.: Spatial-temporal synchronous graph convolutional networks: a new framework for spatial-temporal network data forecasting. In: AAAI, vol. 34, pp. 914–921 (2020)

  57. Tran, A.T., Nguyen, C.V., Hassner, T.: Transferability and hardness of supervised classification tasks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1395–1405 (2019)

  58. Wang, J., Wang, K.C., Rudzicz, F., Brudno, M.: Grad2task: improved few-shot text classification using gradients for task representation. Adv. Neural Inf. Process. Syst. 34, 6542–6554 (2021)

    Google Scholar 

  59. Wang, S., Miao, H., Chen, H., Huang, Z.: Multi-task adversarial spatial-temporal networks for crowd flow prediction. In: International Conference on Information & Knowledge Management, pp. 1555–1564 (2020)

  60. Wang, S., Zhang, M., Miao, H., Peng, Z., Yu, P.S.: Multivariate correlation-aware spatio-temporal graph convolutional networks for multi-scale traffic prediction. ACM Trans. Intell. Syst. Technol. (TIST) 13(3), 1–22 (2022)

    Article  Google Scholar 

  61. Wei, Y., Zhao, P., Huang, J.: Meta-learning hyperparameter performance prediction with neural processes. In: International Conference on Machine Learning, pp. 11058–11067 (2021)

  62. Wong, C., Houlsby, N., Lu, Y., Gesmundo, A.: Transfer learning with neural AutoML. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

  63. Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., Sahoo, D.: Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592 (2024)

  64. Wu, H., Xu, J., Wang, J., Long, M.: Autoformer: decomposition transformers with auto-correlation for long-term series forecasting. Adv. Neural Inf. Process. Syst. 34, 22419–22430 (2021)

    Google Scholar 

  65. Wu, X., Zhang, D., Guo, C., He, C., Yang, B., Jensen, C.S.: AutoCTS: automated correlated time series forecasting. Proc. VLDB Endow. 15(4), 971–983 (2022)

    Article  Google Scholar 

  66. Wu, X., Zhang, D., Zhang, M., Guo, C., Yang, B., Jensen, C.S.: AutoCTS+: joint neural architecture and hyperparameter search for correlated time series forecasting. Proc. ACM Manag. Data 1(1), 1–26 (2023)

    Google Scholar 

  67. Wu, Z., Pan, S., Long, G., Jiang, J., Chang, X., Zhang, C.: Connecting the dots: Multivariate time series forecasting with graph neural networks. In: SIGKDD, pp. 753–763 (2020)

  68. Wu, Z., Pan, S., Long, G., Jiang, J., Zhang, C.: Graph wavenet for deep spatial-temporal graph modeling. In: IJCAI, pp. 1907–1913 (2019)

  69. Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018)

  70. Yang, S.B., Guo, C., Hu, J., Tang, J., Yang, B.: Unsupervised path representation learning with curriculum negative sampling. In: IJCAI, pp. 3286–3292 (2021)

  71. Yang, S.B., Guo, C., Hu, J., Yang, B., Tang, J., Jensen, C.S.: Weakly-supervised temporal path representation learning with contrastive curriculum learning. In: ICDE, pp. 2873–2885 (2022)

  72. Yang, S.B., Guo, C., Yang, B.: Context-aware path ranking in road networks. IEEE Trans. Knowl. Data Eng. 34(7), 3153–3168 (2022)

    Google Scholar 

  73. Ye, J., Sun, L., Du, B., Fu, Y., Xiong, H.: Coupled layer-wise graph convolution for transportation demand prediction. In: AAAI, vol. 35, pp. 4617–4625 (2021)

  74. You, K., Liu, Y., Wang, J., Long, M.: Logme: Practical assessment of pre-trained models for transfer learning. In: International Conference on Machine Learning, pp. 12133–12143. PMLR (2021)

  75. Yu, B., Yin, H., Zhu, Z.: Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. In: IJCAI, pp. 3634–3640 (2018)

  76. Yu, H., Hu, J., Zhou, X., Guo, C., Yang, B., Li, Q.: CGF: A category guidance based PM2.5 sequence forecasting training framework. IEEE Trans. Knowl. Data Eng. 35(10), 125–139 (2023)

    Article  Google Scholar 

  77. Yue, Z., Wang, Y., Duan, J., Yang, T., Huang, C., Tong, Y., Xu, B.: TS2Vec: towards universal representation of time series. In: AAAI, vol. 36, pp. 8980–8987 (2022)

  78. Zamir, A.R., Sax, A., Shen, W., Guibas, L.J., Malik, J., Savarese, S.: Taskonomy: Disentangling task transfer learning. In: Conference on Computer Vision and Pattern Recognition, pp. 3712–3722 (2018)

  79. Zhao, K., Guo, C., Han, P., Zhang, M., Cheng, Y., Yang, B.: Multiple time series forecasting with dynamic graph modeling. In: Proceedings of the VLDB Endowment (2024)

  80. Zhao, L., Song, Y., Zhang, C., Liu, Y., Wang, P., Lin, T., Deng, M., Li, H.: T-GCN: a temporal graph convolutional network for traffic prediction. IEEE Trans. Intell. Transp. Syst. 21(9), 3848–3858 (2019)

    Article  Google Scholar 

  81. Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., Zhang, W.: Informer: beyond efficient transformer for long sequence time-series forecasting. In: AAAI, vol. 35, pp. 11106–11115 (2021)

  82. Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., Jin, R.: Fedformer: frequency enhanced decomposed transformer for long-term series forecasting. In: International Conference on Machine Learning, pp. 27268–27286. PMLR (2022)

  83. Zhou, T., Niu, P., Sun, L., Jin, R., et al.: One fits all: Power general time series analysis by pretrained LM. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

  84. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. In: ICLR. OpenReview.net (2017)

Download references

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (62372179), Huawei Cloud Algorithm Innovation Lab, Independent Research Fund Denmark (8022-00246B and 8048-00038B), Villum Fonden (34328), and the Innovation Fund Denmark center, DIREC.

Author information

Author notes

  1. Xinle Wu and Xingjian Wu have contributed equally to this work.

Authors and Affiliations

  1. Department of Computer Science, Aalborg University, Aalborg, Denmark

    Xinle Wu & Christian S. Jensen

  2. School of Data Science & Engineering, East China Normal University, Shanghai, China

    Xingjian Wu, Bin Yang, Chenjuan Guo, Xiangfei Qiu & Jilin Hu

  3. Algorithm Innovation Lab, Huawei Cloud Computing Technologies, Hangzhou, China

    Lekui Zhou & Zhenli Sheng

Authors
  1. Xinle Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingjian Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lekui Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chenjuan Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiangfei Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jilin Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Zhenli Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Christian S. Jensen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bin Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, X., Wu, X., Yang, B. et al. AutoCTS++: zero-shot joint neural architecture and hyperparameter search for correlated time series forecasting. The VLDB Journal 33, 1743–1770 (2024). https://doi.org/10.1007/s00778-024-00872-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-024-00872-x

Keywords

Search

Navigation