Abstract
We propose the Pre-trained Financial Model (PFM) for price movement forecasting, which is critical in the automated trading systems in the Stock and Futures markets. Inspired by recent successes of pre-trained large language models in tackling NLP tasks, our PFM adopts a pretraining-and-finetuning strategy for obtaining capable models that are adapted to various downstream price-forecasting tasks. During the pre-training stage, we train a sequence prediction backbone with multi-task learning by adopting both a supervised learning objective and an unsupervised regularization target. Our approach differs from the common masked language modeling (MLM) used in NLP studies. We develop a per-step target variable generation strategy for eliciting future predictions from the transformer encoder-decoder architecture. We verify our pre-trained model on various practical downstream forecasting tasks, including lagged movement regression, movement direction classification, and selective trading with best performing stocks. Specifically, during the fine-tuning stage, we retain the pre-trained encoder and replace the decoder with specific downstream task decoders. We then perform supervised task-specific target generation learning as the fine-tuning process. Through extensive numerical studies and analysis, we demonstrate that our fine-tuned financial model can achieve a 5–15% improvement over downstream regression and classification tasks and over 40% in selective trading task.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Blondel, M., Teboul, O., Berthet, Q., Djolonga, J.: Fast differentiable sorting and ranking. In: ICML (2020)
Box, G.E.P., Jenkins, G.M.: Some recent advances in forecasting and control. J. Roy. Statist. Soc. (1968)
Brown, T.B., et al.: Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)
Cartea, A., Donnelly, R., Jaimungal, S.: Enhancing trading strategies with order book signals. Appl. Math. Financ. 25(1), 1–35 (2018)
Castoe, M.: Predicting stock market price direction with uncertainty using quantile regression forest (2020)
Cuturi, M.: Sinkhorn distances: lightspeed computation of optimal transport. In: NeurIPS, pp. 2292–2300 (2013)
Cuturi, M., Teboul, O., Vert, J.P.: Differentiable ranking and sorting using optimal transport. In: NeurIPS (2019)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Fan, C., Lu, H., Huang, A.: A novel differentiable rank learning method towards stock movement quantile forecasting. In: European Conference on Artificial Intelligence (2023)
Fan, C., et al.: Multi-horizon time series forecasting with temporal attention learning. In: SIGKDD (2019)
Feng, F., Chen, H., He, X., Ding, J., Sun, M., Chua, T.S.: Enhancing stock movement prediction with adversarial training. In: IJCAI (2019)
Gould, M.D., Porter, M.A., Williams, S., McDonald, M., Fenn, D.J., Howison, S.D.: Limit order books. Quant. Financ. 13(11), 1709–1742 (2013)
Holt, C.C.: Forecasting seasonals and trends by exponentially weighted moving averages. Int. J. Forecast. (2004)
Ke, G., et al.: Lightgbm: a highly efficient gradient boosting decision tree. In: NeurIPS (2017)
Kearns, M., Kulesza, A., Nevmyvaka, Y.: Empirical limitations on high-frequency trading profitability. J. Trading 5(4), 50–62 (2010)
Koenker, R., Bassett, Jr., G.: Regression quantiles. Econometrica: J. Economet. Soc. 33–50 (1978)
Lim, B., Arık, S.Ö., Loeff, N., Pfister, T.: Temporal fusion transformers for interpretable multi-horizon time series forecasting. Int. J. Forecast. (2021)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: ICCV (2017)
Vaswani, A., et al.: Attention is all you need. In: NIPS (2017)
Wen, R., Torkkola, K., Narayanaswamy, B., Madeka, D.: A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053 (2017)
Winters, P.R.: Forecasting sales by exponentially weighted moving averages. Manag. Sci. (1960)
Xu, Y., Cohen, S.B.: Stock movement prediction from tweets and historical prices. In: ACL (2018)
Yoo, J., Soun, Y., Park, Y., Kang, U.: Accurate multivariate stock movement prediction via data-axis transformer with multi-level contexts. In: SIGKDD (2021)
Zhang, L., Aggarwal, C., Qi, G.J.: Stock price prediction via discovering multi-frequency trading patterns. In: SIGKDD (2017)
Zhang, Z., Zohren, S.: Multi-horizon forecasting for limit order books: novel deep learning approaches and hardware acceleration using intelligent processing units. arXiv preprint arXiv:2105.10430 (2021)
Zhang, Z., Zohren, S., Roberts, S.: Deeplob: deep convolutional neural networks for limit order books. In: IEEE Transactions on Signal Processing (2018)
Acknowledgments
This work is supported by the National Natural Science Foundation of China, Project 62106156, and Starting Fund of South China Normal University.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Fan, C., Pang, T., Huang, A. (2024). Pre-trained Financial Model for Price Movement Forecasting. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1969. Springer, Singapore. https://doi.org/10.1007/978-981-99-8184-7_17
Download citation
DOI: https://doi.org/10.1007/978-981-99-8184-7_17
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8183-0
Online ISBN: 978-981-99-8184-7
eBook Packages: Computer ScienceComputer Science (R0)