Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Time series forecasting by recurrent product unit neural networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Time series forecasting (TSF) consists on estimating models to predict future values based on previously observed values of time series, and it can be applied to solve many real-world problems. TSF has been traditionally tackled by considering autoregressive neural networks (ARNNs) or recurrent neural networks (RNNs), where hidden nodes are usually configured using additive activation functions, such as sigmoidal functions. ARNNs are based on a short-term memory of the time series in the form of lagged time series values used as inputs, while RNNs include a long-term memory structure. The objective of this paper is twofold. First, it explores the potential of multiplicative nodes for ARNNs, by considering product unit (PU) activation functions, motivated by the fact that PUs are specially useful for modelling highly correlated features, such as the lagged time series values used as inputs for ARNNs. Second, it proposes a new hybrid RNN model based on PUs, by estimating the PU outputs from the combination of a long-term reservoir and the short-term lagged time series values. A complete set of experiments with 29 data sets shows competitive performance for both model proposals, and a set of statistical tests confirms that they achieve the state of the art in TSF, with specially promising results for the proposed hybrid RNN. The experiments in this paper show that the recurrent model is very competitive for relatively large time series, where longer forecast horizons are required, while the autoregressive model is a good selection if the data set is small or if a low computational cost is needed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. For the sake of clarity, reservoir representation is simplified: there is a link between each reservoir node and each PU, and all reservoir nodes receive \(y_{t-1}\) time series value as input. The interconnections between reservoir nodes are random. Internal connections of the reservoir are given by \({\varvec{\upkappa }}\).

  2. Available at http://www.neural-forecasting-competition.com.

  3. Which can be found at http://sci2s.ugr.es/keel/timeseries.php.

  4. Scaling the input data to positive values is required to avoid having complex numbers as output of the basis function. Additionally, the scaling considered also avoids having inputs equal to zero or one.

  5. http://www.uco.es/ayrna/datasets/MAPEresultsNCAA.

  6. http://www.uco.es/ayrna/datasets/MAPEresultsNCAA.

  7. In these kind of problems small variations in the inputs could produce large changes in the output of the TS. This situation could be modelled with the product units basis functions (as they are potential basis functions).

References

  1. Furquim G, Pessin G, Faiçal B, Mendiondo E, Ueyama J (2015) Improving the accuracy of a flood forecasting model by means of machine learning and chaos theory: a case study involving a real wireless sensor network deployment in brazil. In: Neural computing and applications, pp 1–13. doi:10.1007/s00521-015-1930-z

  2. Arroyo J, Maté C (2009) Forecasting histogram time series with \(k\)-nearest neighbours methods. Int J Forecast 25(1):192–207

    Article  Google Scholar 

  3. Arriandiaga A, Portillo E, Sánchez J, Cabanes I, Pombo I (2015) A new approach for dynamic modelling of energy consumption in the grinding process using recurrent neural networks. In: Neural computing and applications, pp 1–16. doi:10.1007/s00521-015-1957-1

  4. Hansen J, Nelson R (1997) Neural networks and traditional time series methods: a synergistic combination in state economic forecasts. IEEE Trans Neural Netw 8(4):863–873

    Article  Google Scholar 

  5. Sitte R, Sitte J (2000) Analysis of the predictive ability of time delay neural networks applied to the S&P 500 time series. IEEE Trans Syst Man Cybern Part C Appl Rev 30(4):568–572

    Article  Google Scholar 

  6. Connor J, Martin R, Atlas L (1994) Recurrent neural networks and robust time series prediction. IEEE Trans Neural Netw 5(2):240–254

    Article  Google Scholar 

  7. He X, Li C, Huang T, Li C, Huang J (2014) A recurrent neural network for solving bilevel linear programming problem. IEEE Trans Neural Netw Learn Syst 25(4):824–830

    Article  Google Scholar 

  8. Yan Z, Wang J (2014) Robust model predictive control of nonlinear systems with unmodeled dynamics and bounded uncertainties based on neural networks. IEEE Trans Neural Netw Learn Syst 25(3):457–469

    Article  MathSciNet  Google Scholar 

  9. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  10. Jaeger H (2002) Adaptive nonlinear system identification with echo state networks. In: Advances in neural information processing systems, pp 593–600

  11. Gallicchio C, Micheli A (2011) Architectural and markovian factors of echo state networks. Neural Netw 24(5):440–456

    Article  Google Scholar 

  12. Rodan A, Tino P (2011) Minimum complexity echo state network. IEEE Trans Neural Netw 22(1):131–144

    Article  Google Scholar 

  13. Hecht-Nielsen R (1989) Theory of the backpropagation neural network. In: International joint conference on neural networks 1989 IJCNN. IEEE, pp 593–605

  14. Pan F, Zhang H, Xia M (2009) A hybrid time-series forecasting model using extreme learning machines. In: Second international conference on Intelligent Computation Technology and Automation, ICICTA ’09, vol 1, pp 933–936

  15. Huang G-B, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B Cybern 42(2):513–529

    Article  Google Scholar 

  16. Durbin R, Rumelhart D (1989) Products units: a computationally powerful and biologically plausible extension to backpropagation networks. Neural Comput 1(1):133–142

    Article  Google Scholar 

  17. Goldberg DE et al (1989) Genetic algorithms in search, optimization, and machine learning, vol 412. Addison-Wesley, Reading Menlo Park

    MATH  Google Scholar 

  18. Li P, Tan Z, Yan L, Deng K (2011) Time series prediction of mining subsidence based on genetic algorithm neural network. In: 2011 international symposium on computer science and society (ISCCS), pp 83–86

  19. Luque C, Ferran J, Vinuela P (2007) Time series forecasting by means of evolutionary algorithms. In: IEEE international 2007 parallel and distributed processing symposium, IPDPS 2007, pp 1–7

  20. Cai X, Zhang N, Venayagamoorthy G, Wunsch D (2004) Time series prediction with recurrent neural networks using a hybrid PSO–EA algorithm. In: 2004 IEEE international joint conference on neural networks, 2004. Proceedings, vol 2, pp 1647–1652

  21. Martínez-Estudillo FJ, Hervás-Martínez C, Gutiérrez PA, Martínez-Estudillo AC (2008) Evolutionary product-unit neural networks classifiers. Neurocomputing 72(1–3):548–561

    Article  MATH  Google Scholar 

  22. Martínez-Estudillo AC, Martínez-Estudillo FJ, Hervás-Martínez C, García-Pedrajas N (2006) Evolutionary product unit based neural networks for regression. Neural Netw 19(4):477–486

    Article  MATH  Google Scholar 

  23. Dulakshi AWJ, Karunasingha SK, Li WK (2011) Evolutionary product unit based neural networks for hydrological time series analysis. J Hydroinf 13(4):825–841

    Article  Google Scholar 

  24. Piotrowski AP, Napiorkowski JJ (2012) Product-units neural networks for catchment runoff forecasting. Adv Water Resour 49:97–113

    Article  Google Scholar 

  25. Sundermeyer M, Oparin I, Gauvain J-L, Freiberg B, Schluter R, Ney H (2013) Comparison of feedforward and recurrent neural network language models. In: 2013 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 8430–8434

  26. Lukoševičius M, Jaeger H (2009) Reservoir computing approaches to recurrent neural network training. Comput Sci Rev 3(3):127–149

    Article  MATH  Google Scholar 

  27. Hansen N (2006) The CMA evolution strategy: a comparing review. In: Towards a new evolutionary computation. Studies in fuzziness and soft computing, vol 192. Springer, Berlin, pp 75–102

  28. Jastrebski G, Arnold D (2006) Improving evolution strategies through active covariance matrix adaptation. In: IEEE congress on 2006 evolutionary computation, CEC 2006, pp 2814–2821

  29. Heidrich-Meisner V, Igel C (2009) Neuroevolution strategies for episodic reinforcement learning. J Algorithms 64(4):152–168

    Article  MATH  Google Scholar 

  30. Moriguchi H, Honiden S (2012) CMA-TWEANN: efficient optimization of neural networks via self-adaptation and seamless augmentation. In: Proceedings of the 14th annual conference on genetic and evolutionary computation. ACM, pp 903–910

  31. Gundogdu O, Egrioglu E, Aladag C, Yolcu U (2015) Multiplicative neuron model artificial neural network based on gaussian activation function. Neural Comput Appl 27:927–935

    Article  Google Scholar 

  32. Yadav R, Kalra P, John J (2007) Time series prediction with single multiplicative neuron model. Appl Soft Comput 7(4):1157–1163 (Soft computing for time series prediction)

    Article  Google Scholar 

  33. Zhao L, Yang Y (2009) PSO-based single multiplicative neuron model for time series prediction. Expert Syst Appl 36(2):2805–2812 (Part 2)

    Article  MathSciNet  Google Scholar 

  34. Attia M, Sallam E, Fahmy M (Aug 2012) A proposed generalized mean single multiplicative neuron model. In: 2012 IEEE international conference on intelligent computer communication and processing (ICCP), pp 73–78

  35. Egrioglu E, Yolcu U, Aladag C, Bas E (2015) Recurrent multiplicative neuron model artificial neural network for non-linear time series forecasting. Neural Process Lett 41(2):249–258

    Article  Google Scholar 

  36. Gutiérrez PA, Segovia-Vargas MJ, Salcedo-Sanz S, Hervás-Martínez C, Sanchís A, Portilla-Figueras JA, Fernández-Navarro F (2010) Hybridizing logistic regression with product unit and rbf networks for accurate detection and prediction of banking crises. Omega 38(5):333–344

    Article  Google Scholar 

  37. Saini L, Soni M (2002) Artificial neural network based peak load forecasting using Levenberg–Marquardt and quasi-Newton methods. IEE Proc Gener Transm Distrib 149(5):578–584

    Article  Google Scholar 

  38. Hansen N, Niederberger ASP, Guzzella L, Koumoutsakos P (2009) A method for handling uncertainty in evolutionary optimization with an application to feedback control of combustion. IEEE Trans Evolut Comput 13(1):180–197

    Article  Google Scholar 

  39. Ros R, Hansen N (2008) A simple modification in CMA-ES achieving linear time and space complexity. In: Proceedings of the 10th international conference on parallel problem solving from nature: PPSN X. Springer, pp 296–305

  40. Huang G-B, Zhu Q-Y, Siew C-K (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE international joint conference on neural networks, 2004. Proceedings, vol 2, pp 985–990

  41. Ozturk MC, Xu D, Príncipe JC (2007) Analysis and design of echo state networks. Neural Comput 19(1):111–138

    Article  MATH  Google Scholar 

  42. Bergmeir C, Triguero I, Molina D, Aznarte J, Benitez J (2012) Time series modeling and forecasting using memetic algorithms for regimen-switching models. IEEE Trans Neural Netw Learn Syst 23(11):1841–1847

    Article  Google Scholar 

  43. Alcalá-Fdez J, Fernández A, Luengo J, Derrac J, García S (2011) Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. Mult Valued Logic Soft Comput 17(2–3):255–287

    Google Scholar 

  44. Said SE, Dickey DA (1984) Testing for unit roots in autoregressive-moving average models of unknown order. Biometrika 71(3):599–607

    Article  MathSciNet  MATH  Google Scholar 

  45. Ragulskis M, Lukoseviciute K (2009) Non-uniform attractor embedding for time series forecasting by fuzzy inference systems. Neurocomputing 72(10):2618–2626

    Article  Google Scholar 

  46. Crone S, Dhawan R (2007) Forecasting seasonal time series with neural networks: a sensitivity analysis of architecture parameters. In: International joint conference on neural networks, IJCNN 2007, pp 2099–2104

  47. Chow TWS, Leung C (1996) Nonlinear autoregressive integrated neural network model for short-term load forecasting. IEE Proc Gener Transm Distrib 143(5):500–506

    Article  Google Scholar 

  48. Redel-Macías MD, Fernández-Navarro F, Gutiérrez PA, Cubero-Atienza AJ, Hervás-Martínez C (2013) Ensembles of evolutionary product unit or RBF neural networks for the identification of sound for pass-by noise test in vehicles. Neurocomputing 109:56–65

    Article  Google Scholar 

  49. Demsar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MathSciNet  MATH  Google Scholar 

  50. Zar JH et al (1999) Biostatistical analysis: Pearson Education India. Prentice Hall City, New Jersey

  51. Friedman M (1940) A comparison of alternative tests of significance for the problem of m rankings. Ann Math Stat 11(1):86–92

    Article  MathSciNet  MATH  Google Scholar 

  52. Dunn OJ (1961) Multiple comparisons among means. J Am Stat Assoc 56(293):52–64

    Article  MathSciNet  MATH  Google Scholar 

  53. Hochberg Y, Tamhane AC (1987) Multiple comparison procedures. Wiley, New York

    Book  MATH  Google Scholar 

  54. Aznarte JL, Alcalá-Fdez J, Arauzo-Azofra A, Benítez JM (2012) Financial time series forecasting with a bio-inspired fuzzy model. Expert Syst Appl 39(16):12302–12309

    Article  Google Scholar 

  55. Adhikari R, Agrawal R (2012) Forecasting strong seasonal time series with artificial neural networks. J Sci Ind Res 71(10):657

    Google Scholar 

  56. Rocha T, Paredes S, de Carvalho P, Henriques J (2013) An effective wavelet strategy for the trend prediction of physiological time series with application to phealth systems. In: 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC) (2013). IEEE, pp 6788–6791

  57. Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447

    Article  Google Scholar 

  58. Nikolaou A, Gutiérrez PA, Durán A, Dicaire I, Fernández-Navarro F, Hervás-Martínez C (2015) Detection of early warning signals in paleoclimate data using a genetic time series segmentation algorithm. Clim Dyn 44(7–8):1919–1933

    Article  Google Scholar 

  59. Fernández-Navarro F, Hervás-Martínez C, Gutiérrez PA (2013) Generalised gaussian radial basis function neural networks. Soft Comput 17(3):519–533

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Quantitative Methods, Universidad Loyola Andalucia, Seville, Spain

    F. Fernández-Navarro

  2. Advanced Concepts Team, European Space Research and Technology Centre (ESTEC), European Space Agency (ESA), Noordwijk, Netherlands

    Maria Angeles de la Cruz

  3. Department of Computer Science and Numerical Analysis, University of Córdoba, Córdoba, Spain

    P. A. Gutiérrez & C. Hervás-Martínez

  4. Department of Computer Science, Universidad Politécnica Salesiana, Quito, Ecuador

    A. Castaño

Authors
  1. F. Fernández-Navarro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Angeles de la Cruz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. A. Gutiérrez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. A. Castaño
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. C. Hervás-Martínez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to F. Fernández-Navarro.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fernández-Navarro, F., de la Cruz, M.A., Gutiérrez, P.A. et al. Time series forecasting by recurrent product unit neural networks. Neural Comput & Applic 29, 779–791 (2018). https://doi.org/10.1007/s00521-016-2494-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-016-2494-2

Keywords

Search

Navigation