article

An online gradient method with momentum for two-layer feedforward neural networks

Author:

Naimin ZhangAuthors Info & Claims

Applied Mathematics and Computation, Volume 212, Issue 2

Pages 488 - 498

https://doi.org/10.1016/j.amc.2009.02.038

Published: 01 June 2009 Publication History

Abstract

An online gradient method with momentum for two-layer feedforward neural networks is considered. The momentum coefficient is chosen in an adaptive manner to accelerate and stabilize the learning procedure of the network weights. Corresponding convergence results are proved, that is, the weak convergence result is proved under the uniformly boundedness assumption of the activation function and its derivatives, moreover, if the number of elements of the stationary point set for the error function is finite, then strong convergence result holds.

References

[1]

Bhaya, A. and Kaszkurewicz, E., Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method. Neural Networks. v17. 65-71.

Digital Library

[2]

Crema, A., Loreto, M. and Raydan, M., Spectral projected subgradient with a momentum term for the Lagrangean dual approach. Computers and Operations Research. v34 i10. 3174-3186.

[3]

Fine, T.L. and Mukherjee, S., Parameter convergence and learning curves for neural networks. Neural Computation. v11. 747-769.

Digital Library

[4]

Finnoff, W., Diffusion approximations for the constant learning rate backpropagation algorithm and resistance to local minima. Neural Computation. v6. 285-295.

Digital Library

[5]

Istook, E., Martinez, T., Istook, E. and Martinez, T., Improved backpropagation learning in neural networks with windowed momentum. International Journal of Neural System. v12. 303-318.

[6]

Convergence of an online gradient method for feedforward neural networks with stochastic inputs. Journal of Computer Applied Mathematics. v163. 165-176.

[7]

Lin, Chih-Jen, Projected gradient methods for nonnegative matrix factorization. Neural Computation. v19. 2756-2779.

Digital Library

[8]

Luo, Z., On the convergence of the LMS algorithm with adaptive learning rate for linear feedforward networks. Neural Computation. v3. 226-245.

[9]

Phansalkar, V.V. and Sastry, P.S., Analysis of the back-propagation algorithm with momentum. IEEE Transactions on Neural Networks. v5 i3. 505-506.

[10]

Qian, N., On the momentum term in gradient descent learning algorithms. Neural Networks. v12. 145-151.

Digital Library

[11]

Roy, S. and Shynk, J.J., Analysis of the momentum LMS algorithm. IEEE Transactions on Acoustics, Speech, and Signal Processing. v38 i12. 2088-2098.

[12]

. In: Rumelhart, D.E., McClelland, J.L. (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1. MIT Press, Cambridge, MA.

Digital Library

[13]

Rumelhart, D.E., Hinton, G.E. and Williams, R.J., Learning representations by back-propagating errors. Nature. v323. 533-536.

[14]

Torii, M. and Hagan, M.T., Stability of steepest descent with momentum for quadratic functions. IEEE Transactions on Neural Networks. v13 i3. 752-756.

[15]

Wu, W. and Xu, Y.S., Deterministic convergence of an online gradient method for neural networks. Journal of Computer Applied Mathematics. v144. 335-347.

Digital Library

[16]

Wu, W., Feng, G. and Li, X., Training multilayer perceptrons via minimization of sum of ridge functions. Advances in Computational Mathematics. v17. 331-347.

[17]

Wu, W., Xu, D. and Li, Z., Convergence of gradient method for Elman networks. Applied Mathematics and Mechanics, English Edition. v29 i9. 1231-1238.

[18]

Xu, D., Li, Z. and Wu, W., Convergence of approximated gradient method for Elman network. Neural Networks World. v3. 171-180.

[19]

Yuan, Y.X. and Sun, W.Y., Optimization Theory and Methods. 2001. Science Press, Beijing.

[20]

Zhang, H. and Huang, S., Convergent gradient ascent with momentum in general-sum games. Neurocomputing. v61. 449-454.

Digital Library

[21]

Zhang, N.M., Wu, W. and Zheng, G.F., Convergence of gradient method with momentum for two-layer feedforward neural networks. IEEE Transactions on Neural Networks. v17 i2. 522-525.

[22]

Zhang, N.M., Deterministic convergence of an online gradient method with momentum. ICIC 2006, Lecture Notes in Computer Science. v4113. 94-105.

[23]

Zweiri, H., Seneviratne, L.D. and Althoefer, K., Stability analysis of a three-term backpropagation algorithm. Neural Networks. v18. 1341-1347.

Digital Library

Cited By

Thirugnanasambandam KPrabu USaravanan DAnguraj DRaghav R(2023)Fortified Cuckoo Search Algorithm on training multi-layer perceptron for solving classification problemsPersonal and Ubiquitous Computing10.1007/s00779-023-01716-127:3(1039-1049)Online publication date: 31-Mar-2023
https://dl.acm.org/doi/10.1007/s00779-023-01716-1
Sadiq ATahir MAhmed AAlghushami A(2020)Normal parameter reduction algorithm in soft set based on hybrid binary particle swarm and biogeography optimizerNeural Computing and Applications10.1007/s00521-019-04423-232:16(12221-12239)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1007/s00521-019-04423-2
Aljarah IFaris HMirjalili SAl-Madi NSheta AMafarja M(2019)Evolving neural networks using bird swarm algorithm for data classification and regression applicationsCluster Computing10.1007/s10586-019-02913-522:4(1317-1345)Online publication date: 1-Dec-2019
https://dl.acm.org/doi/10.1007/s10586-019-02913-5
Show More Cited By

Recommendations

Boundedness and convergence of online gradient method with penalty and momentum

In this paper, the deterministic convergence of an online gradient method with penalty and momentum is investigated for training two-layer feedforward neural networks. The monotonicity of the new error function with the penalty term in the training ...
Convergence of gradient method with momentum for two-Layer feedforward neural networks

A gradient method with momentum for two-layer feedforward neural networks is considered. The learning rate is set to be a constant and the momentum factor an adaptive variable. Both the weak and strong convergence results are proved, as well as the ...
Convergence analysis of online gradient method for BP neural networks

This paper considers a class of online gradient learning methods for backpropagation (BP) neural networks with a single hidden layer. We assume that in each training cycle, each sample in the training set is supplied in a stochastic order to the network ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Applied Mathematics and Computation

Applied Mathematics and Computation Volume 212, Issue 2

June, 2009

275 pages

ISSN:0096-3003

Issue’s Table of Contents

Copyright © Elsevier Inc. © 2009.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 June 2009

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Thirugnanasambandam KPrabu USaravanan DAnguraj DRaghav R(2023)Fortified Cuckoo Search Algorithm on training multi-layer perceptron for solving classification problemsPersonal and Ubiquitous Computing10.1007/s00779-023-01716-127:3(1039-1049)Online publication date: 31-Mar-2023
https://dl.acm.org/doi/10.1007/s00779-023-01716-1
Sadiq ATahir MAhmed AAlghushami A(2020)Normal parameter reduction algorithm in soft set based on hybrid binary particle swarm and biogeography optimizerNeural Computing and Applications10.1007/s00521-019-04423-232:16(12221-12239)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1007/s00521-019-04423-2
Aljarah IFaris HMirjalili SAl-Madi NSheta AMafarja M(2019)Evolving neural networks using bird swarm algorithm for data classification and regression applicationsCluster Computing10.1007/s10586-019-02913-522:4(1317-1345)Online publication date: 1-Dec-2019
https://dl.acm.org/doi/10.1007/s10586-019-02913-5
Kaveh MKhishe MMosavi M(2019)Design and implementation of a neighborhood search biogeography-based optimization trainer for classifying sonar dataset using multi-layer perceptron neural networkAnalog Integrated Circuits and Signal Processing10.1007/s10470-018-1366-3100:2(405-428)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s10470-018-1366-3
Mosavi MKhishe MAkbarisani M(2017)Neural Network Trained by Biogeography-Based Optimizer with Chaos for Sonar Data Set ClassificationWireless Personal Communications: An International Journal10.1007/s11277-017-4110-x95:4(4623-4642)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.1007/s11277-017-4110-x
Wu HZhou YLuo QBasset M(2016)Training Feedforward Neural Networks Using Symbiotic Organisms Search AlgorithmComputational Intelligence and Neuroscience10.1155/2016/90630652016(12)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1155/2016/9063065
Varadarajan SWang HMiller PZhou H(2015)Fast convergence of regularised Region-based Mixture of Gaussians for dynamic background modellingComputer Vision and Image Understanding10.1016/j.cviu.2014.12.004136:C(45-58)Online publication date: 1-Jul-2015
https://dl.acm.org/doi/10.1016/j.cviu.2014.12.004
Zhang N(2013)Semistability of steepest descent with momentum for quadratic functionsNeural Computation10.1162/NECO_a_0043625:5(1277-1301)Online publication date: 1-May-2013
https://dl.acm.org/doi/10.1162/NECO_a_00436
Wang JWu WZurada J(2012)Computational properties of cyclic and almost-cyclic learning with momentum for feedforward neural networksProceedings of the 9th international conference on Advances in Neural Networks - Volume Part I10.1007/978-3-642-31346-2_61(545-554)Online publication date: 11-Jul-2012
https://dl.acm.org/doi/10.1007/978-3-642-31346-2_61

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents