Practical Gauss-Newton Optimisation for Deep Learning

Proceedings of the 34th International Conference on Machine Learning

2017

70

Proceedings of Machine Learning Research

0

PMLR

We present an efficient block-diagonal approximation to the Gauss-Newton matrix for feedforward neural networks. Our resulting algorithm is competitive against state-of-the-art first-order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for which hyperparameter tuning of the optimisation parameters is often a laborious process, our approach can provide good performance even when used with default settings. A side result of our work is that for piecewise linear transfer functions, the network objective function can have no differentiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation.

inproceedings

botev17a

Practical {G}auss-{N}ewton Optimisation for Deep Learning

Aleksandar Botev and Hippolyt Ritter and David Barber

557

565

557-565

557

false

given family Doina Precup

given family Yee Whye Teh

given family Aleksandar Botev

given family Hippolyt Ritter

given family David Barber

2017-07-17

Proceedings of the 34th International Conference on Machine Learning

inproceedings

date-parts
2017 7 17

label link Supplementary PDF http://proceedings.mlr.press/v70/botev17a/botev17a-supp.pdf