Practical GaussNewton Optimisation for Deep Learning
[edit]
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:557565, 2017.
Abstract
We present an efficient blockdiagonal approximation to the GaussNewton matrix for feedforward neural networks. Our resulting algorithm is competitive against stateoftheart firstorder optimisation methods, with sometimes significant improvement in optimisation performance. Unlike firstorder methods, for which hyperparameter tuning of the optimisation parameters is often a laborious process, our approach can provide good performance even when used with default settings. A side result of our work is that for piecewise linear transfer functions, the network objective function can have no differentiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation.
Related Material


