Training Neural Networks Without Gradients: A Scalable ADMM Approach

Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, Tom Goldstein
Proceedings of The 33rd International Conference on Machine Learning, PMLR 48:2722-2731, 2016.

Abstract

With the growing importance of large network models and enormous training datasets, GPUs have become increasingly necessary to train neural networks. This is largely because conventional optimization algorithms rely on stochastic gradient methods that don’t scale well to large numbers of cores in a cluster setting. Furthermore, the convergence of all gradient methods, including batch methods, suffers from common problems like saturation effects, poor conditioning, and saddle points. This paper explores an unconventional training method that uses alternating direction methods and Bregman iteration to train networks without gradient descent steps. The proposed method reduces the network training problem to a sequence of minimization sub-steps that can each be solved globally in closed form. The proposed method is advantageous because it avoids many of the caveats that make gradient methods slow on highly non-convex problems. In addition, the method exhibits strong scaling in the distributed setting, yielding linear speedups even when split over thousands of cores.

Cite this Paper

BibTeX
@InProceedings{pmlr-v48-taylor16, title = {Training Neural Networks Without Gradients: A Scalable ADMM Approach}, author = {Taylor, Gavin and Burmeister, Ryan and Xu, Zheng and Singh, Bharat and Patel, Ankit and Goldstein, Tom}, booktitle = {Proceedings of The 33rd International Conference on Machine Learning}, pages = {2722--2731}, year = {2016}, editor = {Balcan, Maria Florina and Weinberger, Kilian Q.}, volume = {48}, series = {Proceedings of Machine Learning Research}, address = {New York, New York, USA}, month = {20--22 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v48/taylor16.pdf}, url = {https://proceedings.mlr.press/v48/taylor16.html}, abstract = {With the growing importance of large network models and enormous training datasets, GPUs have become increasingly necessary to train neural networks. This is largely because conventional optimization algorithms rely on stochastic gradient methods that don’t scale well to large numbers of cores in a cluster setting. Furthermore, the convergence of all gradient methods, including batch methods, suffers from common problems like saturation effects, poor conditioning, and saddle points. This paper explores an unconventional training method that uses alternating direction methods and Bregman iteration to train networks without gradient descent steps. The proposed method reduces the network training problem to a sequence of minimization sub-steps that can each be solved globally in closed form. The proposed method is advantageous because it avoids many of the caveats that make gradient methods slow on highly non-convex problems. In addition, the method exhibits strong scaling in the distributed setting, yielding linear speedups even when split over thousands of cores.} }
Endnote
%0 Conference Paper %T Training Neural Networks Without Gradients: A Scalable ADMM Approach %A Gavin Taylor %A Ryan Burmeister %A Zheng Xu %A Bharat Singh %A Ankit Patel %A Tom Goldstein %B Proceedings of The 33rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2016 %E Maria Florina Balcan %E Kilian Q. Weinberger %F pmlr-v48-taylor16 %I PMLR %P 2722--2731 %U https://proceedings.mlr.press/v48/taylor16.html %V 48 %X With the growing importance of large network models and enormous training datasets, GPUs have become increasingly necessary to train neural networks. This is largely because conventional optimization algorithms rely on stochastic gradient methods that don’t scale well to large numbers of cores in a cluster setting. Furthermore, the convergence of all gradient methods, including batch methods, suffers from common problems like saturation effects, poor conditioning, and saddle points. This paper explores an unconventional training method that uses alternating direction methods and Bregman iteration to train networks without gradient descent steps. The proposed method reduces the network training problem to a sequence of minimization sub-steps that can each be solved globally in closed form. The proposed method is advantageous because it avoids many of the caveats that make gradient methods slow on highly non-convex problems. In addition, the method exhibits strong scaling in the distributed setting, yielding linear speedups even when split over thousands of cores.
RIS
TY - CPAPER TI - Training Neural Networks Without Gradients: A Scalable ADMM Approach AU - Gavin Taylor AU - Ryan Burmeister AU - Zheng Xu AU - Bharat Singh AU - Ankit Patel AU - Tom Goldstein BT - Proceedings of The 33rd International Conference on Machine Learning DA - 2016/06/11 ED - Maria Florina Balcan ED - Kilian Q. Weinberger ID - pmlr-v48-taylor16 PB - PMLR DP - Proceedings of Machine Learning Research VL - 48 SP - 2722 EP - 2731 L1 - http://proceedings.mlr.press/v48/taylor16.pdf UR - https://proceedings.mlr.press/v48/taylor16.html AB - With the growing importance of large network models and enormous training datasets, GPUs have become increasingly necessary to train neural networks. This is largely because conventional optimization algorithms rely on stochastic gradient methods that don’t scale well to large numbers of cores in a cluster setting. Furthermore, the convergence of all gradient methods, including batch methods, suffers from common problems like saturation effects, poor conditioning, and saddle points. This paper explores an unconventional training method that uses alternating direction methods and Bregman iteration to train networks without gradient descent steps. The proposed method reduces the network training problem to a sequence of minimization sub-steps that can each be solved globally in closed form. The proposed method is advantageous because it avoids many of the caveats that make gradient methods slow on highly non-convex problems. In addition, the method exhibits strong scaling in the distributed setting, yielding linear speedups even when split over thousands of cores. ER -
APA
Taylor, G., Burmeister, R., Xu, Z., Singh, B., Patel, A. & Goldstein, T.. (2016). Training Neural Networks Without Gradients: A Scalable ADMM Approach. Proceedings of The 33rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 48:2722-2731 Available from https://proceedings.mlr.press/v48/taylor16.html.

Related Material