Stochastic Spectral Descent for Restricted Boltzmann Machines

David Carlson, Volkan Cevher, Lawrence Carin
Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, PMLR 38:111-119, 2015.

Abstract

Restricted Boltzmann Machines (RBMs) are widely used as building blocks for deep learning models. Learning typically proceeds by using stochastic gradient descent, and the gradients are estimated with sampling methods. However, the gradient estimation is a computational bottleneck, so better use of the gradients will speed up the descent algorithm. To this end, we first derive upper bounds on the RBM cost function, then show that descent methods can have natural ad- vantages by operating in the L∞and Shatten-∞norm. We introduce a new method called “Stochastic Spectral Descent” that updates parameters in the normed space. Empirical results show dramatic improvements over stochastic gradient descent, and have only have a fractional increase on the per-iteration cost.

Cite this Paper

BibTeX
@InProceedings{pmlr-v38-carlson15, title = {{Stochastic Spectral Descent for Restricted Boltzmann Machines}}, author = {Carlson, David and Cevher, Volkan and Carin, Lawrence}, booktitle = {Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics}, pages = {111--119}, year = {2015}, editor = {Lebanon, Guy and Vishwanathan, S. V. N.}, volume = {38}, series = {Proceedings of Machine Learning Research}, address = {San Diego, California, USA}, month = {09--12 May}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v38/carlson15.pdf}, url = {https://proceedings.mlr.press/v38/carlson15.html}, abstract = {Restricted Boltzmann Machines (RBMs) are widely used as building blocks for deep learning models. Learning typically proceeds by using stochastic gradient descent, and the gradients are estimated with sampling methods. However, the gradient estimation is a computational bottleneck, so better use of the gradients will speed up the descent algorithm. To this end, we first derive upper bounds on the RBM cost function, then show that descent methods can have natural ad- vantages by operating in the L∞and Shatten-∞norm. We introduce a new method called “Stochastic Spectral Descent” that updates parameters in the normed space. Empirical results show dramatic improvements over stochastic gradient descent, and have only have a fractional increase on the per-iteration cost.} }
Endnote
%0 Conference Paper %T Stochastic Spectral Descent for Restricted Boltzmann Machines %A David Carlson %A Volkan Cevher %A Lawrence Carin %B Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2015 %E Guy Lebanon %E S. V. N. Vishwanathan %F pmlr-v38-carlson15 %I PMLR %P 111--119 %U https://proceedings.mlr.press/v38/carlson15.html %V 38 %X Restricted Boltzmann Machines (RBMs) are widely used as building blocks for deep learning models. Learning typically proceeds by using stochastic gradient descent, and the gradients are estimated with sampling methods. However, the gradient estimation is a computational bottleneck, so better use of the gradients will speed up the descent algorithm. To this end, we first derive upper bounds on the RBM cost function, then show that descent methods can have natural ad- vantages by operating in the L∞and Shatten-∞norm. We introduce a new method called “Stochastic Spectral Descent” that updates parameters in the normed space. Empirical results show dramatic improvements over stochastic gradient descent, and have only have a fractional increase on the per-iteration cost.
RIS
TY - CPAPER TI - Stochastic Spectral Descent for Restricted Boltzmann Machines AU - David Carlson AU - Volkan Cevher AU - Lawrence Carin BT - Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics DA - 2015/02/21 ED - Guy Lebanon ED - S. V. N. Vishwanathan ID - pmlr-v38-carlson15 PB - PMLR DP - Proceedings of Machine Learning Research VL - 38 SP - 111 EP - 119 L1 - http://proceedings.mlr.press/v38/carlson15.pdf UR - https://proceedings.mlr.press/v38/carlson15.html AB - Restricted Boltzmann Machines (RBMs) are widely used as building blocks for deep learning models. Learning typically proceeds by using stochastic gradient descent, and the gradients are estimated with sampling methods. However, the gradient estimation is a computational bottleneck, so better use of the gradients will speed up the descent algorithm. To this end, we first derive upper bounds on the RBM cost function, then show that descent methods can have natural ad- vantages by operating in the L∞and Shatten-∞norm. We introduce a new method called “Stochastic Spectral Descent” that updates parameters in the normed space. Empirical results show dramatic improvements over stochastic gradient descent, and have only have a fractional increase on the per-iteration cost. ER -
APA
Carlson, D., Cevher, V. & Carin, L.. (2015). Stochastic Spectral Descent for Restricted Boltzmann Machines. Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 38:111-119 Available from https://proceedings.mlr.press/v38/carlson15.html.

Related Material