Learning RBM with a DC programming Approach

Vidyadhar Upadhya, P. S. Sastry
Proceedings of the Ninth Asian Conference on Machine Learning, PMLR 77:498-513, 2017.

Abstract

By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log-likelihood. Interestingly, the traditional contrastive divergence algorithm is a special case of the above formulation and the hyperparameters of the two algorithms can be chosen such that the amount of computation per mini-batch is identical. We show that for a given computational budget the proposed algorithm almost always reaches a higher log-likelihood more rapidly, compared to the standard contrastive divergence algorithm. Further, we modify this algorithm to use the centered gradients and show that it is more efficient and effective compared to the standard centered gradient algorithm on benchmark datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v77-upadhya17a, title = {Learning RBM with a DC programming Approach}, author = {Upadhya, Vidyadhar and Sastry, P. S.}, booktitle = {Proceedings of the Ninth Asian Conference on Machine Learning}, pages = {498--513}, year = {2017}, editor = {Zhang, Min-Ling and Noh, Yung-Kyun}, volume = {77}, series = {Proceedings of Machine Learning Research}, address = {Yonsei University, Seoul, Republic of Korea}, month = {15--17 Nov}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v77/upadhya17a/upadhya17a.pdf}, url = {https://proceedings.mlr.press/v77/upadhya17a.html}, abstract = {By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log-likelihood. Interestingly, the traditional contrastive divergence algorithm is a special case of the above formulation and the hyperparameters of the two algorithms can be chosen such that the amount of computation per mini-batch is identical. We show that for a given computational budget the proposed algorithm almost always reaches a higher log-likelihood more rapidly, compared to the standard contrastive divergence algorithm. Further, we modify this algorithm to use the centered gradients and show that it is more efficient and effective compared to the standard centered gradient algorithm on benchmark datasets.} }
Endnote
%0 Conference Paper %T Learning RBM with a DC programming Approach %A Vidyadhar Upadhya %A P. S. Sastry %B Proceedings of the Ninth Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Min-Ling Zhang %E Yung-Kyun Noh %F pmlr-v77-upadhya17a %I PMLR %P 498--513 %U https://proceedings.mlr.press/v77/upadhya17a.html %V 77 %X By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log-likelihood. Interestingly, the traditional contrastive divergence algorithm is a special case of the above formulation and the hyperparameters of the two algorithms can be chosen such that the amount of computation per mini-batch is identical. We show that for a given computational budget the proposed algorithm almost always reaches a higher log-likelihood more rapidly, compared to the standard contrastive divergence algorithm. Further, we modify this algorithm to use the centered gradients and show that it is more efficient and effective compared to the standard centered gradient algorithm on benchmark datasets.
APA
Upadhya, V. & Sastry, P.S.. (2017). Learning RBM with a DC programming Approach. Proceedings of the Ninth Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 77:498-513 Available from https://proceedings.mlr.press/v77/upadhya17a.html.

Related Material