Evaluating the Variance of Likelihood-Ratio Gradient Estimators

Seiya Tokui, Issei Sato
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:3414-3423, 2017.

Abstract

The likelihood-ratio method is often used to estimate gradients of stochastic computations, for which baselines are required to reduce the estimation variance. Many types of baselines have been proposed, although their degree of optimality is not well understood. In this study, we establish a novel framework of gradient estimation that includes most of the common gradient estimators as special cases. The framework gives a natural derivation of the optimal estimator that can be interpreted as a special case of the likelihood-ratio method so that we can evaluate the optimal degree of practical techniques with it. It bridges the likelihood-ratio method and the reparameterization trick while still supporting discrete variables. It is derived from the exchange property of the differentiation and integration. To be more specific, it is derived by the reparameterization trick and local marginalization analogous to the local expectation gradient. We evaluate various baselines and the optimal estimator for variational learning and show that the performance of the modern estimators is close to the optimal estimator.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-tokui17a, title = {Evaluating the Variance of Likelihood-Ratio Gradient Estimators}, author = {Seiya Tokui and Issei Sato}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {3414--3423}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/tokui17a/tokui17a.pdf}, url = {https://proceedings.mlr.press/v70/tokui17a.html}, abstract = {The likelihood-ratio method is often used to estimate gradients of stochastic computations, for which baselines are required to reduce the estimation variance. Many types of baselines have been proposed, although their degree of optimality is not well understood. In this study, we establish a novel framework of gradient estimation that includes most of the common gradient estimators as special cases. The framework gives a natural derivation of the optimal estimator that can be interpreted as a special case of the likelihood-ratio method so that we can evaluate the optimal degree of practical techniques with it. It bridges the likelihood-ratio method and the reparameterization trick while still supporting discrete variables. It is derived from the exchange property of the differentiation and integration. To be more specific, it is derived by the reparameterization trick and local marginalization analogous to the local expectation gradient. We evaluate various baselines and the optimal estimator for variational learning and show that the performance of the modern estimators is close to the optimal estimator.} }
Endnote
%0 Conference Paper %T Evaluating the Variance of Likelihood-Ratio Gradient Estimators %A Seiya Tokui %A Issei Sato %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-tokui17a %I PMLR %P 3414--3423 %U https://proceedings.mlr.press/v70/tokui17a.html %V 70 %X The likelihood-ratio method is often used to estimate gradients of stochastic computations, for which baselines are required to reduce the estimation variance. Many types of baselines have been proposed, although their degree of optimality is not well understood. In this study, we establish a novel framework of gradient estimation that includes most of the common gradient estimators as special cases. The framework gives a natural derivation of the optimal estimator that can be interpreted as a special case of the likelihood-ratio method so that we can evaluate the optimal degree of practical techniques with it. It bridges the likelihood-ratio method and the reparameterization trick while still supporting discrete variables. It is derived from the exchange property of the differentiation and integration. To be more specific, it is derived by the reparameterization trick and local marginalization analogous to the local expectation gradient. We evaluate various baselines and the optimal estimator for variational learning and show that the performance of the modern estimators is close to the optimal estimator.
APA
Tokui, S. & Sato, I.. (2017). Evaluating the Variance of Likelihood-Ratio Gradient Estimators. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:3414-3423 Available from https://proceedings.mlr.press/v70/tokui17a.html.

Related Material