A Baseline for Any Order Gradient Estimation in Stochastic Computation Graphs
[edit]
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:43434351, 2019.
Abstract
By enabling correct differentiation in Stochastic Computation Graphs (SCGs), the infinitely differentiable MonteCarlo estimator (DiCE) can generate correct estimates for the higher order gradients that arise in, e.g., multiagent reinforcement learning and metalearning. However, the baseline term in DiCE that serves as a control variate for reducing variance applies only to first order gradient estimation, limiting the utility of higherorder gradient estimates. To improve the sample efficiency of DiCE, we propose a new baseline term for higher order gradient estimation. This term may be easily included in the objective, and produces unbiased variancereduced estimators under (automatic) differentiation, without affecting the estimate of the objective itself or of the first order gradient estimate. It reuses the same baseline function (e.g., the statevalue function in reinforcement learning) already used for the first order baseline. We provide theoretical analysis and numerical evaluations of this new baseline, which demonstrate that it can dramatically reduce the variance of DiCE’s second order gradient estimators and also show empirically that it reduces the variance of third and fourth order gradients. This computational tool can be easily used to estimate higher order gradients with unprecedented efficiency and simplicity wherever automatic differentiation is utilised, and it has the potential to unlock applications of higher order gradients in reinforcement learning and metalearning.
Related Material


