Online Variance Reduction with Mixtures

Zalán Borsos, Sebastian Curi, Kfir Yehuda Levy, Andreas Krause
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:705-714, 2019.

Abstract

Adaptive importance sampling for stochastic optimization is a promising approach that offers improved convergence through variance reduction. In this work, we propose a new framework for variance reduction that enables the use of mixtures over predefined sampling distributions, which can naturally encode prior knowledge about the data. While these sampling distributions are fixed, the mixture weights are adapted during the optimization process. We propose VRM, a novel and efficient adaptive scheme that asymptotically recovers the best mixture weights in hindsight and can also accommodate sampling distributions over sets of points. We empirically demonstrate the versatility of VRM in a range of applications.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-borsos19a, title = {Online Variance Reduction with Mixtures}, author = {Borsos, Zal{\'a}n and Curi, Sebastian and Levy, Kfir Yehuda and Krause, Andreas}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {705--714}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/borsos19a/borsos19a.pdf}, url = {https://proceedings.mlr.press/v97/borsos19a.html}, abstract = {Adaptive importance sampling for stochastic optimization is a promising approach that offers improved convergence through variance reduction. In this work, we propose a new framework for variance reduction that enables the use of mixtures over predefined sampling distributions, which can naturally encode prior knowledge about the data. While these sampling distributions are fixed, the mixture weights are adapted during the optimization process. We propose VRM, a novel and efficient adaptive scheme that asymptotically recovers the best mixture weights in hindsight and can also accommodate sampling distributions over sets of points. We empirically demonstrate the versatility of VRM in a range of applications.} }
Endnote
%0 Conference Paper %T Online Variance Reduction with Mixtures %A Zalán Borsos %A Sebastian Curi %A Kfir Yehuda Levy %A Andreas Krause %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-borsos19a %I PMLR %P 705--714 %U https://proceedings.mlr.press/v97/borsos19a.html %V 97 %X Adaptive importance sampling for stochastic optimization is a promising approach that offers improved convergence through variance reduction. In this work, we propose a new framework for variance reduction that enables the use of mixtures over predefined sampling distributions, which can naturally encode prior knowledge about the data. While these sampling distributions are fixed, the mixture weights are adapted during the optimization process. We propose VRM, a novel and efficient adaptive scheme that asymptotically recovers the best mixture weights in hindsight and can also accommodate sampling distributions over sets of points. We empirically demonstrate the versatility of VRM in a range of applications.
APA
Borsos, Z., Curi, S., Levy, K.Y. & Krause, A.. (2019). Online Variance Reduction with Mixtures. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:705-714 Available from https://proceedings.mlr.press/v97/borsos19a.html.

Related Material