Bregman divergence as general framework to estimate unnormalized statistical models

Michael Gutmann, Jun-ichiro Hirayama
Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, PMLR R9:321-328, 2011.

Abstract

We show that the Bregman divergence provides a rich framework to estimate unnormalized statistical models for continuous or discrete random variables, that is, models which do not integrate or sum to one, respectively. We prove that recent estimation methods such as noise-contrastive estimation, ratio matching, and score matching belong to the proposed framework, and explain their interconnection based on supervised learning. Further, we discuss the role of boosting in unsupervised learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR9-gutmann11a, title = {Bregman divergence as general framework to estimate unnormalized statistical models}, author = {Gutmann, Michael and Hirayama, Jun-ichiro}, booktitle = {Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence}, pages = {321--328}, year = {2011}, editor = {Cozman, Fabio and Pfeffer, Avi}, volume = {R9}, series = {Proceedings of Machine Learning Research}, month = {14--17 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r9/main/assets/gutmann11a/gutmann11a.pdf}, url = {https://proceedings.mlr.press/r9/gutmann11a.html}, abstract = {We show that the Bregman divergence provides a rich framework to estimate unnormalized statistical models for continuous or discrete random variables, that is, models which do not integrate or sum to one, respectively. We prove that recent estimation methods such as noise-contrastive estimation, ratio matching, and score matching belong to the proposed framework, and explain their interconnection based on supervised learning. Further, we discuss the role of boosting in unsupervised learning.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Bregman divergence as general framework to estimate unnormalized statistical models %A Michael Gutmann %A Jun-ichiro Hirayama %B Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2011 %E Fabio Cozman %E Avi Pfeffer %F pmlr-vR9-gutmann11a %I PMLR %P 321--328 %U https://proceedings.mlr.press/r9/gutmann11a.html %V R9 %X We show that the Bregman divergence provides a rich framework to estimate unnormalized statistical models for continuous or discrete random variables, that is, models which do not integrate or sum to one, respectively. We prove that recent estimation methods such as noise-contrastive estimation, ratio matching, and score matching belong to the proposed framework, and explain their interconnection based on supervised learning. Further, we discuss the role of boosting in unsupervised learning. %Z Reissued by PMLR on 04 October 2026.
APA
Gutmann, M. & Hirayama, J.. (2011). Bregman divergence as general framework to estimate unnormalized statistical models. Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R9:321-328 Available from https://proceedings.mlr.press/r9/gutmann11a.html. Reissued by PMLR on 04 October 2026.

Related Material