Stochastic Discriminative EM

Andres Masegosa
Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, PMLR R12:785-794, 2014.

Abstract

Stochastic discriminative EM (sdEM) is an online-EM-type algorithm for discriminative training of probabilistic generative models be- longing to the natural exponential family. In this work, we introduce and justify this algorithm as a stochastic natural gradient descent method, i.e. a method which accounts for the informa- tion geometry in the parameter space of the sta- tistical model. We show how this learning algo- rithm can be used to train probabilistic genera- tive models by minimizing different discrimina- tive loss functions, such as the negative condi- tional log-likelihood and the Hinge loss. The re- sulting models trained by sdEM are always gen- erative (i.e. they define a joint probability distri- bution) and, in consequence, allows to deal with missing data and latent variables in a principled way either when being learned or when making predictions. The performance of this method is illustrated by several text classification problems for which a multinomial naive Bayes and a latent Dirichlet allocation based classifier are learned using different discriminative loss functions.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR12-masegosa14a, title = {Stochastic Discriminative {EM}}, author = {Masegosa, Andres}, booktitle = {Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence}, pages = {785--794}, year = {2014}, editor = {Zhang, Nevin L. and Tian, Jin}, volume = {R12}, series = {Proceedings of Machine Learning Research}, month = {23--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r12/main/assets/masegosa14a/masegosa14a.pdf}, url = {https://proceedings.mlr.press/r12/masegosa14a.html}, abstract = {Stochastic discriminative EM (sdEM) is an online-EM-type algorithm for discriminative training of probabilistic generative models be- longing to the natural exponential family. In this work, we introduce and justify this algorithm as a stochastic natural gradient descent method, i.e. a method which accounts for the informa- tion geometry in the parameter space of the sta- tistical model. We show how this learning algo- rithm can be used to train probabilistic genera- tive models by minimizing different discrimina- tive loss functions, such as the negative condi- tional log-likelihood and the Hinge loss. The re- sulting models trained by sdEM are always gen- erative (i.e. they define a joint probability distri- bution) and, in consequence, allows to deal with missing data and latent variables in a principled way either when being learned or when making predictions. The performance of this method is illustrated by several text classification problems for which a multinomial naive Bayes and a latent Dirichlet allocation based classifier are learned using different discriminative loss functions.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Stochastic Discriminative EM %A Andres Masegosa %B Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2014 %E Nevin L. Zhang %E Jin Tian %F pmlr-vR12-masegosa14a %I PMLR %P 785--794 %U https://proceedings.mlr.press/r12/masegosa14a.html %V R12 %X Stochastic discriminative EM (sdEM) is an online-EM-type algorithm for discriminative training of probabilistic generative models be- longing to the natural exponential family. In this work, we introduce and justify this algorithm as a stochastic natural gradient descent method, i.e. a method which accounts for the informa- tion geometry in the parameter space of the sta- tistical model. We show how this learning algo- rithm can be used to train probabilistic genera- tive models by minimizing different discrimina- tive loss functions, such as the negative condi- tional log-likelihood and the Hinge loss. The re- sulting models trained by sdEM are always gen- erative (i.e. they define a joint probability distri- bution) and, in consequence, allows to deal with missing data and latent variables in a principled way either when being learned or when making predictions. The performance of this method is illustrated by several text classification problems for which a multinomial naive Bayes and a latent Dirichlet allocation based classifier are learned using different discriminative loss functions. %Z Reissued by PMLR on 04 October 2026.
APA
Masegosa, A.. (2014). Stochastic Discriminative EM. Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R12:785-794 Available from https://proceedings.mlr.press/r12/masegosa14a.html. Reissued by PMLR on 04 October 2026.

Related Material