Empirical Risk Minimization of Graphical Model Parameters Given Approximate Inference, Decoding, and Model Structure

Veselin Stoyanov, Alexander Ropson, Jason Eisner
Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, PMLR 15:725-733, 2011.

Abstract

Graphical models are often used “inappropriately,” with approximations in the topology, inference, and prediction. Yet it is still common to train their parameters to approximately maximize training likelihood. We argue that instead, one should seek the parameters that minimize the empirical risk of the entire imperfect system. We show how to locally optimize this risk using back-propagation and stochastic meta-descent. Over a range of synthetic-data problems, compared to the usual practice of choosing approximate MAP parameters, our approach significantly reduces loss on test data, sometimes by an order of magnitude.

Cite this Paper

BibTeX
@InProceedings{pmlr-v15-stoyanov11a, title = {Empirical Risk Minimization of Graphical Model Parameters Given Approximate Inference, Decoding, and Model Structure}, author = {Stoyanov, Veselin and Ropson, Alexander and Eisner, Jason}, booktitle = {Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics}, pages = {725--733}, year = {2011}, editor = {Gordon, Geoffrey and Dunson, David and Dudík, Miroslav}, volume = {15}, series = {Proceedings of Machine Learning Research}, address = {Fort Lauderdale, FL, USA}, month = {11--13 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v15/stoyanov11a/stoyanov11a.pdf}, url = {https://proceedings.mlr.press/v15/stoyanov11a.html}, abstract = {Graphical models are often used “inappropriately,” with approximations in the topology, inference, and prediction. Yet it is still common to train their parameters to approximately maximize training likelihood. We argue that instead, one should seek the parameters that minimize the empirical risk of the entire imperfect system. We show how to locally optimize this risk using back-propagation and stochastic meta-descent. Over a range of synthetic-data problems, compared to the usual practice of choosing approximate MAP parameters, our approach significantly reduces loss on test data, sometimes by an order of magnitude.} }
Endnote
%0 Conference Paper %T Empirical Risk Minimization of Graphical Model Parameters Given Approximate Inference, Decoding, and Model Structure %A Veselin Stoyanov %A Alexander Ropson %A Jason Eisner %B Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2011 %E Geoffrey Gordon %E David Dunson %E Miroslav Dudík %F pmlr-v15-stoyanov11a %I PMLR %P 725--733 %U https://proceedings.mlr.press/v15/stoyanov11a.html %V 15 %X Graphical models are often used “inappropriately,” with approximations in the topology, inference, and prediction. Yet it is still common to train their parameters to approximately maximize training likelihood. We argue that instead, one should seek the parameters that minimize the empirical risk of the entire imperfect system. We show how to locally optimize this risk using back-propagation and stochastic meta-descent. Over a range of synthetic-data problems, compared to the usual practice of choosing approximate MAP parameters, our approach significantly reduces loss on test data, sometimes by an order of magnitude.
RIS
TY - CPAPER TI - Empirical Risk Minimization of Graphical Model Parameters Given Approximate Inference, Decoding, and Model Structure AU - Veselin Stoyanov AU - Alexander Ropson AU - Jason Eisner BT - Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics DA - 2011/06/14 ED - Geoffrey Gordon ED - David Dunson ED - Miroslav Dudík ID - pmlr-v15-stoyanov11a PB - PMLR DP - Proceedings of Machine Learning Research VL - 15 SP - 725 EP - 733 L1 - http://proceedings.mlr.press/v15/stoyanov11a/stoyanov11a.pdf UR - https://proceedings.mlr.press/v15/stoyanov11a.html AB - Graphical models are often used “inappropriately,” with approximations in the topology, inference, and prediction. Yet it is still common to train their parameters to approximately maximize training likelihood. We argue that instead, one should seek the parameters that minimize the empirical risk of the entire imperfect system. We show how to locally optimize this risk using back-propagation and stochastic meta-descent. Over a range of synthetic-data problems, compared to the usual practice of choosing approximate MAP parameters, our approach significantly reduces loss on test data, sometimes by an order of magnitude. ER -
APA
Stoyanov, V., Ropson, A. & Eisner, J.. (2011). Empirical Risk Minimization of Graphical Model Parameters Given Approximate Inference, Decoding, and Model Structure. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 15:725-733 Available from https://proceedings.mlr.press/v15/stoyanov11a.html.

Related Material