Inference by Minimizing Size, Divergence, or their Sum

Sebastian Riedel, David Smith, Andrew McCallum
Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence, PMLR R8:491-498, 2010.

Abstract

We speed up marginal inference by ignoring factors that do not significantly contribute to overall accuracy. In order to pick a suitable subset of factors to ignore, we propose three schemes: minimizing the number of model factors under a bound on the KL divergence between pruned and full models; minimizing the KL divergence under a bound on factor count; and minimizing the weighted sum of KL divergence and factor count. All three problems are solved using an approximation of the KL divergence than can be calculated in terms of marginals computed on a sim- ple seed graph. Applied to synthetic im- age denoising and to three different types of NLP parsing models, this technique performs marginal inference up to 11 times faster than loopy BP, with graph sizes reduced up to 98%—at comparable error in marginals and parsing accuracy. We also show that mini- mizing the weighted sum of divergence and size is substantially faster than minimizing either of the other objectives based on the approximation to divergence presented here.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR8-riedel10a, title = {Inference by Minimizing Size, Divergence, or their Sum}, author = {Riedel, Sebastian and Smith, David and McCallum, Andrew}, booktitle = {Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence}, pages = {491--498}, year = {2010}, editor = {Grünwald, Peter and Spirtes, Peter}, volume = {R8}, series = {Proceedings of Machine Learning Research}, month = {08--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r8/main/assets/riedel10a/riedel10a.pdf}, url = {https://proceedings.mlr.press/r8/riedel10a.html}, abstract = {We speed up marginal inference by ignoring factors that do not significantly contribute to overall accuracy. In order to pick a suitable subset of factors to ignore, we propose three schemes: minimizing the number of model factors under a bound on the KL divergence between pruned and full models; minimizing the KL divergence under a bound on factor count; and minimizing the weighted sum of KL divergence and factor count. All three problems are solved using an approximation of the KL divergence than can be calculated in terms of marginals computed on a sim- ple seed graph. Applied to synthetic im- age denoising and to three different types of NLP parsing models, this technique performs marginal inference up to 11 times faster than loopy BP, with graph sizes reduced up to 98%—at comparable error in marginals and parsing accuracy. We also show that mini- mizing the weighted sum of divergence and size is substantially faster than minimizing either of the other objectives based on the approximation to divergence presented here.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Inference by Minimizing Size, Divergence, or their Sum %A Sebastian Riedel %A David Smith %A Andrew McCallum %B Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2010 %E Peter Grünwald %E Peter Spirtes %F pmlr-vR8-riedel10a %I PMLR %P 491--498 %U https://proceedings.mlr.press/r8/riedel10a.html %V R8 %X We speed up marginal inference by ignoring factors that do not significantly contribute to overall accuracy. In order to pick a suitable subset of factors to ignore, we propose three schemes: minimizing the number of model factors under a bound on the KL divergence between pruned and full models; minimizing the KL divergence under a bound on factor count; and minimizing the weighted sum of KL divergence and factor count. All three problems are solved using an approximation of the KL divergence than can be calculated in terms of marginals computed on a sim- ple seed graph. Applied to synthetic im- age denoising and to three different types of NLP parsing models, this technique performs marginal inference up to 11 times faster than loopy BP, with graph sizes reduced up to 98%—at comparable error in marginals and parsing accuracy. We also show that mini- mizing the weighted sum of divergence and size is substantially faster than minimizing either of the other objectives based on the approximation to divergence presented here. %Z Reissued by PMLR on 04 October 2026.
APA
Riedel, S., Smith, D. & McCallum, A.. (2010). Inference by Minimizing Size, Divergence, or their Sum. Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R8:491-498 Available from https://proceedings.mlr.press/r8/riedel10a.html. Reissued by PMLR on 04 October 2026.

Related Material