On Smoothing and Inference for Topic Models

Arthur Asuncion, Max Welling, Padhraic Smyth, Yee Whye Teh
Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, PMLR R7:19-26, 2009.

Abstract

Latent Dirichlet analysis, or topic modeling, is a flexible latent variable framework for modeling high-dimensional sparse count data. Various learning algorithms have been developed in recent years, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation, and this variety motivates the need for careful empirical comparisons. In this paper, we highlight the close connections between these approaches. We find that the main differences are attributable to the amount of smoothing applied to the counts. When the hyperparameters are optimized, the differences in performance among the algorithms diminish significantly. The ability of these algorithms to achieve solutions of comparable accuracy gives us the freedom to select computationally efficient approaches. Using the insights gained from this comparative study, we show how accurate topic models can be learned in several seconds on text corpora with thousands of documents.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR7-asuncion09a, title = {On Smoothing and Inference for Topic Models}, author = {Asuncion, Arthur and Welling, Max and Smyth, Padhraic and Teh, Yee Whye}, booktitle = {Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence}, pages = {19--26}, year = {2009}, editor = {Bilmes, Jeff and Ng, Andrew Y.}, volume = {R7}, series = {Proceedings of Machine Learning Research}, month = {18--21 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r7/main/assets/asuncion09a/asuncion09a.pdf}, url = {https://proceedings.mlr.press/r7/asuncion09a.html}, abstract = {Latent Dirichlet analysis, or topic modeling, is a flexible latent variable framework for modeling high-dimensional sparse count data. Various learning algorithms have been developed in recent years, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation, and this variety motivates the need for careful empirical comparisons. In this paper, we highlight the close connections between these approaches. We find that the main differences are attributable to the amount of smoothing applied to the counts. When the hyperparameters are optimized, the differences in performance among the algorithms diminish significantly. The ability of these algorithms to achieve solutions of comparable accuracy gives us the freedom to select computationally efficient approaches. Using the insights gained from this comparative study, we show how accurate topic models can be learned in several seconds on text corpora with thousands of documents.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T On Smoothing and Inference for Topic Models %A Arthur Asuncion %A Max Welling %A Padhraic Smyth %A Yee Whye Teh %B Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2009 %E Jeff Bilmes %E Andrew Y. Ng %F pmlr-vR7-asuncion09a %I PMLR %P 19--26 %U https://proceedings.mlr.press/r7/asuncion09a.html %V R7 %X Latent Dirichlet analysis, or topic modeling, is a flexible latent variable framework for modeling high-dimensional sparse count data. Various learning algorithms have been developed in recent years, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation, and this variety motivates the need for careful empirical comparisons. In this paper, we highlight the close connections between these approaches. We find that the main differences are attributable to the amount of smoothing applied to the counts. When the hyperparameters are optimized, the differences in performance among the algorithms diminish significantly. The ability of these algorithms to achieve solutions of comparable accuracy gives us the freedom to select computationally efficient approaches. Using the insights gained from this comparative study, we show how accurate topic models can be learned in several seconds on text corpora with thousands of documents. %Z Reissued by PMLR on 04 October 2026.
APA
Asuncion, A., Welling, M., Smyth, P. & Teh, Y.W.. (2009). On Smoothing and Inference for Topic Models. Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R7:19-26 Available from https://proceedings.mlr.press/r7/asuncion09a.html. Reissued by PMLR on 04 October 2026.

Related Material