Modeling Documents with Deep Boltzmann Machines

Nitish Srivastava, Ruslan Salakhutdinov, Geoffrey Hinton
Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence, PMLR R11:223-231, 2013.

Abstract

We introduce a type of Deep Boltzmann Ma- chine (DBM) that is suitable for extracting distributed semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of train- ing a DBM with judicious parameter tying. This enables an efficient pretraining algo- rithm and a state initialization scheme for fast inference. The model can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR11-srivastava13a, title = {Modeling Documents with Deep {B}oltzmann Machines}, author = {Srivastava, Nitish and Salakhutdinov, Ruslan and Hinton, Geoffrey}, booktitle = {Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence}, pages = {223--231}, year = {2013}, editor = {Nicholson, Ann and Smyth, Padhraic}, volume = {R11}, series = {Proceedings of Machine Learning Research}, month = {12--14 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r11/main/assets/srivastava13a/srivastava13a.pdf}, url = {https://proceedings.mlr.press/r11/srivastava13a.html}, abstract = {We introduce a type of Deep Boltzmann Ma- chine (DBM) that is suitable for extracting distributed semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of train- ing a DBM with judicious parameter tying. This enables an efficient pretraining algo- rithm and a state initialization scheme for fast inference. The model can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Modeling Documents with Deep Boltzmann Machines %A Nitish Srivastava %A Ruslan Salakhutdinov %A Geoffrey Hinton %B Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2013 %E Ann Nicholson %E Padhraic Smyth %F pmlr-vR11-srivastava13a %I PMLR %P 223--231 %U https://proceedings.mlr.press/r11/srivastava13a.html %V R11 %X We introduce a type of Deep Boltzmann Ma- chine (DBM) that is suitable for extracting distributed semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of train- ing a DBM with judicious parameter tying. This enables an efficient pretraining algo- rithm and a state initialization scheme for fast inference. The model can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks. %Z Reissued by PMLR on 04 October 2026.
APA
Srivastava, N., Salakhutdinov, R. & Hinton, G.. (2013). Modeling Documents with Deep Boltzmann Machines. Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R11:223-231 Available from https://proceedings.mlr.press/r11/srivastava13a.html. Reissued by PMLR on 04 October 2026.

Related Material