Factorized Multi-Modal Topic Model

Seppo Virtanen, Yangqing Jia, Arto Klami, Trevor Darrell
Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence, PMLR R10:842-850, 2012.

Abstract

Multi-modal data collections, such as corpora of paired images and text snippets, require analysis methods beyond single-view component and topic models. For continuous observations the current dominant approach is based on extensions of canonical correlation analysis, factorizing the variation into components shared by the different modalities and those private to each of them. For count data, multiple variants of topic models attempting to tie the modalities together have been presented. All of these, however, lack the ability to learn components private to one modality, and consequently will try to force dependencies even between minimally correlating modalities. In this work we combine the two approaches by presenting a novel HDP-based topic model that automatically learns both shared and private topics. The model is shown to be especially useful for querying the contents of one domain given samples of the other.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR10-virtanen12a, title = {Factorized Multi-Modal Topic Model}, author = {Virtanen, Seppo and Jia, Yangqing and Klami, Arto and Darrell, Trevor}, booktitle = {Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence}, pages = {842--850}, year = {2012}, editor = {de Freitas, Nando and Murphy, Kevin}, volume = {R10}, series = {Proceedings of Machine Learning Research}, month = {14--18 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r10/main/assets/virtanen12a/virtanen12a.pdf}, url = {https://proceedings.mlr.press/r10/virtanen12a.html}, abstract = {Multi-modal data collections, such as corpora of paired images and text snippets, require analysis methods beyond single-view component and topic models. For continuous observations the current dominant approach is based on extensions of canonical correlation analysis, factorizing the variation into components shared by the different modalities and those private to each of them. For count data, multiple variants of topic models attempting to tie the modalities together have been presented. All of these, however, lack the ability to learn components private to one modality, and consequently will try to force dependencies even between minimally correlating modalities. In this work we combine the two approaches by presenting a novel HDP-based topic model that automatically learns both shared and private topics. The model is shown to be especially useful for querying the contents of one domain given samples of the other.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Factorized Multi-Modal Topic Model %A Seppo Virtanen %A Yangqing Jia %A Arto Klami %A Trevor Darrell %B Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2012 %E Nando de Freitas %E Kevin Murphy %F pmlr-vR10-virtanen12a %I PMLR %P 842--850 %U https://proceedings.mlr.press/r10/virtanen12a.html %V R10 %X Multi-modal data collections, such as corpora of paired images and text snippets, require analysis methods beyond single-view component and topic models. For continuous observations the current dominant approach is based on extensions of canonical correlation analysis, factorizing the variation into components shared by the different modalities and those private to each of them. For count data, multiple variants of topic models attempting to tie the modalities together have been presented. All of these, however, lack the ability to learn components private to one modality, and consequently will try to force dependencies even between minimally correlating modalities. In this work we combine the two approaches by presenting a novel HDP-based topic model that automatically learns both shared and private topics. The model is shown to be especially useful for querying the contents of one domain given samples of the other. %Z Reissued by PMLR on 04 October 2026.
APA
Virtanen, S., Jia, Y., Klami, A. & Darrell, T.. (2012). Factorized Multi-Modal Topic Model. Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R10:842-850 Available from https://proceedings.mlr.press/r10/virtanen12a.html. Reissued by PMLR on 04 October 2026.

Related Material