Unsupervised Learning of Structured Representation via Closed-Loop Transcription

Shengbang Tong, Xili Dai, Yubei Chen, Mingyang Li, ZENGYI LI, Brent Yi, Yann LeCun, Yi Ma
Conference on Parsimony and Learning, PMLR 234:440-457, 2024.

Abstract

This paper proposes an unsupervised method for learning a unified representation that serves both discriminative and generative purposes. While most existing unsupervised learning approaches focus on a representation for only one of these two goals, we show that a unified representation can enjoy the mutual benefits of having both. Such a representation is attainable by generalizing the recently proposed closed-loop transcription framework, known as CTRL, to the unsupervised setting. This entails solving a constrained maximin game over a rate reduction objective that expands features of all samples while compressing features of augmentations of each sample. Through this process, we see discriminative low-dimensional structures emerge in the resulting representations. Under comparable experimental conditions and network complexities, we demonstrate that these structured representations enable classification performance close to state-of-the-art unsupervised discriminative representations, and conditionally generated image quality significantly higher than that of state-of-the-art unsupervised generative models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v234-tong24a, title = {Unsupervised Learning of Structured Representation via Closed-Loop Transcription}, author = {Tong, Shengbang and Dai, Xili and Chen, Yubei and Li, Mingyang and LI, ZENGYI and Yi, Brent and LeCun, Yann and Ma, Yi}, booktitle = {Conference on Parsimony and Learning}, pages = {440--457}, year = {2024}, editor = {Chi, Yuejie and Dziugaite, Gintare Karolina and Qu, Qing and Wang, Atlas Wang and Zhu, Zhihui}, volume = {234}, series = {Proceedings of Machine Learning Research}, month = {03--06 Jan}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v234/tong24a/tong24a.pdf}, url = {https://proceedings.mlr.press/v234/tong24a.html}, abstract = {This paper proposes an unsupervised method for learning a unified representation that serves both discriminative and generative purposes. While most existing unsupervised learning approaches focus on a representation for only one of these two goals, we show that a unified representation can enjoy the mutual benefits of having both. Such a representation is attainable by generalizing the recently proposed closed-loop transcription framework, known as CTRL, to the unsupervised setting. This entails solving a constrained maximin game over a rate reduction objective that expands features of all samples while compressing features of augmentations of each sample. Through this process, we see discriminative low-dimensional structures emerge in the resulting representations. Under comparable experimental conditions and network complexities, we demonstrate that these structured representations enable classification performance close to state-of-the-art unsupervised discriminative representations, and conditionally generated image quality significantly higher than that of state-of-the-art unsupervised generative models.} }
Endnote
%0 Conference Paper %T Unsupervised Learning of Structured Representation via Closed-Loop Transcription %A Shengbang Tong %A Xili Dai %A Yubei Chen %A Mingyang Li %A ZENGYI LI %A Brent Yi %A Yann LeCun %A Yi Ma %B Conference on Parsimony and Learning %C Proceedings of Machine Learning Research %D 2024 %E Yuejie Chi %E Gintare Karolina Dziugaite %E Qing Qu %E Atlas Wang Wang %E Zhihui Zhu %F pmlr-v234-tong24a %I PMLR %P 440--457 %U https://proceedings.mlr.press/v234/tong24a.html %V 234 %X This paper proposes an unsupervised method for learning a unified representation that serves both discriminative and generative purposes. While most existing unsupervised learning approaches focus on a representation for only one of these two goals, we show that a unified representation can enjoy the mutual benefits of having both. Such a representation is attainable by generalizing the recently proposed closed-loop transcription framework, known as CTRL, to the unsupervised setting. This entails solving a constrained maximin game over a rate reduction objective that expands features of all samples while compressing features of augmentations of each sample. Through this process, we see discriminative low-dimensional structures emerge in the resulting representations. Under comparable experimental conditions and network complexities, we demonstrate that these structured representations enable classification performance close to state-of-the-art unsupervised discriminative representations, and conditionally generated image quality significantly higher than that of state-of-the-art unsupervised generative models.
APA
Tong, S., Dai, X., Chen, Y., Li, M., LI, Z., Yi, B., LeCun, Y. & Ma, Y.. (2024). Unsupervised Learning of Structured Representation via Closed-Loop Transcription. Conference on Parsimony and Learning, in Proceedings of Machine Learning Research 234:440-457 Available from https://proceedings.mlr.press/v234/tong24a.html.

Related Material