Unsupervised Learning of Categorical Structure

Matteo Alleman, Stefano Fusi
Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models, PMLR 285:100-114, 2024.

Abstract

Humans occasionally reason using logic and abstract categories, and yet most state of the art neural models use continuous distributed representations. These representations are impressive in their learning capabilities, but have proven difficult to interpret, or to compare to biological representations. But continuous representations can sometimes be interpreted symbolically, and a distributed code can seem to be constructed by composing abstract categories. We ask whether it is possible to detect and get back this structure, and we answer that it sort of is. The demixing problem is equivalent to factorizing the data into a continuous and a binary part $\mathbf{X}= \mathbf{W}\mathbf{S}^T$. After establishing some general facts and intuitions, we present two algorithms which work on low-rank or full-rank data, assess their reliability on extensive simulated data, and use them to interpret neural word embeddings where we expect some compositional structure. We hope this problem is interesting and that our simple algorithms provide a promising direction for solving it.

Cite this Paper

BibTeX
@InProceedings{pmlr-v285-alleman24a, title = {Unsupervised Learning of Categorical Structure}, author = {Alleman, Matteo and Fusi, Stefano}, booktitle = {Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models}, pages = {100--114}, year = {2024}, editor = {Fumero, Marco and Domine, Clementine and Lähner, Zorah and Crisostomi, Donato and Moschella, Luca and Stachenfeld, Kimberly}, volume = {285}, series = {Proceedings of Machine Learning Research}, month = {14 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v285/main/assets/alleman24a/alleman24a.pdf}, url = {https://proceedings.mlr.press/v285/alleman24a.html}, abstract = {Humans occasionally reason using logic and abstract categories, and yet most state of the art neural models use continuous distributed representations. These representations are impressive in their learning capabilities, but have proven difficult to interpret, or to compare to biological representations. But continuous representations can sometimes be interpreted symbolically, and a distributed code can seem to be constructed by composing abstract categories. We ask whether it is possible to detect and get back this structure, and we answer that it sort of is. The demixing problem is equivalent to factorizing the data into a continuous and a binary part $\mathbf{X}= \mathbf{W}\mathbf{S}^T$. After establishing some general facts and intuitions, we present two algorithms which work on low-rank or full-rank data, assess their reliability on extensive simulated data, and use them to interpret neural word embeddings where we expect some compositional structure. We hope this problem is interesting and that our simple algorithms provide a promising direction for solving it.} }
Endnote
%0 Conference Paper %T Unsupervised Learning of Categorical Structure %A Matteo Alleman %A Stefano Fusi %B Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models %C Proceedings of Machine Learning Research %D 2024 %E Marco Fumero %E Clementine Domine %E Zorah Lähner %E Donato Crisostomi %E Luca Moschella %E Kimberly Stachenfeld %F pmlr-v285-alleman24a %I PMLR %P 100--114 %U https://proceedings.mlr.press/v285/alleman24a.html %V 285 %X Humans occasionally reason using logic and abstract categories, and yet most state of the art neural models use continuous distributed representations. These representations are impressive in their learning capabilities, but have proven difficult to interpret, or to compare to biological representations. But continuous representations can sometimes be interpreted symbolically, and a distributed code can seem to be constructed by composing abstract categories. We ask whether it is possible to detect and get back this structure, and we answer that it sort of is. The demixing problem is equivalent to factorizing the data into a continuous and a binary part $\mathbf{X}= \mathbf{W}\mathbf{S}^T$. After establishing some general facts and intuitions, we present two algorithms which work on low-rank or full-rank data, assess their reliability on extensive simulated data, and use them to interpret neural word embeddings where we expect some compositional structure. We hope this problem is interesting and that our simple algorithms provide a promising direction for solving it.
APA
Alleman, M. & Fusi, S.. (2024). Unsupervised Learning of Categorical Structure. Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models, in Proceedings of Machine Learning Research 285:100-114 Available from https://proceedings.mlr.press/v285/alleman24a.html.

Related Material