Structural Similarities Between Language Models and Neural Response Measurements

Jiaang Li, Antonia Karamolegkou, Yova Kementchedjhieva, Mostafa Abdou, Anders S\ogaard
Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neural Representations, PMLR 228:346-365, 2024.

Abstract

Large language models have complicated internal dynamics, but induce representations of words and phrases whose geometry we can study. Human language processing is also opaque, but neural response measurements can provide (noisy) recordings of activations during listening or reading, from which we can extract similar representations of words and phrases. Here we study the extent to which the geometries induced by these representations, share similarities in the context of brain decoding. We find that the larger neural language models get, the more their representations are structurally similar to neural response measurements from brain imaging.

Cite this Paper

BibTeX
@InProceedings{pmlr-v228-li24a, title = {Structural Similarities Between Language Models and Neural Response Measurements}, author = {Li, Jiaang and Karamolegkou, Antonia and Kementchedjhieva, Yova and Abdou, Mostafa and S\ogaard, Anders}, booktitle = {Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neural Representations}, pages = {346--365}, year = {2024}, editor = {Sanborn, Sophia and Shewmake, Christian and Azeglio, Simone and Miolane, Nina}, volume = {228}, series = {Proceedings of Machine Learning Research}, month = {16 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v228/main/assets/li24a/li24a.pdf}, url = {https://proceedings.mlr.press/v228/li24a.html}, abstract = {Large language models have complicated internal dynamics, but induce representations of words and phrases whose geometry we can study. Human language processing is also opaque, but neural response measurements can provide (noisy) recordings of activations during listening or reading, from which we can extract similar representations of words and phrases. Here we study the extent to which the geometries induced by these representations, share similarities in the context of brain decoding. We find that the larger neural language models get, the more their representations are structurally similar to neural response measurements from brain imaging.} }
Endnote
%0 Conference Paper %T Structural Similarities Between Language Models and Neural Response Measurements %A Jiaang Li %A Antonia Karamolegkou %A Yova Kementchedjhieva %A Mostafa Abdou %A Anders S\ogaard %B Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neural Representations %C Proceedings of Machine Learning Research %D 2024 %E Sophia Sanborn %E Christian Shewmake %E Simone Azeglio %E Nina Miolane %F pmlr-v228-li24a %I PMLR %P 346--365 %U https://proceedings.mlr.press/v228/li24a.html %V 228 %X Large language models have complicated internal dynamics, but induce representations of words and phrases whose geometry we can study. Human language processing is also opaque, but neural response measurements can provide (noisy) recordings of activations during listening or reading, from which we can extract similar representations of words and phrases. Here we study the extent to which the geometries induced by these representations, share similarities in the context of brain decoding. We find that the larger neural language models get, the more their representations are structurally similar to neural response measurements from brain imaging.
APA
Li, J., Karamolegkou, A., Kementchedjhieva, Y., Abdou, M. & S\ogaard, A.. (2024). Structural Similarities Between Language Models and Neural Response Measurements. Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neural Representations, in Proceedings of Machine Learning Research 228:346-365 Available from https://proceedings.mlr.press/v228/li24a.html.

Related Material