Investigating the role of modality and training objective on representational alignment between transformers and the brain

Hyewon Willow Han, Ruchira Dhar, Qingqing Yang, Maryam Hoseini Behbahani, María Alejandra Martínez Ortiz, Tolulope Samuel Oladele, Diana C Dima, Hsin-Hung Li, Anders Søgaard, Yalda Mohsenzadeh
Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models, PMLR 285:40-54, 2024.

Abstract

The remarkable performance of transformer models in both linguistic and real-world reasoning tasks coupled with their ubiquitous use has prompted much research on their alignment with brain activations. However, there remain some unanswered questions: what aspects of these models lead to representational alignment- the input modality or the training objective? Moreover, is the alignment limited to modality-specialized brain regions, or can representations align with brain regions involved in higher cognitive functions? To address this, we analyze the representations of different transformer architectures, including text-based and vision-based language models, and compare them with neural representations across multiple brain regions obtained during a visual processing task. Our findings reveal that both training data modality and training objective are important in determining alignment, and that models align with neural representations within and beyond the modality-specific regions. Additionally, the training modality and objectives seem to have an impact on alignment quality as we progress through the layers, suggesting that multimodal data along with a predictive processing objective may confer superior representational capabilities compared to other training objectives.

Cite this Paper

BibTeX
@InProceedings{pmlr-v285-han24a, title = {Investigating the role of modality and training objective on representational alignment between transformers and the brain}, author = {Han, Hyewon Willow and Dhar, Ruchira and Yang, Qingqing and Behbahani, Maryam Hoseini and Ortiz, Mar{\'i}a Alejandra Mart{\'i}nez and Oladele, Tolulope Samuel and Dima, Diana C and Li, Hsin-Hung and S{\o}gaard, Anders and Mohsenzadeh, Yalda}, booktitle = {Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models}, pages = {40--54}, year = {2024}, editor = {Fumero, Marco and Domine, Clementine and Lähner, Zorah and Crisostomi, Donato and Moschella, Luca and Stachenfeld, Kimberly}, volume = {285}, series = {Proceedings of Machine Learning Research}, month = {14 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v285/main/assets/han24a/han24a.pdf}, url = {https://proceedings.mlr.press/v285/han24a.html}, abstract = {The remarkable performance of transformer models in both linguistic and real-world reasoning tasks coupled with their ubiquitous use has prompted much research on their alignment with brain activations. However, there remain some unanswered questions: what aspects of these models lead to representational alignment- the input modality or the training objective? Moreover, is the alignment limited to modality-specialized brain regions, or can representations align with brain regions involved in higher cognitive functions? To address this, we analyze the representations of different transformer architectures, including text-based and vision-based language models, and compare them with neural representations across multiple brain regions obtained during a visual processing task. Our findings reveal that both training data modality and training objective are important in determining alignment, and that models align with neural representations within and beyond the modality-specific regions. Additionally, the training modality and objectives seem to have an impact on alignment quality as we progress through the layers, suggesting that multimodal data along with a predictive processing objective may confer superior representational capabilities compared to other training objectives.} }
Endnote
%0 Conference Paper %T Investigating the role of modality and training objective on representational alignment between transformers and the brain %A Hyewon Willow Han %A Ruchira Dhar %A Qingqing Yang %A Maryam Hoseini Behbahani %A María Alejandra Martínez Ortiz %A Tolulope Samuel Oladele %A Diana C Dima %A Hsin-Hung Li %A Anders Søgaard %A Yalda Mohsenzadeh %B Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models %C Proceedings of Machine Learning Research %D 2024 %E Marco Fumero %E Clementine Domine %E Zorah Lähner %E Donato Crisostomi %E Luca Moschella %E Kimberly Stachenfeld %F pmlr-v285-han24a %I PMLR %P 40--54 %U https://proceedings.mlr.press/v285/han24a.html %V 285 %X The remarkable performance of transformer models in both linguistic and real-world reasoning tasks coupled with their ubiquitous use has prompted much research on their alignment with brain activations. However, there remain some unanswered questions: what aspects of these models lead to representational alignment- the input modality or the training objective? Moreover, is the alignment limited to modality-specialized brain regions, or can representations align with brain regions involved in higher cognitive functions? To address this, we analyze the representations of different transformer architectures, including text-based and vision-based language models, and compare them with neural representations across multiple brain regions obtained during a visual processing task. Our findings reveal that both training data modality and training objective are important in determining alignment, and that models align with neural representations within and beyond the modality-specific regions. Additionally, the training modality and objectives seem to have an impact on alignment quality as we progress through the layers, suggesting that multimodal data along with a predictive processing objective may confer superior representational capabilities compared to other training objectives.
APA
Han, H.W., Dhar, R., Yang, Q., Behbahani, M.H., Ortiz, M.A.M., Oladele, T.S., Dima, D.C., Li, H., Søgaard, A. & Mohsenzadeh, Y.. (2024). Investigating the role of modality and training objective on representational alignment between transformers and the brain. Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models, in Proceedings of Machine Learning Research 285:40-54 Available from https://proceedings.mlr.press/v285/han24a.html.

Related Material