Multimodal Scaling Laws for Task & Data-Optimized Models of Visual Cortex

Abdulkadir Gokce, Yingtian Tang, Martin Schrimpf
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:35514-35571, 2026.

Abstract

Task-optimized neural networks are the leading in-silico models of sensory cortex, yet the field lacks a unified understanding of which modeling choices drive improved brain alignment. Prior NeuroAI work is fragmented across datasets and modalities, making it difficult to determine robust scaling trends. Here, we systematically investigate the scaling laws of model-to-brain alignment across 8 neural datasets (spanning electrophysiology, fMRI, EEG, and MEG) and over 600 models with diverse architectures and pretraining configurations. We report three scaling trends: (1) Pretraining saturation: Alignment improves with pretraining compute and data scale but saturates across all recording modalities. (2) Complementary fine-tuning: Hybrid task & neural data optimization yields consistent improvements in alignment that generalize across datasets and modalities. (3) Mapping scaling: Increasing the number of neural samples to fit model-to-brain mappings yields log-linear gains with the largest impact on alignment. Finally, we propose a novel subject-shared cross-attention mapping which drastically reduces parameter count and improves alignment. Taken together, these results establish multimodal scaling laws that guide resource allocation for next-generation brain models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-gokce26a, title = {Multimodal Scaling Laws for Task & Data-Optimized Models of Visual Cortex}, author = {Gokce, Abdulkadir and Tang, Yingtian and Schrimpf, Martin}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {35514--35571}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/gokce26a/gokce26a.pdf}, url = {https://proceedings.mlr.press/v306/gokce26a.html}, abstract = {Task-optimized neural networks are the leading in-silico models of sensory cortex, yet the field lacks a unified understanding of which modeling choices drive improved brain alignment. Prior NeuroAI work is fragmented across datasets and modalities, making it difficult to determine robust scaling trends. Here, we systematically investigate the scaling laws of model-to-brain alignment across 8 neural datasets (spanning electrophysiology, fMRI, EEG, and MEG) and over 600 models with diverse architectures and pretraining configurations. We report three scaling trends: (1) Pretraining saturation: Alignment improves with pretraining compute and data scale but saturates across all recording modalities. (2) Complementary fine-tuning: Hybrid task & neural data optimization yields consistent improvements in alignment that generalize across datasets and modalities. (3) Mapping scaling: Increasing the number of neural samples to fit model-to-brain mappings yields log-linear gains with the largest impact on alignment. Finally, we propose a novel subject-shared cross-attention mapping which drastically reduces parameter count and improves alignment. Taken together, these results establish multimodal scaling laws that guide resource allocation for next-generation brain models.} }
Endnote
%0 Conference Paper %T Multimodal Scaling Laws for Task & Data-Optimized Models of Visual Cortex %A Abdulkadir Gokce %A Yingtian Tang %A Martin Schrimpf %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-gokce26a %I PMLR %P 35514--35571 %U https://proceedings.mlr.press/v306/gokce26a.html %V 306 %X Task-optimized neural networks are the leading in-silico models of sensory cortex, yet the field lacks a unified understanding of which modeling choices drive improved brain alignment. Prior NeuroAI work is fragmented across datasets and modalities, making it difficult to determine robust scaling trends. Here, we systematically investigate the scaling laws of model-to-brain alignment across 8 neural datasets (spanning electrophysiology, fMRI, EEG, and MEG) and over 600 models with diverse architectures and pretraining configurations. We report three scaling trends: (1) Pretraining saturation: Alignment improves with pretraining compute and data scale but saturates across all recording modalities. (2) Complementary fine-tuning: Hybrid task & neural data optimization yields consistent improvements in alignment that generalize across datasets and modalities. (3) Mapping scaling: Increasing the number of neural samples to fit model-to-brain mappings yields log-linear gains with the largest impact on alignment. Finally, we propose a novel subject-shared cross-attention mapping which drastically reduces parameter count and improves alignment. Taken together, these results establish multimodal scaling laws that guide resource allocation for next-generation brain models.
APA
Gokce, A., Tang, Y. & Schrimpf, M.. (2026). Multimodal Scaling Laws for Task & Data-Optimized Models of Visual Cortex. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:35514-35571 Available from https://proceedings.mlr.press/v306/gokce26a.html.

Related Material