MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-text Decoding

Weikang Qiu, Zheng Huang, Haoyu Hu, Aosong Feng, Yujun Yan, Zhitao Ying
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:50572-50593, 2025.

Abstract

Decoding functional magnetic resonance imaging (fMRI) signals into text has been a key challenge in the neuroscience community, with the potential to advance brain-computer interfaces and uncover deeper insights into brain mechanisms. However, existing approaches often struggle with suboptimal predictive performance, limited task variety, and poor generalization across subjects. In response to this, we propose MindLLM, a model designed for subject-agnostic and versatile fMRI-to-text decoding. MindLLM consists of an fMRI encoder and an off-the-shelf LLM. The fMRI encoder employs a neuroscience-informed attention mechanism, which is capable of accommodating subjects with varying input shapes and thus achieves high-performance subject-agnostic decoding. Moreover, we introduce Brain Instruction Tuning (BIT), a novel approach that enhances the model’s ability to capture diverse semantic representations from fMRI signals, facilitating more versatile decoding. We evaluate MindLLM on comprehensive fMRI-to-text benchmarks. Results demonstrate that our model outperforms the baselines, improving downstream tasks by $12.0%$, unseen subject generalization by $24.5%$, and novel task adaptation by $25.0%$. Furthermore, the attention patterns in MindLLM provide interpretable insights into its decision-making process.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-qiu25e, title = {{M}ind{LLM}: A Subject-Agnostic and Versatile Model for f{MRI}-to-text Decoding}, author = {Qiu, Weikang and Huang, Zheng and Hu, Haoyu and Feng, Aosong and Yan, Yujun and Ying, Zhitao}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {50572--50593}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/qiu25e/qiu25e.pdf}, url = {https://proceedings.mlr.press/v267/qiu25e.html}, abstract = {Decoding functional magnetic resonance imaging (fMRI) signals into text has been a key challenge in the neuroscience community, with the potential to advance brain-computer interfaces and uncover deeper insights into brain mechanisms. However, existing approaches often struggle with suboptimal predictive performance, limited task variety, and poor generalization across subjects. In response to this, we propose MindLLM, a model designed for subject-agnostic and versatile fMRI-to-text decoding. MindLLM consists of an fMRI encoder and an off-the-shelf LLM. The fMRI encoder employs a neuroscience-informed attention mechanism, which is capable of accommodating subjects with varying input shapes and thus achieves high-performance subject-agnostic decoding. Moreover, we introduce Brain Instruction Tuning (BIT), a novel approach that enhances the model’s ability to capture diverse semantic representations from fMRI signals, facilitating more versatile decoding. We evaluate MindLLM on comprehensive fMRI-to-text benchmarks. Results demonstrate that our model outperforms the baselines, improving downstream tasks by $12.0%$, unseen subject generalization by $24.5%$, and novel task adaptation by $25.0%$. Furthermore, the attention patterns in MindLLM provide interpretable insights into its decision-making process.} }
Endnote
%0 Conference Paper %T MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-text Decoding %A Weikang Qiu %A Zheng Huang %A Haoyu Hu %A Aosong Feng %A Yujun Yan %A Zhitao Ying %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-qiu25e %I PMLR %P 50572--50593 %U https://proceedings.mlr.press/v267/qiu25e.html %V 267 %X Decoding functional magnetic resonance imaging (fMRI) signals into text has been a key challenge in the neuroscience community, with the potential to advance brain-computer interfaces and uncover deeper insights into brain mechanisms. However, existing approaches often struggle with suboptimal predictive performance, limited task variety, and poor generalization across subjects. In response to this, we propose MindLLM, a model designed for subject-agnostic and versatile fMRI-to-text decoding. MindLLM consists of an fMRI encoder and an off-the-shelf LLM. The fMRI encoder employs a neuroscience-informed attention mechanism, which is capable of accommodating subjects with varying input shapes and thus achieves high-performance subject-agnostic decoding. Moreover, we introduce Brain Instruction Tuning (BIT), a novel approach that enhances the model’s ability to capture diverse semantic representations from fMRI signals, facilitating more versatile decoding. We evaluate MindLLM on comprehensive fMRI-to-text benchmarks. Results demonstrate that our model outperforms the baselines, improving downstream tasks by $12.0%$, unseen subject generalization by $24.5%$, and novel task adaptation by $25.0%$. Furthermore, the attention patterns in MindLLM provide interpretable insights into its decision-making process.
APA
Qiu, W., Huang, Z., Hu, H., Feng, A., Yan, Y. & Ying, Z.. (2025). MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-text Decoding. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:50572-50593 Available from https://proceedings.mlr.press/v267/qiu25e.html.

Related Material