MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

Paul Steven Scotti, Mihir Tripathy, Cesar Torrico, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A. Norman, Tanishq Mathew Abraham
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:44038-44059, 2024.

Abstract

Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility. All code is available on Github: https://github.com/MedARC-AI/MindEyeV2

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-scotti24a, title = {{M}ind{E}ye2: Shared-Subject Models Enable f{MRI}-To-Image With 1 Hour of Data}, author = {Scotti, Paul Steven and Tripathy, Mihir and Torrico, Cesar and Kneeland, Reese and Chen, Tong and Narang, Ashutosh and Santhirasegaran, Charan and Xu, Jonathan and Naselaris, Thomas and Norman, Kenneth A. and Abraham, Tanishq Mathew}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {44038--44059}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/scotti24a/scotti24a.pdf}, url = {https://proceedings.mlr.press/v235/scotti24a.html}, abstract = {Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility. All code is available on Github: https://github.com/MedARC-AI/MindEyeV2} }
Endnote
%0 Conference Paper %T MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data %A Paul Steven Scotti %A Mihir Tripathy %A Cesar Torrico %A Reese Kneeland %A Tong Chen %A Ashutosh Narang %A Charan Santhirasegaran %A Jonathan Xu %A Thomas Naselaris %A Kenneth A. Norman %A Tanishq Mathew Abraham %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-scotti24a %I PMLR %P 44038--44059 %U https://proceedings.mlr.press/v235/scotti24a.html %V 235 %X Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility. All code is available on Github: https://github.com/MedARC-AI/MindEyeV2
APA
Scotti, P.S., Tripathy, M., Torrico, C., Kneeland, R., Chen, T., Narang, A., Santhirasegaran, C., Xu, J., Naselaris, T., Norman, K.A. & Abraham, T.M.. (2024). MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:44038-44059 Available from https://proceedings.mlr.press/v235/scotti24a.html.

Related Material