Attend and Decode: 4D fMRI Task State Decoding Using Attention Models

Sam Nguyen, Brenda Ng, Alan D. Kaplan, Priyadip Ray
Proceedings of the Machine Learning for Health NeurIPS Workshop, PMLR 136:267-279, 2020.

Abstract

Functional magnetic resonance imaging (fMRI) is a neuroimaging modality that captures the blood oxygen level in a subject’s brain while the subject either rests or performs a variety of functional tasks under different conditions. Given fMRI data, the problem of inferring the task, known as task state decoding, is challenging due to the high dimensionality (hundreds of million sampling points per datum) and complex spatio-temporal blood flow patterns inherent in the data. In this work, we propose to tackle the fMRI task state decoding problem by casting it as a 4D spatio-temporal classification problem. We present a novel architecture called Brain Attend and Decode (BAnD), that uses residual convolutional neural networks for spatial feature extraction and self-attention mechanisms for temporal modeling. We achieve significant performance gain compared to previous works on a 7-task benchmark from the large-scale Human Connectome Project-Young Adult (HCP-YA) dataset. We also investigate the transferability of BAnD’s extracted features on unseen HCP tasks, either by freezing the spatial feature extraction layers and retraining the temporal model, or finetuning the entire model. The pretrained features from BAnD are useful on similar tasks while finetuning them yields competitive results on unseen tasks/conditions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v136-nguyen20a, title = {Attend and Decode: 4D fMRI Task State Decoding Using Attention Models}, author = {Nguyen, Sam and Ng, Brenda and Kaplan, Alan D. and Ray, Priyadip}, booktitle = {Proceedings of the Machine Learning for Health NeurIPS Workshop}, pages = {267--279}, year = {2020}, editor = {Alsentzer, Emily and McDermott, Matthew B. A. and Falck, Fabian and Sarkar, Suproteem K. and Roy, Subhrajit and Hyland, Stephanie L.}, volume = {136}, series = {Proceedings of Machine Learning Research}, month = {11 Dec}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v136/nguyen20a/nguyen20a.pdf}, url = {https://proceedings.mlr.press/v136/nguyen20a.html}, abstract = {Functional magnetic resonance imaging (fMRI) is a neuroimaging modality that captures the blood oxygen level in a subject’s brain while the subject either rests or performs a variety of functional tasks under different conditions. Given fMRI data, the problem of inferring the task, known as task state decoding, is challenging due to the high dimensionality (hundreds of million sampling points per datum) and complex spatio-temporal blood flow patterns inherent in the data. In this work, we propose to tackle the fMRI task state decoding problem by casting it as a 4D spatio-temporal classification problem. We present a novel architecture called Brain Attend and Decode (BAnD), that uses residual convolutional neural networks for spatial feature extraction and self-attention mechanisms for temporal modeling. We achieve significant performance gain compared to previous works on a 7-task benchmark from the large-scale Human Connectome Project-Young Adult (HCP-YA) dataset. We also investigate the transferability of BAnD’s extracted features on unseen HCP tasks, either by freezing the spatial feature extraction layers and retraining the temporal model, or finetuning the entire model. The pretrained features from BAnD are useful on similar tasks while finetuning them yields competitive results on unseen tasks/conditions.} }
Endnote
%0 Conference Paper %T Attend and Decode: 4D fMRI Task State Decoding Using Attention Models %A Sam Nguyen %A Brenda Ng %A Alan D. Kaplan %A Priyadip Ray %B Proceedings of the Machine Learning for Health NeurIPS Workshop %C Proceedings of Machine Learning Research %D 2020 %E Emily Alsentzer %E Matthew B. A. McDermott %E Fabian Falck %E Suproteem K. Sarkar %E Subhrajit Roy %E Stephanie L. Hyland %F pmlr-v136-nguyen20a %I PMLR %P 267--279 %U https://proceedings.mlr.press/v136/nguyen20a.html %V 136 %X Functional magnetic resonance imaging (fMRI) is a neuroimaging modality that captures the blood oxygen level in a subject’s brain while the subject either rests or performs a variety of functional tasks under different conditions. Given fMRI data, the problem of inferring the task, known as task state decoding, is challenging due to the high dimensionality (hundreds of million sampling points per datum) and complex spatio-temporal blood flow patterns inherent in the data. In this work, we propose to tackle the fMRI task state decoding problem by casting it as a 4D spatio-temporal classification problem. We present a novel architecture called Brain Attend and Decode (BAnD), that uses residual convolutional neural networks for spatial feature extraction and self-attention mechanisms for temporal modeling. We achieve significant performance gain compared to previous works on a 7-task benchmark from the large-scale Human Connectome Project-Young Adult (HCP-YA) dataset. We also investigate the transferability of BAnD’s extracted features on unseen HCP tasks, either by freezing the spatial feature extraction layers and retraining the temporal model, or finetuning the entire model. The pretrained features from BAnD are useful on similar tasks while finetuning them yields competitive results on unseen tasks/conditions.
APA
Nguyen, S., Ng, B., Kaplan, A.D. & Ray, P.. (2020). Attend and Decode: 4D fMRI Task State Decoding Using Attention Models. Proceedings of the Machine Learning for Health NeurIPS Workshop, in Proceedings of Machine Learning Research 136:267-279 Available from https://proceedings.mlr.press/v136/nguyen20a.html.

Related Material