Generative Decoding of Visual Stimuli

Eleni Miliotou, Panagiotis Kyriakis, Jason D Hinman, Andrei Irimia, Paul Bogdan
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:24775-24784, 2023.

Abstract

Reconstructing natural images from fMRI recordings is a challenging task of great importance in neuroscience. The current architectures are bottlenecked because they fail to effectively capture the hierarchical processing of visual stimuli that takes place in the human brain. Motivated by that fact, we introduce a novel neural network architecture for the problem of neural decoding. Our architecture uses Hierarchical Variational Autoencoders (HVAEs) to learn meaningful representations of natural images and leverages their latent space hierarchy to learn voxel-to-image mappings. By mapping the early stages of the visual pathway to the first set of latent variables and the higher visual cortex areas to the deeper layers in the latent hierarchy, we are able to construct a latent variable neural decoding model that replicates the hierarchical visual information processing. Our model achieves better reconstructions compared to the state of the art and our ablation study indicates that the hierarchical structure of the latent space is responsible for that performance.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-miliotou23a, title = {Generative Decoding of Visual Stimuli}, author = {Miliotou, Eleni and Kyriakis, Panagiotis and Hinman, Jason D and Irimia, Andrei and Bogdan, Paul}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {24775--24784}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/miliotou23a/miliotou23a.pdf}, url = {https://proceedings.mlr.press/v202/miliotou23a.html}, abstract = {Reconstructing natural images from fMRI recordings is a challenging task of great importance in neuroscience. The current architectures are bottlenecked because they fail to effectively capture the hierarchical processing of visual stimuli that takes place in the human brain. Motivated by that fact, we introduce a novel neural network architecture for the problem of neural decoding. Our architecture uses Hierarchical Variational Autoencoders (HVAEs) to learn meaningful representations of natural images and leverages their latent space hierarchy to learn voxel-to-image mappings. By mapping the early stages of the visual pathway to the first set of latent variables and the higher visual cortex areas to the deeper layers in the latent hierarchy, we are able to construct a latent variable neural decoding model that replicates the hierarchical visual information processing. Our model achieves better reconstructions compared to the state of the art and our ablation study indicates that the hierarchical structure of the latent space is responsible for that performance.} }
Endnote
%0 Conference Paper %T Generative Decoding of Visual Stimuli %A Eleni Miliotou %A Panagiotis Kyriakis %A Jason D Hinman %A Andrei Irimia %A Paul Bogdan %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-miliotou23a %I PMLR %P 24775--24784 %U https://proceedings.mlr.press/v202/miliotou23a.html %V 202 %X Reconstructing natural images from fMRI recordings is a challenging task of great importance in neuroscience. The current architectures are bottlenecked because they fail to effectively capture the hierarchical processing of visual stimuli that takes place in the human brain. Motivated by that fact, we introduce a novel neural network architecture for the problem of neural decoding. Our architecture uses Hierarchical Variational Autoencoders (HVAEs) to learn meaningful representations of natural images and leverages their latent space hierarchy to learn voxel-to-image mappings. By mapping the early stages of the visual pathway to the first set of latent variables and the higher visual cortex areas to the deeper layers in the latent hierarchy, we are able to construct a latent variable neural decoding model that replicates the hierarchical visual information processing. Our model achieves better reconstructions compared to the state of the art and our ablation study indicates that the hierarchical structure of the latent space is responsible for that performance.
APA
Miliotou, E., Kyriakis, P., Hinman, J.D., Irimia, A. & Bogdan, P.. (2023). Generative Decoding of Visual Stimuli. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:24775-24784 Available from https://proceedings.mlr.press/v202/miliotou23a.html.

Related Material