GeST: Towards Building A Generative Pretrained Transformer for Learning Cellular Spatial Context

Minsheng Hao, Nan Yan, Haiyang Bian, Yixin Chen, Jin Gu, Lei Wei, Xuegong Zhang
Proceedings of the 20th Machine Learning in Computational Biology meeting, PMLR 311:1-11, 2025.

Abstract

Learning spatial context of cells through pretraining on spatial transcriptomics (ST) data may empower us to decipher tissue organization and cellular interactions. Yet, transformer-based generative models often focus on modeling individual cells, overlooking the intricate spatial relationships within them. To address this limitation, we develop GeST, a deep transformer model pretrained by a novel spatially informed generation task: Predicting cellular expression profile of a given location based on the information from its neighboring cells. GeST integrates a specialized spatial attention mechanism for efficient pretraining, a flexible serialization strategy for sequentializing ST data, and a cell tokenization method for quantizing gene expression profiles. We pretrained GeST on large-scale ST datasets across multiple ST technologies, achieving superior performance in generating previously unseen spatial cell profiles, extracting spatial niche embeddings in a zero-shot manner, and annotating spatial regions. Furthermore, GeST can simulate gene expression changes in response to perturbations of cells within spatial context, closely matching existing experimental results. GeST offers a powerful generative pre-training framework for learning spatial contexts.

Cite this Paper

BibTeX
@InProceedings{pmlr-v311-hao25a, title = {GeST: Towards Building A Generative Pretrained Transformer for Learning Cellular Spatial Context}, author = {Hao, Minsheng and Yan, Nan and Bian, Haiyang and Chen, Yixin and Gu, Jin and Wei, Lei and Zhang, Xuegong}, booktitle = {Proceedings of the 20th Machine Learning in Computational Biology meeting}, pages = {1--11}, year = {2025}, editor = {Knowles, David A and Koo, Peter K}, volume = {311}, series = {Proceedings of Machine Learning Research}, month = {10--11 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v311/main/assets/hao25a/hao25a.pdf}, url = {https://proceedings.mlr.press/v311/hao25a.html}, abstract = {Learning spatial context of cells through pretraining on spatial transcriptomics (ST) data may empower us to decipher tissue organization and cellular interactions. Yet, transformer-based generative models often focus on modeling individual cells, overlooking the intricate spatial relationships within them. To address this limitation, we develop GeST, a deep transformer model pretrained by a novel spatially informed generation task: Predicting cellular expression profile of a given location based on the information from its neighboring cells. GeST integrates a specialized spatial attention mechanism for efficient pretraining, a flexible serialization strategy for sequentializing ST data, and a cell tokenization method for quantizing gene expression profiles. We pretrained GeST on large-scale ST datasets across multiple ST technologies, achieving superior performance in generating previously unseen spatial cell profiles, extracting spatial niche embeddings in a zero-shot manner, and annotating spatial regions. Furthermore, GeST can simulate gene expression changes in response to perturbations of cells within spatial context, closely matching existing experimental results. GeST offers a powerful generative pre-training framework for learning spatial contexts.} }
Endnote
%0 Conference Paper %T GeST: Towards Building A Generative Pretrained Transformer for Learning Cellular Spatial Context %A Minsheng Hao %A Nan Yan %A Haiyang Bian %A Yixin Chen %A Jin Gu %A Lei Wei %A Xuegong Zhang %B Proceedings of the 20th Machine Learning in Computational Biology meeting %C Proceedings of Machine Learning Research %D 2025 %E David A Knowles %E Peter K Koo %F pmlr-v311-hao25a %I PMLR %P 1--11 %U https://proceedings.mlr.press/v311/hao25a.html %V 311 %X Learning spatial context of cells through pretraining on spatial transcriptomics (ST) data may empower us to decipher tissue organization and cellular interactions. Yet, transformer-based generative models often focus on modeling individual cells, overlooking the intricate spatial relationships within them. To address this limitation, we develop GeST, a deep transformer model pretrained by a novel spatially informed generation task: Predicting cellular expression profile of a given location based on the information from its neighboring cells. GeST integrates a specialized spatial attention mechanism for efficient pretraining, a flexible serialization strategy for sequentializing ST data, and a cell tokenization method for quantizing gene expression profiles. We pretrained GeST on large-scale ST datasets across multiple ST technologies, achieving superior performance in generating previously unseen spatial cell profiles, extracting spatial niche embeddings in a zero-shot manner, and annotating spatial regions. Furthermore, GeST can simulate gene expression changes in response to perturbations of cells within spatial context, closely matching existing experimental results. GeST offers a powerful generative pre-training framework for learning spatial contexts.
APA
Hao, M., Yan, N., Bian, H., Chen, Y., Gu, J., Wei, L. & Zhang, X.. (2025). GeST: Towards Building A Generative Pretrained Transformer for Learning Cellular Spatial Context. Proceedings of the 20th Machine Learning in Computational Biology meeting, in Proceedings of Machine Learning Research 311:1-11 Available from https://proceedings.mlr.press/v311/hao25a.html.

Related Material