PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction

Aaron Wenteler, Martina Occhetta, Nikhil Branson, Victor Curean, Magdalena Huebner, William Dee, William Connell, Siu Pui Chung, Alex Hawkins-Hooker, Yasha Ektefaie, César Miguel Valdez Córdova, Amaya Gallagher-Syed
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:66633-66677, 2025.

Abstract

In silico modeling of transcriptional responses to perturbations is crucial for advancing our understanding of cellular processes and disease mechanisms. We present PertEval-scFM, a standardized framework designed to evaluate models for perturbation effect prediction. We apply PertEval-scFM to benchmark zero-shot single-cell foundation model (scFM) embeddings against baseline models to assess whether these contextualized representations enhance perturbation effect prediction. Our results show that scFM embeddings offer limited improvement over simple baseline models in the zero-shot setting, particularly under distribution shift. Overall, this study provides a systematic evaluation of zero-shot scFM embeddings for perturbation effect prediction, highlighting the challenges of this task and the limitations of current-generation scFMs. Our findings underscore the need for specialized models and high-quality datasets that capture a broader range of cellular states. Source code and documentation can be found at: https://github.com/aaronwtr/PertEval.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-wenteler25a, title = {{P}ert{E}val-sc{FM}: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction}, author = {Wenteler, Aaron and Occhetta, Martina and Branson, Nikhil and Curean, Victor and Huebner, Magdalena and Dee, William and Connell, William and Chung, Siu Pui and Hawkins-Hooker, Alex and Ektefaie, Yasha and C\'{o}rdova, C\'{e}sar Miguel Valdez and Gallagher-Syed, Amaya}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {66633--66677}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/wenteler25a/wenteler25a.pdf}, url = {https://proceedings.mlr.press/v267/wenteler25a.html}, abstract = {In silico modeling of transcriptional responses to perturbations is crucial for advancing our understanding of cellular processes and disease mechanisms. We present PertEval-scFM, a standardized framework designed to evaluate models for perturbation effect prediction. We apply PertEval-scFM to benchmark zero-shot single-cell foundation model (scFM) embeddings against baseline models to assess whether these contextualized representations enhance perturbation effect prediction. Our results show that scFM embeddings offer limited improvement over simple baseline models in the zero-shot setting, particularly under distribution shift. Overall, this study provides a systematic evaluation of zero-shot scFM embeddings for perturbation effect prediction, highlighting the challenges of this task and the limitations of current-generation scFMs. Our findings underscore the need for specialized models and high-quality datasets that capture a broader range of cellular states. Source code and documentation can be found at: https://github.com/aaronwtr/PertEval.} }
Endnote
%0 Conference Paper %T PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction %A Aaron Wenteler %A Martina Occhetta %A Nikhil Branson %A Victor Curean %A Magdalena Huebner %A William Dee %A William Connell %A Siu Pui Chung %A Alex Hawkins-Hooker %A Yasha Ektefaie %A César Miguel Valdez Córdova %A Amaya Gallagher-Syed %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-wenteler25a %I PMLR %P 66633--66677 %U https://proceedings.mlr.press/v267/wenteler25a.html %V 267 %X In silico modeling of transcriptional responses to perturbations is crucial for advancing our understanding of cellular processes and disease mechanisms. We present PertEval-scFM, a standardized framework designed to evaluate models for perturbation effect prediction. We apply PertEval-scFM to benchmark zero-shot single-cell foundation model (scFM) embeddings against baseline models to assess whether these contextualized representations enhance perturbation effect prediction. Our results show that scFM embeddings offer limited improvement over simple baseline models in the zero-shot setting, particularly under distribution shift. Overall, this study provides a systematic evaluation of zero-shot scFM embeddings for perturbation effect prediction, highlighting the challenges of this task and the limitations of current-generation scFMs. Our findings underscore the need for specialized models and high-quality datasets that capture a broader range of cellular states. Source code and documentation can be found at: https://github.com/aaronwtr/PertEval.
APA
Wenteler, A., Occhetta, M., Branson, N., Curean, V., Huebner, M., Dee, W., Connell, W., Chung, S.P., Hawkins-Hooker, A., Ektefaie, Y., Córdova, C.M.V. & Gallagher-Syed, A.. (2025). PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:66633-66677 Available from https://proceedings.mlr.press/v267/wenteler25a.html.

Related Material