Needles in the Haystack: Addressing Signal Dilution Improves scRNA-seq Perturbation Response Modeling and Evaluation

Gabriel Mateo Mejia, Henry E Miller, Francis J.A. Leblanc, Bo Wang, Brendan Swain, Lucas Paulo De Lima Camillo
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:87494-87511, 2026.

Abstract

Recent benchmarks reveal that single-cell perturbation response models are often outperformed by simply predicting the dataset mean. Through large-scale in silico simulations, together with analyses of two real-world perturbation datasets, we trace this anomaly to a metric artifact: unweighted error metrics systematically reward mean predictions when perturbation effects are sparse. To address this limitation, we introduce differentially expressed gene (DEG)-aware metrics—weighted mean-squared error (WMSE) and weighted delta $R^{2}$ ($R^{2}_{w}(\Delta)$)—that sensitively measure error in niche, perturbation-specific signals. We further propose explicit negative and positive performance baselines to calibrate these metrics. Under this framework, the mean baseline sinks to null performance, while genuinely informative predictors are correctly rewarded. Finally, we show that using WMSE as a training objective reduces mode collapse and improves predictive performance across multiple model architectures.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-mejia26a, title = {Needles in the Haystack: Addressing Signal Dilution Improves sc{RNA}-seq Perturbation Response Modeling and Evaluation}, author = {Mejia, Gabriel Mateo and Miller, Henry E and Leblanc, Francis J.A. and Wang, Bo and Swain, Brendan and De Lima Camillo, Lucas Paulo}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {87494--87511}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/mejia26a/mejia26a.pdf}, url = {https://proceedings.mlr.press/v306/mejia26a.html}, abstract = {Recent benchmarks reveal that single-cell perturbation response models are often outperformed by simply predicting the dataset mean. Through large-scale in silico simulations, together with analyses of two real-world perturbation datasets, we trace this anomaly to a metric artifact: unweighted error metrics systematically reward mean predictions when perturbation effects are sparse. To address this limitation, we introduce differentially expressed gene (DEG)-aware metrics—weighted mean-squared error (WMSE) and weighted delta $R^{2}$ ($R^{2}_{w}(\Delta)$)—that sensitively measure error in niche, perturbation-specific signals. We further propose explicit negative and positive performance baselines to calibrate these metrics. Under this framework, the mean baseline sinks to null performance, while genuinely informative predictors are correctly rewarded. Finally, we show that using WMSE as a training objective reduces mode collapse and improves predictive performance across multiple model architectures.} }
Endnote
%0 Conference Paper %T Needles in the Haystack: Addressing Signal Dilution Improves scRNA-seq Perturbation Response Modeling and Evaluation %A Gabriel Mateo Mejia %A Henry E Miller %A Francis J.A. Leblanc %A Bo Wang %A Brendan Swain %A Lucas Paulo De Lima Camillo %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-mejia26a %I PMLR %P 87494--87511 %U https://proceedings.mlr.press/v306/mejia26a.html %V 306 %X Recent benchmarks reveal that single-cell perturbation response models are often outperformed by simply predicting the dataset mean. Through large-scale in silico simulations, together with analyses of two real-world perturbation datasets, we trace this anomaly to a metric artifact: unweighted error metrics systematically reward mean predictions when perturbation effects are sparse. To address this limitation, we introduce differentially expressed gene (DEG)-aware metrics—weighted mean-squared error (WMSE) and weighted delta $R^{2}$ ($R^{2}_{w}(\Delta)$)—that sensitively measure error in niche, perturbation-specific signals. We further propose explicit negative and positive performance baselines to calibrate these metrics. Under this framework, the mean baseline sinks to null performance, while genuinely informative predictors are correctly rewarded. Finally, we show that using WMSE as a training objective reduces mode collapse and improves predictive performance across multiple model architectures.
APA
Mejia, G.M., Miller, H.E., Leblanc, F.J., Wang, B., Swain, B. & De Lima Camillo, L.P.. (2026). Needles in the Haystack: Addressing Signal Dilution Improves scRNA-seq Perturbation Response Modeling and Evaluation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:87494-87511 Available from https://proceedings.mlr.press/v306/mejia26a.html.

Related Material