When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs

Bogdan Zagribelnyy, Ivan Ilin, Maksim Kuznetsov, Nikita Bondarev, Roman Schutski, Thomas Macdougall, Rim Shayakhmetov, Zulfat Miftahutdinov, Mikolaj Mizera, Vladimir Aladinskiy, Alex Aliper, Alex Zhavoronkov
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:152836-152919, 2026.

Abstract

Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evaluation of retrosynthesis performance remains limited. Existing benchmarks and metrics typically rely on published synthetic procedures and Top-K accuracy based on single ground-truth, which does not capture the open-ended nature of real-world synthesis planning. We propose a new benchmarking framework for single-step retrosynthesis that evaluates both general-purpose and chemistry-specialized LLMs using ChemCensor, a novel metric for chemical plausibility. By emphasizing plausibility over exact match, this approach better aligns with human synthesis planning practices. We also introduce CREED, a novel dataset comprising millions of ChemCensor-validated reaction records for LLM training, and use it to train a model that improves over the LLM baselines under this benchmark.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zagribelnyy26a, title = {When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for {LLM}s}, author = {Zagribelnyy, Bogdan and Ilin, Ivan and Kuznetsov, Maksim and Bondarev, Nikita and Schutski, Roman and Macdougall, Thomas and Shayakhmetov, Rim and Miftahutdinov, Zulfat and Mizera, Mikolaj and Aladinskiy, Vladimir and Aliper, Alex and Zhavoronkov, Alex}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {152836--152919}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zagribelnyy26a/zagribelnyy26a.pdf}, url = {https://proceedings.mlr.press/v306/zagribelnyy26a.html}, abstract = {Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evaluation of retrosynthesis performance remains limited. Existing benchmarks and metrics typically rely on published synthetic procedures and Top-K accuracy based on single ground-truth, which does not capture the open-ended nature of real-world synthesis planning. We propose a new benchmarking framework for single-step retrosynthesis that evaluates both general-purpose and chemistry-specialized LLMs using ChemCensor, a novel metric for chemical plausibility. By emphasizing plausibility over exact match, this approach better aligns with human synthesis planning practices. We also introduce CREED, a novel dataset comprising millions of ChemCensor-validated reaction records for LLM training, and use it to train a model that improves over the LLM baselines under this benchmark.} }
Endnote
%0 Conference Paper %T When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs %A Bogdan Zagribelnyy %A Ivan Ilin %A Maksim Kuznetsov %A Nikita Bondarev %A Roman Schutski %A Thomas Macdougall %A Rim Shayakhmetov %A Zulfat Miftahutdinov %A Mikolaj Mizera %A Vladimir Aladinskiy %A Alex Aliper %A Alex Zhavoronkov %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zagribelnyy26a %I PMLR %P 152836--152919 %U https://proceedings.mlr.press/v306/zagribelnyy26a.html %V 306 %X Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evaluation of retrosynthesis performance remains limited. Existing benchmarks and metrics typically rely on published synthetic procedures and Top-K accuracy based on single ground-truth, which does not capture the open-ended nature of real-world synthesis planning. We propose a new benchmarking framework for single-step retrosynthesis that evaluates both general-purpose and chemistry-specialized LLMs using ChemCensor, a novel metric for chemical plausibility. By emphasizing plausibility over exact match, this approach better aligns with human synthesis planning practices. We also introduce CREED, a novel dataset comprising millions of ChemCensor-validated reaction records for LLM training, and use it to train a model that improves over the LLM baselines under this benchmark.
APA
Zagribelnyy, B., Ilin, I., Kuznetsov, M., Bondarev, N., Schutski, R., Macdougall, T., Shayakhmetov, R., Miftahutdinov, Z., Mizera, M., Aladinskiy, V., Aliper, A. & Zhavoronkov, A.. (2026). When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:152836-152919 Available from https://proceedings.mlr.press/v306/zagribelnyy26a.html.

Related Material