SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents

Yujiong Shen, Yajie Yang, Zhiheng Xi, Binze Hu, Huayu Sha, Qiyuan Peng, Jiazheng Zhang, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Ming Zhang, Shihan Dou, Zhenfei Yin, Xingjun Ma, Lei Bai, Tao Gui, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:111430-111454, 2026.

Abstract

Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely overlook agents’ ability to orchestrate tools for such rigorous workflows. To bridge this gap, we introduce SciAgentGym, a scalable interactive environment featuring 1,780 domain-specific tools across four natural science disciplines, supported by a robust execution infrastructure. Complementing this, we present SciAgentBench, a tiered evaluation suite designed to stress-test agentic capabilities from elementary actions to long-horizon workflows. Our evaluation identifies a critical bottleneck: state-of-the-art models still struggle with complex scientific tool-use, and their performance degrades substantially as interaction horizons extend. To address this, we propose SciForge, a data synthesis method that models the tool action space as a dependency graph to generate logic-aware training trajectories. By fine-tuning on these trajectories, our SciAgent-8B outperforms the significantly larger Qwen3-VL-235B-Instruct while exhibiting positive cross-domain transfer of scientific tool-use capabilities. These results underscore the promising potential of next-generation autonomous scientific agents.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-shen26aa, title = {{S}ci{A}gent{G}ym: Benchmarking Multi-Step Scientific Tool-Use in {LLM} Agents}, author = {Shen, Yujiong and Yang, Yajie and Xi, Zhiheng and Hu, Binze and Sha, Huayu and Peng, Qiyuan and Zhang, Jiazheng and Shang, Junlin and Huang, Jixuan and Fan, Yutao and Tong, Jingqi and Zhang, Ming and Dou, Shihan and Yin, Zhenfei and Ma, Xingjun and Bai, Lei and Gui, Tao and Zhang, Qi and Huang, Xuanjing and Jiang, Yu-Gang}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {111430--111454}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/shen26aa/shen26aa.pdf}, url = {https://proceedings.mlr.press/v306/shen26aa.html}, abstract = {Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely overlook agents’ ability to orchestrate tools for such rigorous workflows. To bridge this gap, we introduce SciAgentGym, a scalable interactive environment featuring 1,780 domain-specific tools across four natural science disciplines, supported by a robust execution infrastructure. Complementing this, we present SciAgentBench, a tiered evaluation suite designed to stress-test agentic capabilities from elementary actions to long-horizon workflows. Our evaluation identifies a critical bottleneck: state-of-the-art models still struggle with complex scientific tool-use, and their performance degrades substantially as interaction horizons extend. To address this, we propose SciForge, a data synthesis method that models the tool action space as a dependency graph to generate logic-aware training trajectories. By fine-tuning on these trajectories, our SciAgent-8B outperforms the significantly larger Qwen3-VL-235B-Instruct while exhibiting positive cross-domain transfer of scientific tool-use capabilities. These results underscore the promising potential of next-generation autonomous scientific agents.} }
Endnote
%0 Conference Paper %T SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents %A Yujiong Shen %A Yajie Yang %A Zhiheng Xi %A Binze Hu %A Huayu Sha %A Qiyuan Peng %A Jiazheng Zhang %A Junlin Shang %A Jixuan Huang %A Yutao Fan %A Jingqi Tong %A Ming Zhang %A Shihan Dou %A Zhenfei Yin %A Xingjun Ma %A Lei Bai %A Tao Gui %A Qi Zhang %A Xuanjing Huang %A Yu-Gang Jiang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-shen26aa %I PMLR %P 111430--111454 %U https://proceedings.mlr.press/v306/shen26aa.html %V 306 %X Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely overlook agents’ ability to orchestrate tools for such rigorous workflows. To bridge this gap, we introduce SciAgentGym, a scalable interactive environment featuring 1,780 domain-specific tools across four natural science disciplines, supported by a robust execution infrastructure. Complementing this, we present SciAgentBench, a tiered evaluation suite designed to stress-test agentic capabilities from elementary actions to long-horizon workflows. Our evaluation identifies a critical bottleneck: state-of-the-art models still struggle with complex scientific tool-use, and their performance degrades substantially as interaction horizons extend. To address this, we propose SciForge, a data synthesis method that models the tool action space as a dependency graph to generate logic-aware training trajectories. By fine-tuning on these trajectories, our SciAgent-8B outperforms the significantly larger Qwen3-VL-235B-Instruct while exhibiting positive cross-domain transfer of scientific tool-use capabilities. These results underscore the promising potential of next-generation autonomous scientific agents.
APA
Shen, Y., Yang, Y., Xi, Z., Hu, B., Sha, H., Peng, Q., Zhang, J., Shang, J., Huang, J., Fan, Y., Tong, J., Zhang, M., Dou, S., Yin, Z., Ma, X., Bai, L., Gui, T., Zhang, Q., Huang, X. & Jiang, Y.. (2026). SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:111430-111454 Available from https://proceedings.mlr.press/v306/shen26aa.html.

Related Material