Training AI Co-Scientists Using Rubric Rewards

Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F. Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping, Chenxi Whitehouse
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:35291-35406, 2026.

Abstract

AI co-scientists are emerging as a useful tool for human researchers, with a crucial ability being proposing a research plan for a given research goal. In this work, we study how to train language models that generate better research plans by leveraging the vast corpus of existing research papers. To collect diverse training data, we automatically extract research goals and goal-specific grading rubrics from papers across domains. We then train models for research plan generation via reinforcement learning, with a frozen copy of the initial policy acting as the grader, using the rubrics to evaluate plans generated by the training policy. To validate this approach, we conduct a human study for machine learning research goals spanning 225 expert hours. The experts prefer plans generated by our finetuned Qwen3-30B-A3B model over the initial model for 70% goals, and over Grok-4-Thinking for 59.6% goals. To assess generality, we also extend our approach to goals from medical papers, and recent arXiv preprints, evaluating with a jury of frontier models. Our finetuning yields 12-22% relative improvements and significant cross-domain generalization, proving effective even in problem settings like medical research where execution feedback is infeasible. Overall, we demonstrate the potential of a scalable training recipe as a step towards improving general AI co-scientists.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-goel26b, title = {Training {AI} Co-Scientists Using Rubric Rewards}, author = {Goel, Shashwat and Hazra, Rishi and Jayalath, Dulhan and Willi, Timon and Jain, Parag and Shen, William F. and Leontiadis, Ilias and Barbieri, Francesco and Bachrach, Yoram and Geiping, Jonas and Whitehouse, Chenxi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {35291--35406}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/goel26b/goel26b.pdf}, url = {https://proceedings.mlr.press/v306/goel26b.html}, abstract = {AI co-scientists are emerging as a useful tool for human researchers, with a crucial ability being proposing a research plan for a given research goal. In this work, we study how to train language models that generate better research plans by leveraging the vast corpus of existing research papers. To collect diverse training data, we automatically extract research goals and goal-specific grading rubrics from papers across domains. We then train models for research plan generation via reinforcement learning, with a frozen copy of the initial policy acting as the grader, using the rubrics to evaluate plans generated by the training policy. To validate this approach, we conduct a human study for machine learning research goals spanning 225 expert hours. The experts prefer plans generated by our finetuned Qwen3-30B-A3B model over the initial model for 70% goals, and over Grok-4-Thinking for 59.6% goals. To assess generality, we also extend our approach to goals from medical papers, and recent arXiv preprints, evaluating with a jury of frontier models. Our finetuning yields 12-22% relative improvements and significant cross-domain generalization, proving effective even in problem settings like medical research where execution feedback is infeasible. Overall, we demonstrate the potential of a scalable training recipe as a step towards improving general AI co-scientists.} }
Endnote
%0 Conference Paper %T Training AI Co-Scientists Using Rubric Rewards %A Shashwat Goel %A Rishi Hazra %A Dulhan Jayalath %A Timon Willi %A Parag Jain %A William F. Shen %A Ilias Leontiadis %A Francesco Barbieri %A Yoram Bachrach %A Jonas Geiping %A Chenxi Whitehouse %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-goel26b %I PMLR %P 35291--35406 %U https://proceedings.mlr.press/v306/goel26b.html %V 306 %X AI co-scientists are emerging as a useful tool for human researchers, with a crucial ability being proposing a research plan for a given research goal. In this work, we study how to train language models that generate better research plans by leveraging the vast corpus of existing research papers. To collect diverse training data, we automatically extract research goals and goal-specific grading rubrics from papers across domains. We then train models for research plan generation via reinforcement learning, with a frozen copy of the initial policy acting as the grader, using the rubrics to evaluate plans generated by the training policy. To validate this approach, we conduct a human study for machine learning research goals spanning 225 expert hours. The experts prefer plans generated by our finetuned Qwen3-30B-A3B model over the initial model for 70% goals, and over Grok-4-Thinking for 59.6% goals. To assess generality, we also extend our approach to goals from medical papers, and recent arXiv preprints, evaluating with a jury of frontier models. Our finetuning yields 12-22% relative improvements and significant cross-domain generalization, proving effective even in problem settings like medical research where execution feedback is infeasible. Overall, we demonstrate the potential of a scalable training recipe as a step towards improving general AI co-scientists.
APA
Goel, S., Hazra, R., Jayalath, D., Willi, T., Jain, P., Shen, W.F., Leontiadis, I., Barbieri, F., Bachrach, Y., Geiping, J. & Whitehouse, C.. (2026). Training AI Co-Scientists Using Rubric Rewards. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:35291-35406 Available from https://proceedings.mlr.press/v306/goel26b.html.

Related Material