UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms

Peng Lai, Yichao Du, Junchao Wu, Weibo Gao, Linan Yue, Longyue Wang, Weihua Luo, Derek F. Wong, Guanhua Chen
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:62175-62205, 2026.

Abstract

Reinforcement learning (RL) excels on tasks with verifiable rewards, but in open-ended tasks, the reliability of reward models remains a key challenge. Existing solutions either depend on costly proprietary LLM-as-a-Judge systems or opaque scalar reward models that lack interpretability. Recent works on generative reward models offer a promising alternative, but they remain constrained by static evaluation criteria, fragmented evaluation paradigms, and limited multilingual support. To address these challenges, we introduce MixReward, a large-scale multilingual dataset spanning six domains and 103 languages, containing both pairwise and listwise data, and propose UniRRM, a unified reasoning reward model supporting multiple languages and evaluation paradigms. UniRRM uses a staged reasoning chain to dynamically generate task-generic and instruction-specific criteria, enabling fine-grained, input-adaptive judgments while maintaining consistency across languages. Experiments demonstrate that UniRRM-8B and UniRRM-14B achieve performance close to the state-of-the-art for models of comparable size across multiple benchmarks, and are effective for unseen evaluation paradigms. In addition, ablation studies validate the reliability and effectiveness of UniRRM.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-lai26a, title = {{U}ni{RRM}: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms}, author = {Lai, Peng and Du, Yichao and Wu, Junchao and Gao, Weibo and Yue, Linan and Wang, Longyue and Luo, Weihua and Wong, Derek F. and Chen, Guanhua}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {62175--62205}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/lai26a/lai26a.pdf}, url = {https://proceedings.mlr.press/v306/lai26a.html}, abstract = {Reinforcement learning (RL) excels on tasks with verifiable rewards, but in open-ended tasks, the reliability of reward models remains a key challenge. Existing solutions either depend on costly proprietary LLM-as-a-Judge systems or opaque scalar reward models that lack interpretability. Recent works on generative reward models offer a promising alternative, but they remain constrained by static evaluation criteria, fragmented evaluation paradigms, and limited multilingual support. To address these challenges, we introduce MixReward, a large-scale multilingual dataset spanning six domains and 103 languages, containing both pairwise and listwise data, and propose UniRRM, a unified reasoning reward model supporting multiple languages and evaluation paradigms. UniRRM uses a staged reasoning chain to dynamically generate task-generic and instruction-specific criteria, enabling fine-grained, input-adaptive judgments while maintaining consistency across languages. Experiments demonstrate that UniRRM-8B and UniRRM-14B achieve performance close to the state-of-the-art for models of comparable size across multiple benchmarks, and are effective for unseen evaluation paradigms. In addition, ablation studies validate the reliability and effectiveness of UniRRM.} }
Endnote
%0 Conference Paper %T UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms %A Peng Lai %A Yichao Du %A Junchao Wu %A Weibo Gao %A Linan Yue %A Longyue Wang %A Weihua Luo %A Derek F. Wong %A Guanhua Chen %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-lai26a %I PMLR %P 62175--62205 %U https://proceedings.mlr.press/v306/lai26a.html %V 306 %X Reinforcement learning (RL) excels on tasks with verifiable rewards, but in open-ended tasks, the reliability of reward models remains a key challenge. Existing solutions either depend on costly proprietary LLM-as-a-Judge systems or opaque scalar reward models that lack interpretability. Recent works on generative reward models offer a promising alternative, but they remain constrained by static evaluation criteria, fragmented evaluation paradigms, and limited multilingual support. To address these challenges, we introduce MixReward, a large-scale multilingual dataset spanning six domains and 103 languages, containing both pairwise and listwise data, and propose UniRRM, a unified reasoning reward model supporting multiple languages and evaluation paradigms. UniRRM uses a staged reasoning chain to dynamically generate task-generic and instruction-specific criteria, enabling fine-grained, input-adaptive judgments while maintaining consistency across languages. Experiments demonstrate that UniRRM-8B and UniRRM-14B achieve performance close to the state-of-the-art for models of comparable size across multiple benchmarks, and are effective for unseen evaluation paradigms. In addition, ablation studies validate the reliability and effectiveness of UniRRM.
APA
Lai, P., Du, Y., Wu, J., Gao, W., Yue, L., Wang, L., Luo, W., Wong, D.F. & Chen, G.. (2026). UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:62175-62205 Available from https://proceedings.mlr.press/v306/lai26a.html.

Related Material