ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning

Xiaoxuan Wang, Han Zhang, Haixin Wang, Yidan Shi, Ruoyan Li, Kaiqiao Han, Chenyi Tong, Haoran Deng, Alexander K Taylor, Renliang Sun, Yanqiao Zhu, Jason Cong, Yizhou Sun, Wei Wang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:132514-132540, 2026.

Abstract

Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks. Despite encouraging early results, ARL remains highly unstable, often leading to training collapse. This instability limits scalability to larger environments and longer interaction horizons, and constrains systematic exploration of algorithmic design choices. In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting. ARLArena first constructs a clean and standardized testbed. Then, we decompose policy gradient into four core design dimensions and assess the performance and stability of each dimension. Through this fine-grained analysis, we propose SAMPO, a stable agentic policy optimization method designed to mitigate the dominant sources of instability in ARL. Empirically, SAMPO achieves consistently stable training and strong performance across diverse agentic tasks. Our code is open-sourced at: https://github.com/WillDreamer/ARL-Arena.git

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-wang26kv, title = {{ARLA}rena: A Unified Framework for Stable Agentic Reinforcement Learning}, author = {Wang, Xiaoxuan and Zhang, Han and Wang, Haixin and Shi, Yidan and Li, Ruoyan and Han, Kaiqiao and Tong, Chenyi and Deng, Haoran and Taylor, Alexander K and Sun, Renliang and Zhu, Yanqiao and Cong, Jason and Sun, Yizhou and Wang, Wei}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {132514--132540}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wang26kv/wang26kv.pdf}, url = {https://proceedings.mlr.press/v306/wang26kv.html}, abstract = {Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks. Despite encouraging early results, ARL remains highly unstable, often leading to training collapse. This instability limits scalability to larger environments and longer interaction horizons, and constrains systematic exploration of algorithmic design choices. In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting. ARLArena first constructs a clean and standardized testbed. Then, we decompose policy gradient into four core design dimensions and assess the performance and stability of each dimension. Through this fine-grained analysis, we propose SAMPO, a stable agentic policy optimization method designed to mitigate the dominant sources of instability in ARL. Empirically, SAMPO achieves consistently stable training and strong performance across diverse agentic tasks. Our code is open-sourced at: https://github.com/WillDreamer/ARL-Arena.git} }
Endnote
%0 Conference Paper %T ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning %A Xiaoxuan Wang %A Han Zhang %A Haixin Wang %A Yidan Shi %A Ruoyan Li %A Kaiqiao Han %A Chenyi Tong %A Haoran Deng %A Alexander K Taylor %A Renliang Sun %A Yanqiao Zhu %A Jason Cong %A Yizhou Sun %A Wei Wang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wang26kv %I PMLR %P 132514--132540 %U https://proceedings.mlr.press/v306/wang26kv.html %V 306 %X Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks. Despite encouraging early results, ARL remains highly unstable, often leading to training collapse. This instability limits scalability to larger environments and longer interaction horizons, and constrains systematic exploration of algorithmic design choices. In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting. ARLArena first constructs a clean and standardized testbed. Then, we decompose policy gradient into four core design dimensions and assess the performance and stability of each dimension. Through this fine-grained analysis, we propose SAMPO, a stable agentic policy optimization method designed to mitigate the dominant sources of instability in ARL. Empirically, SAMPO achieves consistently stable training and strong performance across diverse agentic tasks. Our code is open-sourced at: https://github.com/WillDreamer/ARL-Arena.git
APA
Wang, X., Zhang, H., Wang, H., Shi, Y., Li, R., Han, K., Tong, C., Deng, H., Taylor, A.K., Sun, R., Zhu, Y., Cong, J., Sun, Y. & Wang, W.. (2026). ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:132514-132540 Available from https://proceedings.mlr.press/v306/wang26kv.html.

Related Material