From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

Yishuo Cai, Xingyu Guo, Xuancheng Huang, Jinhua Du, Can Huang, Wenxuan Huang, Wenhan Ma, Yuyang Hu, Aohan Zeng, Jie Tang, Xu Sun
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:10503-10520, 2026.

Abstract

Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM’s performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold’em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including Deepseek-V3.2. Our code is publicly available.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-cai26b, title = {From Player to Master: Enhancing Test-Time Learning of {LLM} Agents via Reinforcement Learning over Memory}, author = {Cai, Yishuo and Guo, Xingyu and Huang, Xuancheng and Du, Jinhua and Huang, Can and Huang, Wenxuan and Ma, Wenhan and Hu, Yuyang and Zeng, Aohan and Tang, Jie and Sun, Xu}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {10503--10520}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/cai26b/cai26b.pdf}, url = {https://proceedings.mlr.press/v306/cai26b.html}, abstract = {Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM’s performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold’em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including Deepseek-V3.2. Our code is publicly available.} }
Endnote
%0 Conference Paper %T From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory %A Yishuo Cai %A Xingyu Guo %A Xuancheng Huang %A Jinhua Du %A Can Huang %A Wenxuan Huang %A Wenhan Ma %A Yuyang Hu %A Aohan Zeng %A Jie Tang %A Xu Sun %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-cai26b %I PMLR %P 10503--10520 %U https://proceedings.mlr.press/v306/cai26b.html %V 306 %X Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM’s performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold’em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including Deepseek-V3.2. Our code is publicly available.
APA
Cai, Y., Guo, X., Huang, X., Du, J., Huang, C., Huang, W., Ma, W., Hu, Y., Zeng, A., Tang, J. & Sun, X.. (2026). From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:10503-10520 Available from https://proceedings.mlr.press/v306/cai26b.html.

Related Material