Hista and Numca: Estimate State Value Effectively for Large Language Model Reinforcement Learning

Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, James Cheng
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:14050-14079, 2026.

Abstract

Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post-training. In this work, we introduce the State Value Estimation Benchmark (SVEB) to assess state estimation within existing RL frameworks and show that critics in standard approaches like PPO collapse to a coarse group-average baseline. To address this, we propose two techniques: Numca, which leverages numerical spans as gradable milestones for state value estimation, and Hista, a framework that uses LLM’s hidden states as representation to weighted average disjoint rollouts and their return. Extensive experiments demonstrate that both methods yield more accurate state value estimates and enhance training performance across different RL algorithms and model sizes without incurring significant computational overhead. Code available at https://github.com/VOXXXX1874/Hista.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26w, title = {Hista and Numca: Estimate State Value Effectively for Large Language Model Reinforcement Learning}, author = {Chen, Zizhe and Dong, Jiqian and Tian, Yizhou and Yang, Garry and Chen, Yongqiang and Chen, Zhitang and Cheng, James}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {14050--14079}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26w/chen26w.pdf}, url = {https://proceedings.mlr.press/v306/chen26w.html}, abstract = {Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post-training. In this work, we introduce the State Value Estimation Benchmark (SVEB) to assess state estimation within existing RL frameworks and show that critics in standard approaches like PPO collapse to a coarse group-average baseline. To address this, we propose two techniques: Numca, which leverages numerical spans as gradable milestones for state value estimation, and Hista, a framework that uses LLM’s hidden states as representation to weighted average disjoint rollouts and their return. Extensive experiments demonstrate that both methods yield more accurate state value estimates and enhance training performance across different RL algorithms and model sizes without incurring significant computational overhead. Code available at https://github.com/VOXXXX1874/Hista.} }
Endnote
%0 Conference Paper %T Hista and Numca: Estimate State Value Effectively for Large Language Model Reinforcement Learning %A Zizhe Chen %A Jiqian Dong %A Yizhou Tian %A Garry Yang %A Yongqiang Chen %A Zhitang Chen %A James Cheng %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26w %I PMLR %P 14050--14079 %U https://proceedings.mlr.press/v306/chen26w.html %V 306 %X Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post-training. In this work, we introduce the State Value Estimation Benchmark (SVEB) to assess state estimation within existing RL frameworks and show that critics in standard approaches like PPO collapse to a coarse group-average baseline. To address this, we propose two techniques: Numca, which leverages numerical spans as gradable milestones for state value estimation, and Hista, a framework that uses LLM’s hidden states as representation to weighted average disjoint rollouts and their return. Extensive experiments demonstrate that both methods yield more accurate state value estimates and enhance training performance across different RL algorithms and model sizes without incurring significant computational overhead. Code available at https://github.com/VOXXXX1874/Hista.
APA
Chen, Z., Dong, J., Tian, Y., Yang, G., Chen, Y., Chen, Z. & Cheng, J.. (2026). Hista and Numca: Estimate State Value Effectively for Large Language Model Reinforcement Learning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:14050-14079 Available from https://proceedings.mlr.press/v306/chen26w.html.

Related Material