$V_0$: A Generalist Value Model for Any Policy at State Zero

Yi-Kai Zhang, Zhiyuan Yao, Hongyan Hao, Yueqing Sun, Qi Gu, Hui Su, Xunliang Cai, De-Chuan Zhan, Han-Jia Ye
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:159992-160014, 2026.

Abstract

Policy gradient methods rely on a baseline to measure the relative advantage of an action. In Actor-Critic LLM training (e.g., PPO), this baseline is typically estimated by a Value Model as large as the policy. However, as the policy evolves, the value model requires expensive, synchronous training. To avoid this overhead, GRPO eliminates the value model by using the average reward of group rollouts as the baseline; yet, this necessitates extensive sampling to maintain stability. In this paper, we propose $V_0$, a Generalist Value Model capable of estimating the expected performance of any model on unseen prompts without parameter updates. We reframe value estimation by treating the policy’s dynamic capability as an explicit context input, leveraging a history of instruction-performance pairs to dynamically profile the model. Acting as a resource scheduler, $V_0$ predicts success rates prior to rollout during GRPO training for efficient sampling budget allocation. During deployment, it functions as a router dispatching instructions to the most cost-effective model. Empirical results demonstrate that $V_0$ significantly outperforms heuristic budget allocation and achieves a Pareto-optimal performance-cost trade-off in LLM routing.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-zhang26hu, title = {$V_0$: A Generalist Value Model for Any Policy at State Zero}, author = {Zhang, Yi-Kai and Yao, Zhiyuan and Hao, Hongyan and Sun, Yueqing and Gu, Qi and Su, Hui and Cai, Xunliang and Zhan, De-Chuan and Ye, Han-Jia}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {159992--160014}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26hu/zhang26hu.pdf}, url = {https://proceedings.mlr.press/v306/zhang26hu.html}, abstract = {Policy gradient methods rely on a baseline to measure the relative advantage of an action. In Actor-Critic LLM training (e.g., PPO), this baseline is typically estimated by a Value Model as large as the policy. However, as the policy evolves, the value model requires expensive, synchronous training. To avoid this overhead, GRPO eliminates the value model by using the average reward of group rollouts as the baseline; yet, this necessitates extensive sampling to maintain stability. In this paper, we propose $V_0$, a Generalist Value Model capable of estimating the expected performance of any model on unseen prompts without parameter updates. We reframe value estimation by treating the policy’s dynamic capability as an explicit context input, leveraging a history of instruction-performance pairs to dynamically profile the model. Acting as a resource scheduler, $V_0$ predicts success rates prior to rollout during GRPO training for efficient sampling budget allocation. During deployment, it functions as a router dispatching instructions to the most cost-effective model. Empirical results demonstrate that $V_0$ significantly outperforms heuristic budget allocation and achieves a Pareto-optimal performance-cost trade-off in LLM routing.} }
Endnote
%0 Conference Paper %T $V_0$: A Generalist Value Model for Any Policy at State Zero %A Yi-Kai Zhang %A Zhiyuan Yao %A Hongyan Hao %A Yueqing Sun %A Qi Gu %A Hui Su %A Xunliang Cai %A De-Chuan Zhan %A Han-Jia Ye %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26hu %I PMLR %P 159992--160014 %U https://proceedings.mlr.press/v306/zhang26hu.html %V 306 %X Policy gradient methods rely on a baseline to measure the relative advantage of an action. In Actor-Critic LLM training (e.g., PPO), this baseline is typically estimated by a Value Model as large as the policy. However, as the policy evolves, the value model requires expensive, synchronous training. To avoid this overhead, GRPO eliminates the value model by using the average reward of group rollouts as the baseline; yet, this necessitates extensive sampling to maintain stability. In this paper, we propose $V_0$, a Generalist Value Model capable of estimating the expected performance of any model on unseen prompts without parameter updates. We reframe value estimation by treating the policy’s dynamic capability as an explicit context input, leveraging a history of instruction-performance pairs to dynamically profile the model. Acting as a resource scheduler, $V_0$ predicts success rates prior to rollout during GRPO training for efficient sampling budget allocation. During deployment, it functions as a router dispatching instructions to the most cost-effective model. Empirical results demonstrate that $V_0$ significantly outperforms heuristic budget allocation and achieves a Pareto-optimal performance-cost trade-off in LLM routing.
APA
Zhang, Y., Yao, Z., Hao, H., Sun, Y., Gu, Q., Su, H., Cai, X., Zhan, D. & Ye, H.. (2026). $V_0$: A Generalist Value Model for Any Policy at State Zero. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:159992-160014 Available from https://proceedings.mlr.press/v306/zhang26hu.html.

Related Material