Reinforcement Learning with Verifiable Rewards: GRPO’s Loss, Dynamics, and Success Amplification

Youssef Mroueh
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:90821-90847, 2026.

Abstract

Group Relative Policy Optimization (GRPO) was introduced recently and used to train DeepSeek–R1 for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean+variance calibration of these rewards induces a contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs. mean+variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (mirror), penalizing divergence from a fixed reference model $\pi_{\mathrm{ref}}$, or combining both forms of regularization. For each, the optimal policy $\pi_n$ admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under $\pi_{n-1}$, as well as the policies $\pi_{n-1}$ and $\pi_{\mathrm{ref}}$. Iterating results in a sequence ${\pi_n}$ whose probability of success (PoS) obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy’s probability of success.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-mroueh26a, title = {Reinforcement Learning with Verifiable Rewards: {GRPO}’s Loss, Dynamics, and Success Amplification}, author = {Mroueh, Youssef}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {90821--90847}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/mroueh26a/mroueh26a.pdf}, url = {https://proceedings.mlr.press/v306/mroueh26a.html}, abstract = {Group Relative Policy Optimization (GRPO) was introduced recently and used to train DeepSeek–R1 for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean+variance calibration of these rewards induces a contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs. mean+variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (mirror), penalizing divergence from a fixed reference model $\pi_{\mathrm{ref}}$, or combining both forms of regularization. For each, the optimal policy $\pi_n$ admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under $\pi_{n-1}$, as well as the policies $\pi_{n-1}$ and $\pi_{\mathrm{ref}}$. Iterating results in a sequence ${\pi_n}$ whose probability of success (PoS) obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy’s probability of success.} }
Endnote
%0 Conference Paper %T Reinforcement Learning with Verifiable Rewards: GRPO’s Loss, Dynamics, and Success Amplification %A Youssef Mroueh %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-mroueh26a %I PMLR %P 90821--90847 %U https://proceedings.mlr.press/v306/mroueh26a.html %V 306 %X Group Relative Policy Optimization (GRPO) was introduced recently and used to train DeepSeek–R1 for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean+variance calibration of these rewards induces a contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs. mean+variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (mirror), penalizing divergence from a fixed reference model $\pi_{\mathrm{ref}}$, or combining both forms of regularization. For each, the optimal policy $\pi_n$ admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under $\pi_{n-1}$, as well as the policies $\pi_{n-1}$ and $\pi_{\mathrm{ref}}$. Iterating results in a sequence ${\pi_n}$ whose probability of success (PoS) obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy’s probability of success.
APA
Mroueh, Y.. (2026). Reinforcement Learning with Verifiable Rewards: GRPO’s Loss, Dynamics, and Success Amplification. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:90821-90847 Available from https://proceedings.mlr.press/v306/mroueh26a.html.

Related Material