Rate or Fate? RLV$^\varepsilon$R: Reinforcement Learning with Verifiable Noisy Rewards

Ali Rad, Khashayar Filom, Darioush Keivan, Peyman Mohajerin Esfahani, Ehsan Kamalinejad
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:102933-102973, 2026.

Abstract

Reinforcement learning with verifiable rewards (RLVR) trains a policy by verifying sampled completions and reinforcing higher-scoring outputs, but practical verifiers (e.g., incomplete unit tests or noisy judges) are prone to false positives and false negatives. We ask when such noise merely slows learning and when it reverses it. Modeling GRPO-style RLVR as a bandit over recurring reasoning modes, we derive mean-field replicator-style (natural-selection) flow on the probability simplex. The dynamics decouples into within-correct-mode competition and a one-dimensional evolution for the mass on incorrect modes, whose drift is determined solely by Youden’s index $J=\mathrm{TPR}-\mathrm{FPR}$. This yields a sharp phase transition: when $J>0$, the incorrect mass is driven toward extinction (learning); when $J=0$, the process is neutral; and when $J<0$, incorrect modes amplify until they dominate (anti-learning and collapse). In the learning regime $J>0$, noise primarily rescales convergence time (“rate, not fate”). Experiments on verifiable programming tasks under synthetic noise reproduce the predicted $J=0$ boundary. Beyond noise, the framework offers a general lens for analyzing RLVR stability, convergence, and algorithmic interventions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-rad26b, title = {Rate or Fate? {RLV}$^{\varepsilon}$R: Reinforcement Learning with Verifiable Noisy Rewards}, author = {Rad, Ali and Filom, Khashayar and Keivan, Darioush and Mohajerin Esfahani, Peyman and Kamalinejad, Ehsan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {102933--102973}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/rad26b/rad26b.pdf}, url = {https://proceedings.mlr.press/v306/rad26b.html}, abstract = {Reinforcement learning with verifiable rewards (RLVR) trains a policy by verifying sampled completions and reinforcing higher-scoring outputs, but practical verifiers (e.g., incomplete unit tests or noisy judges) are prone to false positives and false negatives. We ask when such noise merely slows learning and when it reverses it. Modeling GRPO-style RLVR as a bandit over recurring reasoning modes, we derive mean-field replicator-style (natural-selection) flow on the probability simplex. The dynamics decouples into within-correct-mode competition and a one-dimensional evolution for the mass on incorrect modes, whose drift is determined solely by Youden’s index $J=\mathrm{TPR}-\mathrm{FPR}$. This yields a sharp phase transition: when $J>0$, the incorrect mass is driven toward extinction (learning); when $J=0$, the process is neutral; and when $J<0$, incorrect modes amplify until they dominate (anti-learning and collapse). In the learning regime $J>0$, noise primarily rescales convergence time (“rate, not fate”). Experiments on verifiable programming tasks under synthetic noise reproduce the predicted $J=0$ boundary. Beyond noise, the framework offers a general lens for analyzing RLVR stability, convergence, and algorithmic interventions.} }
Endnote
%0 Conference Paper %T Rate or Fate? RLV$^\varepsilon$R: Reinforcement Learning with Verifiable Noisy Rewards %A Ali Rad %A Khashayar Filom %A Darioush Keivan %A Peyman Mohajerin Esfahani %A Ehsan Kamalinejad %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-rad26b %I PMLR %P 102933--102973 %U https://proceedings.mlr.press/v306/rad26b.html %V 306 %X Reinforcement learning with verifiable rewards (RLVR) trains a policy by verifying sampled completions and reinforcing higher-scoring outputs, but practical verifiers (e.g., incomplete unit tests or noisy judges) are prone to false positives and false negatives. We ask when such noise merely slows learning and when it reverses it. Modeling GRPO-style RLVR as a bandit over recurring reasoning modes, we derive mean-field replicator-style (natural-selection) flow on the probability simplex. The dynamics decouples into within-correct-mode competition and a one-dimensional evolution for the mass on incorrect modes, whose drift is determined solely by Youden’s index $J=\mathrm{TPR}-\mathrm{FPR}$. This yields a sharp phase transition: when $J>0$, the incorrect mass is driven toward extinction (learning); when $J=0$, the process is neutral; and when $J<0$, incorrect modes amplify until they dominate (anti-learning and collapse). In the learning regime $J>0$, noise primarily rescales convergence time (“rate, not fate”). Experiments on verifiable programming tasks under synthetic noise reproduce the predicted $J=0$ boundary. Beyond noise, the framework offers a general lens for analyzing RLVR stability, convergence, and algorithmic interventions.
APA
Rad, A., Filom, K., Keivan, D., Mohajerin Esfahani, P. & Kamalinejad, E.. (2026). Rate or Fate? RLV$^\varepsilon$R: Reinforcement Learning with Verifiable Noisy Rewards. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:102933-102973 Available from https://proceedings.mlr.press/v306/rad26b.html.

Related Material