The Unlearnability Phenomenon in RLVR for Language Models

Yulin Chen, He He, Chen Zhao
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:14538-14561, 2026.

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has proven effective in improving Large Language Model’s (LLM) reasoning ability. However, the learning dynamics of RLVR remain underexplored. In this paper, we reveal a counterintuitive phenomenon: among hard examples that the model initially struggles with, a substantial subset remains unlearnable even when correct rollouts are present. To understand the phenomenon, we first demonstrate that existing optimization and sampling techniques fail to resolve unlearnability. With cross-example gradient analysis, we show that unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples and ungeneralizable reasoning patterns. We further show that representation flaws are difficult to mitigate in RL, as data augmentation does not improve gradient similarity. Our study provides the first systematic characterization of unlearnable data in RLVR training and reveals fundamental limitations in current RL approaches for reasoning tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26aq, title = {The Unlearnability Phenomenon in {RLVR} for Language Models}, author = {Chen, Yulin and He, He and Zhao, Chen}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {14538--14561}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26aq/chen26aq.pdf}, url = {https://proceedings.mlr.press/v306/chen26aq.html}, abstract = {Reinforcement Learning with Verifiable Reward (RLVR) has proven effective in improving Large Language Model’s (LLM) reasoning ability. However, the learning dynamics of RLVR remain underexplored. In this paper, we reveal a counterintuitive phenomenon: among hard examples that the model initially struggles with, a substantial subset remains unlearnable even when correct rollouts are present. To understand the phenomenon, we first demonstrate that existing optimization and sampling techniques fail to resolve unlearnability. With cross-example gradient analysis, we show that unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples and ungeneralizable reasoning patterns. We further show that representation flaws are difficult to mitigate in RL, as data augmentation does not improve gradient similarity. Our study provides the first systematic characterization of unlearnable data in RLVR training and reveals fundamental limitations in current RL approaches for reasoning tasks.} }
Endnote
%0 Conference Paper %T The Unlearnability Phenomenon in RLVR for Language Models %A Yulin Chen %A He He %A Chen Zhao %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26aq %I PMLR %P 14538--14561 %U https://proceedings.mlr.press/v306/chen26aq.html %V 306 %X Reinforcement Learning with Verifiable Reward (RLVR) has proven effective in improving Large Language Model’s (LLM) reasoning ability. However, the learning dynamics of RLVR remain underexplored. In this paper, we reveal a counterintuitive phenomenon: among hard examples that the model initially struggles with, a substantial subset remains unlearnable even when correct rollouts are present. To understand the phenomenon, we first demonstrate that existing optimization and sampling techniques fail to resolve unlearnability. With cross-example gradient analysis, we show that unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples and ungeneralizable reasoning patterns. We further show that representation flaws are difficult to mitigate in RL, as data augmentation does not improve gradient similarity. Our study provides the first systematic characterization of unlearnable data in RLVR training and reveals fundamental limitations in current RL approaches for reasoning tasks.
APA
Chen, Y., He, H. & Zhao, C.. (2026). The Unlearnability Phenomenon in RLVR for Language Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:14538-14561 Available from https://proceedings.mlr.press/v306/chen26aq.html.

Related Material