OCNR: Stabilizing Self-Play by Mitigating Iteration-Collapse With One-Class Novelty Rewards

Seungyoo Lee, Giung Nam, Hyungi Lee, Juho Lee
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:64837-64861, 2026.

Abstract

Training large language models via self-play often suffers from a persistent iteration-collapse, where performance initially improves but subsequently regresses as training iterations increase. We analyze this phenomenon as arising from cross-iteration degeneration, where the task-generation distribution becomes increasingly confined to a narrow subset of familiar (seen) problems, weakening the effective learning signal and destabilizing training. To address this issue, we propose a plug-in approach that augments existing self-play pipelines with a one-class novelty reward. A Seen Detector trained on a historical buffer of previously used training problems identifies in-support instances and discourages redundant generation by the questioner, thereby steering exploration toward under-explored yet learnable regions. Experimental results show that the proposed method mitigates iteration-collapse during iterative training and yields consistent improvements.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-lee26ay, title = {{OCNR}: Stabilizing Self-Play by Mitigating Iteration-Collapse With One-Class Novelty Rewards}, author = {Lee, Seungyoo and Nam, Giung and Lee, Hyungi and Lee, Juho}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {64837--64861}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/lee26ay/lee26ay.pdf}, url = {https://proceedings.mlr.press/v306/lee26ay.html}, abstract = {Training large language models via self-play often suffers from a persistent iteration-collapse, where performance initially improves but subsequently regresses as training iterations increase. We analyze this phenomenon as arising from cross-iteration degeneration, where the task-generation distribution becomes increasingly confined to a narrow subset of familiar (seen) problems, weakening the effective learning signal and destabilizing training. To address this issue, we propose a plug-in approach that augments existing self-play pipelines with a one-class novelty reward. A Seen Detector trained on a historical buffer of previously used training problems identifies in-support instances and discourages redundant generation by the questioner, thereby steering exploration toward under-explored yet learnable regions. Experimental results show that the proposed method mitigates iteration-collapse during iterative training and yields consistent improvements.} }
Endnote
%0 Conference Paper %T OCNR: Stabilizing Self-Play by Mitigating Iteration-Collapse With One-Class Novelty Rewards %A Seungyoo Lee %A Giung Nam %A Hyungi Lee %A Juho Lee %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-lee26ay %I PMLR %P 64837--64861 %U https://proceedings.mlr.press/v306/lee26ay.html %V 306 %X Training large language models via self-play often suffers from a persistent iteration-collapse, where performance initially improves but subsequently regresses as training iterations increase. We analyze this phenomenon as arising from cross-iteration degeneration, where the task-generation distribution becomes increasingly confined to a narrow subset of familiar (seen) problems, weakening the effective learning signal and destabilizing training. To address this issue, we propose a plug-in approach that augments existing self-play pipelines with a one-class novelty reward. A Seen Detector trained on a historical buffer of previously used training problems identifies in-support instances and discourages redundant generation by the questioner, thereby steering exploration toward under-explored yet learnable regions. Experimental results show that the proposed method mitigates iteration-collapse during iterative training and yields consistent improvements.
APA
Lee, S., Nam, G., Lee, H. & Lee, J.. (2026). OCNR: Stabilizing Self-Play by Mitigating Iteration-Collapse With One-Class Novelty Rewards. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:64837-64861 Available from https://proceedings.mlr.press/v306/lee26ay.html.

Related Material