A Regret Minimization Framework on Preference Learning in Large Language Models

Suhwan Kim, Taehyun Cho, Geon-Hyeong Kim, Yu Jin Kim, Youngsoo Jang, Moontae Lee, Jungwoo Lee
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:57276-57308, 2026.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has enabled progress on reasoning-intensive tasks by relying on task-specific verifiers that provide automated correctness signals. However, many realistic language tasks are difficult to equip with reliable verifiers, motivating a growing reliance on reinforcement learning from human feedback (RLHF). In this setting, we argue that a closer examination of how human feedback should be interpreted is essential. We introduce Regret-based Preference Optimization (RePO), which reframes RLHF through regret minimization rather than reward maximization. Human preferences are often shaped by prospective anticipation of outcomes and counterfactual comparisons to alternative behaviors, rather than by immediate, outcome-independent utility. RePO captures this structure by modeling preferences as behavior-conditioned assessments of relative suboptimality. Within a KL-regularized reinforcement learning framework, RePO admits a closed-form policy update compatible with direct preference optimization. Experiments on mathematical reasoning benchmarks and human-annotated preference datasets demonstrate consistent performance gains, indicating that regret-based preference learning is an effective and human-aligned approach for training large language models.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-kim26g, title = {A Regret Minimization Framework on Preference Learning in Large Language Models}, author = {Kim, Suhwan and Cho, Taehyun and Kim, Geon-Hyeong and Kim, Yu Jin and Jang, Youngsoo and Lee, Moontae and Lee, Jungwoo}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {57276--57308}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/kim26g/kim26g.pdf}, url = {https://proceedings.mlr.press/v306/kim26g.html}, abstract = {Reinforcement learning with verifiable rewards (RLVR) has enabled progress on reasoning-intensive tasks by relying on task-specific verifiers that provide automated correctness signals. However, many realistic language tasks are difficult to equip with reliable verifiers, motivating a growing reliance on reinforcement learning from human feedback (RLHF). In this setting, we argue that a closer examination of how human feedback should be interpreted is essential. We introduce Regret-based Preference Optimization (RePO), which reframes RLHF through regret minimization rather than reward maximization. Human preferences are often shaped by prospective anticipation of outcomes and counterfactual comparisons to alternative behaviors, rather than by immediate, outcome-independent utility. RePO captures this structure by modeling preferences as behavior-conditioned assessments of relative suboptimality. Within a KL-regularized reinforcement learning framework, RePO admits a closed-form policy update compatible with direct preference optimization. Experiments on mathematical reasoning benchmarks and human-annotated preference datasets demonstrate consistent performance gains, indicating that regret-based preference learning is an effective and human-aligned approach for training large language models.} }
Endnote
%0 Conference Paper %T A Regret Minimization Framework on Preference Learning in Large Language Models %A Suhwan Kim %A Taehyun Cho %A Geon-Hyeong Kim %A Yu Jin Kim %A Youngsoo Jang %A Moontae Lee %A Jungwoo Lee %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-kim26g %I PMLR %P 57276--57308 %U https://proceedings.mlr.press/v306/kim26g.html %V 306 %X Reinforcement learning with verifiable rewards (RLVR) has enabled progress on reasoning-intensive tasks by relying on task-specific verifiers that provide automated correctness signals. However, many realistic language tasks are difficult to equip with reliable verifiers, motivating a growing reliance on reinforcement learning from human feedback (RLHF). In this setting, we argue that a closer examination of how human feedback should be interpreted is essential. We introduce Regret-based Preference Optimization (RePO), which reframes RLHF through regret minimization rather than reward maximization. Human preferences are often shaped by prospective anticipation of outcomes and counterfactual comparisons to alternative behaviors, rather than by immediate, outcome-independent utility. RePO captures this structure by modeling preferences as behavior-conditioned assessments of relative suboptimality. Within a KL-regularized reinforcement learning framework, RePO admits a closed-form policy update compatible with direct preference optimization. Experiments on mathematical reasoning benchmarks and human-annotated preference datasets demonstrate consistent performance gains, indicating that regret-based preference learning is an effective and human-aligned approach for training large language models.
APA
Kim, S., Cho, T., Kim, G., Kim, Y.J., Jang, Y., Lee, M. & Lee, J.. (2026). A Regret Minimization Framework on Preference Learning in Large Language Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:57276-57308 Available from https://proceedings.mlr.press/v306/kim26g.html.

Related Material