Unbiased Reward Modeling from Implicit Feedback for LLM Alignment

Hao Wang, Haocheng Yang, Licheng Pan, Zhichao Chen, Xiaoxi Li, Yinuo Wang, Lei Shen, Yuan Lu, Haoxuan Li, Zhouchen Lin
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:131692-131706, 2026.

Abstract

Despite the success of reinforcement learning from human feedback (RLHF), existing reward modeling methods largely rely on explicit feedback, which is costly to collect and difficult to scale. This work studies implicit reward modeling, learning reward models from implicit user feedback, such as clicks, copies and skips. While scalable and cost-effective, implicit feedback poses two key challenges: (1) It lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; (2) It suffers from selection bias, where responses have heterogeneous propensities to elicit feedback, which further obscures definitive negative samples. To address these challenges, we propose ImplicitRM, which learns unbiased reward models from implicit feedback. It stratifies training samples into four latent groups using a stratification model and derives a likelihood-maximization objective that is theoretically unbiased, thereby addressing both challenges. Experiments across diverse LLM backbones and benchmark datasets validate that ImplicitRM learns accurate reward models from implicit feedback and improves performance on downstream RLHF tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-wang26jq, title = {Unbiased Reward Modeling from Implicit Feedback for {LLM} Alignment}, author = {Wang, Hao and Yang, Haocheng and Pan, Licheng and Chen, Zhichao and Li, Xiaoxi and Wang, Yinuo and Shen, Lei and Lu, Yuan and Li, Haoxuan and Lin, Zhouchen}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {131692--131706}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wang26jq/wang26jq.pdf}, url = {https://proceedings.mlr.press/v306/wang26jq.html}, abstract = {Despite the success of reinforcement learning from human feedback (RLHF), existing reward modeling methods largely rely on explicit feedback, which is costly to collect and difficult to scale. This work studies implicit reward modeling, learning reward models from implicit user feedback, such as clicks, copies and skips. While scalable and cost-effective, implicit feedback poses two key challenges: (1) It lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; (2) It suffers from selection bias, where responses have heterogeneous propensities to elicit feedback, which further obscures definitive negative samples. To address these challenges, we propose ImplicitRM, which learns unbiased reward models from implicit feedback. It stratifies training samples into four latent groups using a stratification model and derives a likelihood-maximization objective that is theoretically unbiased, thereby addressing both challenges. Experiments across diverse LLM backbones and benchmark datasets validate that ImplicitRM learns accurate reward models from implicit feedback and improves performance on downstream RLHF tasks.} }
Endnote
%0 Conference Paper %T Unbiased Reward Modeling from Implicit Feedback for LLM Alignment %A Hao Wang %A Haocheng Yang %A Licheng Pan %A Zhichao Chen %A Xiaoxi Li %A Yinuo Wang %A Lei Shen %A Yuan Lu %A Haoxuan Li %A Zhouchen Lin %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wang26jq %I PMLR %P 131692--131706 %U https://proceedings.mlr.press/v306/wang26jq.html %V 306 %X Despite the success of reinforcement learning from human feedback (RLHF), existing reward modeling methods largely rely on explicit feedback, which is costly to collect and difficult to scale. This work studies implicit reward modeling, learning reward models from implicit user feedback, such as clicks, copies and skips. While scalable and cost-effective, implicit feedback poses two key challenges: (1) It lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; (2) It suffers from selection bias, where responses have heterogeneous propensities to elicit feedback, which further obscures definitive negative samples. To address these challenges, we propose ImplicitRM, which learns unbiased reward models from implicit feedback. It stratifies training samples into four latent groups using a stratification model and derives a likelihood-maximization objective that is theoretically unbiased, thereby addressing both challenges. Experiments across diverse LLM backbones and benchmark datasets validate that ImplicitRM learns accurate reward models from implicit feedback and improves performance on downstream RLHF tasks.
APA
Wang, H., Yang, H., Pan, L., Chen, Z., Li, X., Wang, Y., Shen, L., Lu, Y., Li, H. & Lin, Z.. (2026). Unbiased Reward Modeling from Implicit Feedback for LLM Alignment. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:131692-131706 Available from https://proceedings.mlr.press/v306/wang26jq.html.

Related Material