T-POP: Test-Time Personalization with Online Preference Feedback

Zikun Qu, Min Zhang, Mingze Kong, Xiang Li, Zhiwei Shang, Zhiyong Wang, Yikun Ban, Shuang Qiu, Yao Shu, Zhongxiang Dai
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:102732-102750, 2026.

Abstract

Personalizing large language models (LLMs) to individual user preferences is a critical step beyond generating generically helpful responses. However, current personalization methods are ill-suited for new users, as they typically require either slow, resource-intensive fine-tuning or a substantial amount of pre-existing user data, creating a significant cold-start problem. To address this challenge, we introduce a new paradigm for real-time personalization by learning from online pairwise preference feedback collected during text generation. We propose T-POP (Test-Time Personalization with Online Preference Feedback), a novel algorithm that synergistically combines test-time alignment with dueling bandits. Without updating the LLM parameters, T-POP steers the decoding process of a frozen LLM by learning a reward function online that captures user preferences. By leveraging dueling bandits, T-POP intelligently queries the user to efficiently balance between exploring their preferences and exploiting the learned knowledge to generate personalized text. Extensive experiments demonstrate that T-POP achieves rapid and data-efficient personalization, significantly outperforming existing baselines and showing consistent improvement with more user interactions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-qu26i, title = {T-{POP}: Test-Time Personalization with Online Preference Feedback}, author = {Qu, Zikun and Zhang, Min and Kong, Mingze and Li, Xiang and Shang, Zhiwei and Wang, Zhiyong and Ban, Yikun and Qiu, Shuang and Shu, Yao and Dai, Zhongxiang}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {102732--102750}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/qu26i/qu26i.pdf}, url = {https://proceedings.mlr.press/v306/qu26i.html}, abstract = {Personalizing large language models (LLMs) to individual user preferences is a critical step beyond generating generically helpful responses. However, current personalization methods are ill-suited for new users, as they typically require either slow, resource-intensive fine-tuning or a substantial amount of pre-existing user data, creating a significant cold-start problem. To address this challenge, we introduce a new paradigm for real-time personalization by learning from online pairwise preference feedback collected during text generation. We propose T-POP (Test-Time Personalization with Online Preference Feedback), a novel algorithm that synergistically combines test-time alignment with dueling bandits. Without updating the LLM parameters, T-POP steers the decoding process of a frozen LLM by learning a reward function online that captures user preferences. By leveraging dueling bandits, T-POP intelligently queries the user to efficiently balance between exploring their preferences and exploiting the learned knowledge to generate personalized text. Extensive experiments demonstrate that T-POP achieves rapid and data-efficient personalization, significantly outperforming existing baselines and showing consistent improvement with more user interactions.} }
Endnote
%0 Conference Paper %T T-POP: Test-Time Personalization with Online Preference Feedback %A Zikun Qu %A Min Zhang %A Mingze Kong %A Xiang Li %A Zhiwei Shang %A Zhiyong Wang %A Yikun Ban %A Shuang Qiu %A Yao Shu %A Zhongxiang Dai %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-qu26i %I PMLR %P 102732--102750 %U https://proceedings.mlr.press/v306/qu26i.html %V 306 %X Personalizing large language models (LLMs) to individual user preferences is a critical step beyond generating generically helpful responses. However, current personalization methods are ill-suited for new users, as they typically require either slow, resource-intensive fine-tuning or a substantial amount of pre-existing user data, creating a significant cold-start problem. To address this challenge, we introduce a new paradigm for real-time personalization by learning from online pairwise preference feedback collected during text generation. We propose T-POP (Test-Time Personalization with Online Preference Feedback), a novel algorithm that synergistically combines test-time alignment with dueling bandits. Without updating the LLM parameters, T-POP steers the decoding process of a frozen LLM by learning a reward function online that captures user preferences. By leveraging dueling bandits, T-POP intelligently queries the user to efficiently balance between exploring their preferences and exploiting the learned knowledge to generate personalized text. Extensive experiments demonstrate that T-POP achieves rapid and data-efficient personalization, significantly outperforming existing baselines and showing consistent improvement with more user interactions.
APA
Qu, Z., Zhang, M., Kong, M., Li, X., Shang, Z., Wang, Z., Ban, Y., Qiu, S., Shu, Y. & Dai, Z.. (2026). T-POP: Test-Time Personalization with Online Preference Feedback. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:102732-102750 Available from https://proceedings.mlr.press/v306/qu26i.html.

Related Material