TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van, Duy Minh Ho Nguyen, Khoa D Doan, Trung Le
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:92636-92658, 2026.

Abstract

Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley–Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO), which posits a token-level Bradley–Terry preference model over next-token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-nguyen26l, title = {{T}oken{R}atio: Principled Token-Level Preference Optimization via Ratio Matching}, author = {Nguyen, Truong and Nguyen, Tien-Phat and Van, Linh Ngo and Nguyen, Duy Minh Ho and Doan, Khoa D and Le, Trung}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {92636--92658}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/nguyen26l/nguyen26l.pdf}, url = {https://proceedings.mlr.press/v306/nguyen26l.html}, abstract = {Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley–Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO), which posits a token-level Bradley–Terry preference model over next-token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines.} }
Endnote
%0 Conference Paper %T TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching %A Truong Nguyen %A Tien-Phat Nguyen %A Linh Ngo Van %A Duy Minh Ho Nguyen %A Khoa D Doan %A Trung Le %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-nguyen26l %I PMLR %P 92636--92658 %U https://proceedings.mlr.press/v306/nguyen26l.html %V 306 %X Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley–Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO), which posits a token-level Bradley–Terry preference model over next-token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines.
APA
Nguyen, T., Nguyen, T., Van, L.N., Nguyen, D.M.H., Doan, K.D. & Le, T.. (2026). TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:92636-92658 Available from https://proceedings.mlr.press/v306/nguyen26l.html.

Related Material