$\textitS$-SPPO: Semantic-Calibrated Self-Play Preference Optimization

Xiwen Chen, Wenhui Zhu, Jingjing Wang, Peijie Qiu, Zhipeng Wang, Huayu Li, Zhengxiao He, Xuanzhao Dong, Prayag Tiwari, Mingkun Xu, Yujian Xiong, Feng Luo, Abolfazl Razi, Brendan Hogan Rappazzo, Anderson Schneider, Yuriy Nevmyvaka
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:18694-18711, 2026.

Abstract

Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO). However, the standard Bradley-Terry instantiation of DPO is limited in modeling common departures from transitivity in human preferences. To address this, recent work has introduced Self-Play Preference Optimization (SPPO), which iteratively refines the policy by training on self-generated win-lose pairs. Our investigation, however, reveals a critical instability in SPPO: the optimization is prone to policy degeneration when the preference oracle assigns overly confident wins to semantically indistinguishable responses. To mitigate this, we propose $\textit{S}$-SPPO, a dual-space semantic calibration framework comprising: i) $\textit{Supervision Calibration}$ via semantic gating, which anneals win rate targets toward the maximum-entropy baseline as semantic overlap increases; and ii) $\textit{Representation Calibration}$ via latent repulsion to enforce geometric diversity to prevent manifold collapse and maintain latent diversity between chosen and rejected samples. Theoretically, we show that the calibration preserves the constant-sum game structure, facilitating convergence to a Nash Equilibrium. Empirically, $\textit{S}$-SPPO avoids the performance degradation seen in prior methods, achieving 52.19% win rate and 47.46% length-controlled win rate on AlpacaEval 2.0 with Llama-3-8B, without using additional human-annotated preferences during training.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26hc, title = {$\textit{S}$-{SPPO}: Semantic-Calibrated Self-Play Preference Optimization}, author = {Chen, Xiwen and Zhu, Wenhui and Wang, Jingjing and Qiu, Peijie and Wang, Zhipeng and Li, Huayu and He, Zhengxiao and Dong, Xuanzhao and Tiwari, Prayag and Xu, Mingkun and Xiong, Yujian and Luo, Feng and Razi, Abolfazl and Rappazzo, Brendan Hogan and Schneider, Anderson and Nevmyvaka, Yuriy}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {18694--18711}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26hc/chen26hc.pdf}, url = {https://proceedings.mlr.press/v306/chen26hc.html}, abstract = {Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO). However, the standard Bradley-Terry instantiation of DPO is limited in modeling common departures from transitivity in human preferences. To address this, recent work has introduced Self-Play Preference Optimization (SPPO), which iteratively refines the policy by training on self-generated win-lose pairs. Our investigation, however, reveals a critical instability in SPPO: the optimization is prone to policy degeneration when the preference oracle assigns overly confident wins to semantically indistinguishable responses. To mitigate this, we propose $\textit{S}$-SPPO, a dual-space semantic calibration framework comprising: i) $\textit{Supervision Calibration}$ via semantic gating, which anneals win rate targets toward the maximum-entropy baseline as semantic overlap increases; and ii) $\textit{Representation Calibration}$ via latent repulsion to enforce geometric diversity to prevent manifold collapse and maintain latent diversity between chosen and rejected samples. Theoretically, we show that the calibration preserves the constant-sum game structure, facilitating convergence to a Nash Equilibrium. Empirically, $\textit{S}$-SPPO avoids the performance degradation seen in prior methods, achieving 52.19% win rate and 47.46% length-controlled win rate on AlpacaEval 2.0 with Llama-3-8B, without using additional human-annotated preferences during training.} }
Endnote
%0 Conference Paper %T $\textitS$-SPPO: Semantic-Calibrated Self-Play Preference Optimization %A Xiwen Chen %A Wenhui Zhu %A Jingjing Wang %A Peijie Qiu %A Zhipeng Wang %A Huayu Li %A Zhengxiao He %A Xuanzhao Dong %A Prayag Tiwari %A Mingkun Xu %A Yujian Xiong %A Feng Luo %A Abolfazl Razi %A Brendan Hogan Rappazzo %A Anderson Schneider %A Yuriy Nevmyvaka %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26hc %I PMLR %P 18694--18711 %U https://proceedings.mlr.press/v306/chen26hc.html %V 306 %X Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO). However, the standard Bradley-Terry instantiation of DPO is limited in modeling common departures from transitivity in human preferences. To address this, recent work has introduced Self-Play Preference Optimization (SPPO), which iteratively refines the policy by training on self-generated win-lose pairs. Our investigation, however, reveals a critical instability in SPPO: the optimization is prone to policy degeneration when the preference oracle assigns overly confident wins to semantically indistinguishable responses. To mitigate this, we propose $\textit{S}$-SPPO, a dual-space semantic calibration framework comprising: i) $\textit{Supervision Calibration}$ via semantic gating, which anneals win rate targets toward the maximum-entropy baseline as semantic overlap increases; and ii) $\textit{Representation Calibration}$ via latent repulsion to enforce geometric diversity to prevent manifold collapse and maintain latent diversity between chosen and rejected samples. Theoretically, we show that the calibration preserves the constant-sum game structure, facilitating convergence to a Nash Equilibrium. Empirically, $\textit{S}$-SPPO avoids the performance degradation seen in prior methods, achieving 52.19% win rate and 47.46% length-controlled win rate on AlpacaEval 2.0 with Llama-3-8B, without using additional human-annotated preferences during training.
APA
Chen, X., Zhu, W., Wang, J., Qiu, P., Wang, Z., Li, H., He, Z., Dong, X., Tiwari, P., Xu, M., Xiong, Y., Luo, F., Razi, A., Rappazzo, B.H., Schneider, A. & Nevmyvaka, Y.. (2026). $\textitS$-SPPO: Semantic-Calibrated Self-Play Preference Optimization. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:18694-18711 Available from https://proceedings.mlr.press/v306/chen26hc.html.

Related Material