Clipping Low-Probability Tokens in SFT Yields a Generalizable Initialization for RL

Tian-Shuo Liu, Chengxing Jia, Haoyu Liu, Pengyuan Wang, Shiyuan Zhang, Jie Fu, Yang Yu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:76871-76896, 2026.

Abstract

Supervised Fine-Tuning (SFT) is a critical step for adapting Large Language Models (LLMs) to specialized domains, often serving as an initialization for subsequent reinforcement learning (RL). However, SFT can overfit a small set of expert data, harming generalization and eroding prior knowledge. This can limit downstream RL, which benefits from a strong, generalizable initialization for exploration. Here, we demonstrate that prior knowledge degradation primarily results from tokens in the expert data to which the base model assigns low probability. Specifically, these low-probability tokens represent a significant deviation from the model’s current prior knowledge. Due to the nature of the log-likelihood objective, they produce larger gradient magnitudes, which speed up adaptation to the new data but degrade generalization. In this paper, we study the token-wise clipping strategy, a commonly used trust-region method for bounding per-token updates. We find that it reshapes token-level learning priorities, promoting more progressive adaptation that fits the new data while preserving general abilities. Compared with standard SFT, clipping low-probability tokens reduces out-of-distribution forgetting by 11.54% and improves final RL performance by 7.09% across the agentic benchmarks. Moreover, latent-space analysis shows smaller representational drift under clipping, indicating that it provides a generalizable initialization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-liu26bm, title = {Clipping Low-Probability Tokens in {SFT} Yields a Generalizable Initialization for {RL}}, author = {Liu, Tian-Shuo and Jia, Chengxing and Liu, Haoyu and Wang, Pengyuan and Zhang, Shiyuan and Fu, Jie and Yu, Yang}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {76871--76896}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/liu26bm/liu26bm.pdf}, url = {https://proceedings.mlr.press/v306/liu26bm.html}, abstract = {Supervised Fine-Tuning (SFT) is a critical step for adapting Large Language Models (LLMs) to specialized domains, often serving as an initialization for subsequent reinforcement learning (RL). However, SFT can overfit a small set of expert data, harming generalization and eroding prior knowledge. This can limit downstream RL, which benefits from a strong, generalizable initialization for exploration. Here, we demonstrate that prior knowledge degradation primarily results from tokens in the expert data to which the base model assigns low probability. Specifically, these low-probability tokens represent a significant deviation from the model’s current prior knowledge. Due to the nature of the log-likelihood objective, they produce larger gradient magnitudes, which speed up adaptation to the new data but degrade generalization. In this paper, we study the token-wise clipping strategy, a commonly used trust-region method for bounding per-token updates. We find that it reshapes token-level learning priorities, promoting more progressive adaptation that fits the new data while preserving general abilities. Compared with standard SFT, clipping low-probability tokens reduces out-of-distribution forgetting by 11.54% and improves final RL performance by 7.09% across the agentic benchmarks. Moreover, latent-space analysis shows smaller representational drift under clipping, indicating that it provides a generalizable initialization.} }
Endnote
%0 Conference Paper %T Clipping Low-Probability Tokens in SFT Yields a Generalizable Initialization for RL %A Tian-Shuo Liu %A Chengxing Jia %A Haoyu Liu %A Pengyuan Wang %A Shiyuan Zhang %A Jie Fu %A Yang Yu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-liu26bm %I PMLR %P 76871--76896 %U https://proceedings.mlr.press/v306/liu26bm.html %V 306 %X Supervised Fine-Tuning (SFT) is a critical step for adapting Large Language Models (LLMs) to specialized domains, often serving as an initialization for subsequent reinforcement learning (RL). However, SFT can overfit a small set of expert data, harming generalization and eroding prior knowledge. This can limit downstream RL, which benefits from a strong, generalizable initialization for exploration. Here, we demonstrate that prior knowledge degradation primarily results from tokens in the expert data to which the base model assigns low probability. Specifically, these low-probability tokens represent a significant deviation from the model’s current prior knowledge. Due to the nature of the log-likelihood objective, they produce larger gradient magnitudes, which speed up adaptation to the new data but degrade generalization. In this paper, we study the token-wise clipping strategy, a commonly used trust-region method for bounding per-token updates. We find that it reshapes token-level learning priorities, promoting more progressive adaptation that fits the new data while preserving general abilities. Compared with standard SFT, clipping low-probability tokens reduces out-of-distribution forgetting by 11.54% and improves final RL performance by 7.09% across the agentic benchmarks. Moreover, latent-space analysis shows smaller representational drift under clipping, indicating that it provides a generalizable initialization.
APA
Liu, T., Jia, C., Liu, H., Wang, P., Zhang, S., Fu, J. & Yu, Y.. (2026). Clipping Low-Probability Tokens in SFT Yields a Generalizable Initialization for RL. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:76871-76896 Available from https://proceedings.mlr.press/v306/liu26bm.html.

Related Material