Navigating the Pareto Frontier of Alignment: Spectrum-Adaptive Fine-Tuning for LLMs

Yaoyou Fan, Chao Zhang, Xiaoyu Tan, Chenxing Sun, Yu Yuan, Haoyu Feng, Lu Pan, Ke Zeng, Xunliang Cai
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:28960-28983, 2026.

Abstract

Supervised Fine-Tuning with Negative Log-Likelihood (NLL) remains the standard post-training paradigm for Large Language Models, yet it imposes a disproportionately aggressive update force on low-probability target tokens. This focus forces the model to prioritize minimizing the loss of difficult samples over optimizing the overall quality of the generation, often leading to unwarranted overconfidence. On the other hand, alternatives like Dynamic Fine-Tuning suffer from vanishing gradients on these tokens, which severely hinders the acquisition of new concepts. To bridge this gap, we propose Spectrum-Adaptive Fine-Tuning (SAFT), a unified framework that interpolates between the aggressive learning signal of NLL and the robust nature of probability-weighted optimization. By adaptively balancing these objectives, SAFT effectively mitigates outlier sensitivity without sacrificing learning efficiency. Empirically, our method achieves state-of-the-art performance on mathematical reasoning benchmarks, demonstrating superior generalization on out-of-distribution tasks. Furthermore, evaluations on general conversational alignment validate SAFT’s broad adaptability across diverse data regimes. Our code is available at https://github.com/sjtu-scx/SAFT.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-fan26j, title = {Navigating the Pareto Frontier of Alignment: Spectrum-Adaptive Fine-Tuning for {LLM}s}, author = {Fan, Yaoyou and Zhang, Chao and Tan, Xiaoyu and Sun, Chenxing and Yuan, Yu and Feng, Haoyu and Pan, Lu and Zeng, Ke and Cai, Xunliang}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {28960--28983}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/fan26j/fan26j.pdf}, url = {https://proceedings.mlr.press/v306/fan26j.html}, abstract = {Supervised Fine-Tuning with Negative Log-Likelihood (NLL) remains the standard post-training paradigm for Large Language Models, yet it imposes a disproportionately aggressive update force on low-probability target tokens. This focus forces the model to prioritize minimizing the loss of difficult samples over optimizing the overall quality of the generation, often leading to unwarranted overconfidence. On the other hand, alternatives like Dynamic Fine-Tuning suffer from vanishing gradients on these tokens, which severely hinders the acquisition of new concepts. To bridge this gap, we propose Spectrum-Adaptive Fine-Tuning (SAFT), a unified framework that interpolates between the aggressive learning signal of NLL and the robust nature of probability-weighted optimization. By adaptively balancing these objectives, SAFT effectively mitigates outlier sensitivity without sacrificing learning efficiency. Empirically, our method achieves state-of-the-art performance on mathematical reasoning benchmarks, demonstrating superior generalization on out-of-distribution tasks. Furthermore, evaluations on general conversational alignment validate SAFT’s broad adaptability across diverse data regimes. Our code is available at https://github.com/sjtu-scx/SAFT.} }
Endnote
%0 Conference Paper %T Navigating the Pareto Frontier of Alignment: Spectrum-Adaptive Fine-Tuning for LLMs %A Yaoyou Fan %A Chao Zhang %A Xiaoyu Tan %A Chenxing Sun %A Yu Yuan %A Haoyu Feng %A Lu Pan %A Ke Zeng %A Xunliang Cai %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-fan26j %I PMLR %P 28960--28983 %U https://proceedings.mlr.press/v306/fan26j.html %V 306 %X Supervised Fine-Tuning with Negative Log-Likelihood (NLL) remains the standard post-training paradigm for Large Language Models, yet it imposes a disproportionately aggressive update force on low-probability target tokens. This focus forces the model to prioritize minimizing the loss of difficult samples over optimizing the overall quality of the generation, often leading to unwarranted overconfidence. On the other hand, alternatives like Dynamic Fine-Tuning suffer from vanishing gradients on these tokens, which severely hinders the acquisition of new concepts. To bridge this gap, we propose Spectrum-Adaptive Fine-Tuning (SAFT), a unified framework that interpolates between the aggressive learning signal of NLL and the robust nature of probability-weighted optimization. By adaptively balancing these objectives, SAFT effectively mitigates outlier sensitivity without sacrificing learning efficiency. Empirically, our method achieves state-of-the-art performance on mathematical reasoning benchmarks, demonstrating superior generalization on out-of-distribution tasks. Furthermore, evaluations on general conversational alignment validate SAFT’s broad adaptability across diverse data regimes. Our code is available at https://github.com/sjtu-scx/SAFT.
APA
Fan, Y., Zhang, C., Tan, X., Sun, C., Yuan, Y., Feng, H., Pan, L., Zeng, K. & Cai, X.. (2026). Navigating the Pareto Frontier of Alignment: Spectrum-Adaptive Fine-Tuning for LLMs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:28960-28983 Available from https://proceedings.mlr.press/v306/fan26j.html.

Related Material