The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning

Will Hawkins, Kai Rawal, Jonathan Rystrøm, Stratis Tsirtsis, Zihao Fu, Greta Warren, Ryan Brown, Eoin D. Delaney, Sandra Wachter, Brent Mittelstadt, Chris Russell
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:40933-40950, 2026.

Abstract

Fine-tuning a large language model is a ubiquitous method for enhancing its capability on a specific downstream task. However, prior work has shown that this increase in capability comes with a cost: it can increase a model’s tendency to respond to unsafe adversarial prompts, even when fine-tuning with non-adversarial data. We present the first comprehensive empirical study of this phenomenon in multilingual settings by fine-tuning Llama-3.2, Qwen3, and Gemma-3 models using benign data translated across nine languages. We find that safety outcomes are highly sensitive to both the choice of fine-tuning language and the evaluation language, with adversarial compliance rates increasing four-fold in some settings. Multilingual safety drift is decoupled from general capability metrics, and occurs heterogeneously across languages and models. Fine-tuning in non-English languages often induces smaller internal representational drifts than English, but these shifts lead models to default to either exaggerated compliance or refusal. As such, assessing fine-tuning impacts solely in English provides inadequate assurance for deployment. To facilitate further research into these cross-lingual safety blind spots, we release the Multilingual-Benign-Tune dataset and the SORRY-Bench-Multilingual evaluation suite.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-hawkins26a, title = {The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning}, author = {Hawkins, Will and Rawal, Kai and Rystr{\o}m, Jonathan and Tsirtsis, Stratis and Fu, Zihao and Warren, Greta and Brown, Ryan and Delaney, Eoin D. and Wachter, Sandra and Mittelstadt, Brent and Russell, Chris}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {40933--40950}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/hawkins26a/hawkins26a.pdf}, url = {https://proceedings.mlr.press/v306/hawkins26a.html}, abstract = {Fine-tuning a large language model is a ubiquitous method for enhancing its capability on a specific downstream task. However, prior work has shown that this increase in capability comes with a cost: it can increase a model’s tendency to respond to unsafe adversarial prompts, even when fine-tuning with non-adversarial data. We present the first comprehensive empirical study of this phenomenon in multilingual settings by fine-tuning Llama-3.2, Qwen3, and Gemma-3 models using benign data translated across nine languages. We find that safety outcomes are highly sensitive to both the choice of fine-tuning language and the evaluation language, with adversarial compliance rates increasing four-fold in some settings. Multilingual safety drift is decoupled from general capability metrics, and occurs heterogeneously across languages and models. Fine-tuning in non-English languages often induces smaller internal representational drifts than English, but these shifts lead models to default to either exaggerated compliance or refusal. As such, assessing fine-tuning impacts solely in English provides inadequate assurance for deployment. To facilitate further research into these cross-lingual safety blind spots, we release the Multilingual-Benign-Tune dataset and the SORRY-Bench-Multilingual evaluation suite.} }
Endnote
%0 Conference Paper %T The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning %A Will Hawkins %A Kai Rawal %A Jonathan Rystrøm %A Stratis Tsirtsis %A Zihao Fu %A Greta Warren %A Ryan Brown %A Eoin D. Delaney %A Sandra Wachter %A Brent Mittelstadt %A Chris Russell %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-hawkins26a %I PMLR %P 40933--40950 %U https://proceedings.mlr.press/v306/hawkins26a.html %V 306 %X Fine-tuning a large language model is a ubiquitous method for enhancing its capability on a specific downstream task. However, prior work has shown that this increase in capability comes with a cost: it can increase a model’s tendency to respond to unsafe adversarial prompts, even when fine-tuning with non-adversarial data. We present the first comprehensive empirical study of this phenomenon in multilingual settings by fine-tuning Llama-3.2, Qwen3, and Gemma-3 models using benign data translated across nine languages. We find that safety outcomes are highly sensitive to both the choice of fine-tuning language and the evaluation language, with adversarial compliance rates increasing four-fold in some settings. Multilingual safety drift is decoupled from general capability metrics, and occurs heterogeneously across languages and models. Fine-tuning in non-English languages often induces smaller internal representational drifts than English, but these shifts lead models to default to either exaggerated compliance or refusal. As such, assessing fine-tuning impacts solely in English provides inadequate assurance for deployment. To facilitate further research into these cross-lingual safety blind spots, we release the Multilingual-Benign-Tune dataset and the SORRY-Bench-Multilingual evaluation suite.
APA
Hawkins, W., Rawal, K., Rystrøm, J., Tsirtsis, S., Fu, Z., Warren, G., Brown, R., Delaney, E.D., Wachter, S., Mittelstadt, B. & Russell, C.. (2026). The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:40933-40950 Available from https://proceedings.mlr.press/v306/hawkins26a.html.

Related Material