FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain

Rohan Deb, Kiran Koshy Thekumparampil, Kousha Kalantari, Gaurush Hiranandani, Shoham Sabach, Branislav Kveton
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:12924-12943, 2025.

Abstract

Supervised fine-tuning (SFT) is the most common way of adapting large language models (LLMs) to a new domain. In this paper, we improve the efficiency of SFT by selecting an informative subset of training examples. Specifically, for a fixed budget of training examples, which determines the computational cost of fine-tuning, we select those that maximize information gain, as measured by the Fisher information matrix of the SFT objective. We approximate it efficiently by linearization at the last layer of the LLM. Our approach is computationally efficient, analyzable, and performs well empirically. We demonstrate this on several problems, with both quantitative results and LLM-as-a-judge evaluations.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-deb25a, title = {{F}isher{SFT}: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain}, author = {Deb, Rohan and Thekumparampil, Kiran Koshy and Kalantari, Kousha and Hiranandani, Gaurush and Sabach, Shoham and Kveton, Branislav}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {12924--12943}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/deb25a/deb25a.pdf}, url = {https://proceedings.mlr.press/v267/deb25a.html}, abstract = {Supervised fine-tuning (SFT) is the most common way of adapting large language models (LLMs) to a new domain. In this paper, we improve the efficiency of SFT by selecting an informative subset of training examples. Specifically, for a fixed budget of training examples, which determines the computational cost of fine-tuning, we select those that maximize information gain, as measured by the Fisher information matrix of the SFT objective. We approximate it efficiently by linearization at the last layer of the LLM. Our approach is computationally efficient, analyzable, and performs well empirically. We demonstrate this on several problems, with both quantitative results and LLM-as-a-judge evaluations.} }
Endnote
%0 Conference Paper %T FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain %A Rohan Deb %A Kiran Koshy Thekumparampil %A Kousha Kalantari %A Gaurush Hiranandani %A Shoham Sabach %A Branislav Kveton %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-deb25a %I PMLR %P 12924--12943 %U https://proceedings.mlr.press/v267/deb25a.html %V 267 %X Supervised fine-tuning (SFT) is the most common way of adapting large language models (LLMs) to a new domain. In this paper, we improve the efficiency of SFT by selecting an informative subset of training examples. Specifically, for a fixed budget of training examples, which determines the computational cost of fine-tuning, we select those that maximize information gain, as measured by the Fisher information matrix of the SFT objective. We approximate it efficiently by linearization at the last layer of the LLM. Our approach is computationally efficient, analyzable, and performs well empirically. We demonstrate this on several problems, with both quantitative results and LLM-as-a-judge evaluations.
APA
Deb, R., Thekumparampil, K.K., Kalantari, K., Hiranandani, G., Sabach, S. & Kveton, B.. (2025). FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:12924-12943 Available from https://proceedings.mlr.press/v267/deb25a.html.

Related Material