Beyond Static Allocation: Dynamic Sensitivity-Aware Fine-Tuning for Vision Transformers

Yuanyang Cao, Xichun Liu, Fuwei Zhang, Shangqi Deng, Ziyang Ren, Jianji Wang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:11392-11417, 2026.

Abstract

Existing Parameter-Efficient Fine-Tuning (PEFT) methods are fundamentally constrained by a static allocation paradigm, which overlooks the model’s evolving optimization priorities during training. To address this, we introduce Dynamic Adaptive Fine-tuning (DAF), a novel framework that periodically evaluates and reconfigures the trainable structure based on a context-aware decoupled sensitivity analysis. DAF employs a Rebuild-and-Refocus strategy to preserve learned knowledge by freezing outdated modules while decisively reallocating the parameter budget to newly identified critical regions. Extensive experiments on challenging vision benchmarks demonstrate that DAF significantly outperforms mainstream static PEFT methods and achieves superior performance and efficiency, particularly under extreme parameter budgets. Our work fundamentally challenges the static nature of the field, offering a more intelligent and efficient paradigm for adapting large pretrained models. The code is available at https://github.com/E-green11/DAF.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-cao26o, title = {Beyond Static Allocation: Dynamic Sensitivity-Aware Fine-Tuning for Vision Transformers}, author = {Cao, Yuanyang and Liu, Xichun and Zhang, Fuwei and Deng, Shangqi and Ren, Ziyang and Wang, Jianji}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {11392--11417}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/cao26o/cao26o.pdf}, url = {https://proceedings.mlr.press/v306/cao26o.html}, abstract = {Existing Parameter-Efficient Fine-Tuning (PEFT) methods are fundamentally constrained by a static allocation paradigm, which overlooks the model’s evolving optimization priorities during training. To address this, we introduce Dynamic Adaptive Fine-tuning (DAF), a novel framework that periodically evaluates and reconfigures the trainable structure based on a context-aware decoupled sensitivity analysis. DAF employs a Rebuild-and-Refocus strategy to preserve learned knowledge by freezing outdated modules while decisively reallocating the parameter budget to newly identified critical regions. Extensive experiments on challenging vision benchmarks demonstrate that DAF significantly outperforms mainstream static PEFT methods and achieves superior performance and efficiency, particularly under extreme parameter budgets. Our work fundamentally challenges the static nature of the field, offering a more intelligent and efficient paradigm for adapting large pretrained models. The code is available at https://github.com/E-green11/DAF.} }
Endnote
%0 Conference Paper %T Beyond Static Allocation: Dynamic Sensitivity-Aware Fine-Tuning for Vision Transformers %A Yuanyang Cao %A Xichun Liu %A Fuwei Zhang %A Shangqi Deng %A Ziyang Ren %A Jianji Wang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-cao26o %I PMLR %P 11392--11417 %U https://proceedings.mlr.press/v306/cao26o.html %V 306 %X Existing Parameter-Efficient Fine-Tuning (PEFT) methods are fundamentally constrained by a static allocation paradigm, which overlooks the model’s evolving optimization priorities during training. To address this, we introduce Dynamic Adaptive Fine-tuning (DAF), a novel framework that periodically evaluates and reconfigures the trainable structure based on a context-aware decoupled sensitivity analysis. DAF employs a Rebuild-and-Refocus strategy to preserve learned knowledge by freezing outdated modules while decisively reallocating the parameter budget to newly identified critical regions. Extensive experiments on challenging vision benchmarks demonstrate that DAF significantly outperforms mainstream static PEFT methods and achieves superior performance and efficiency, particularly under extreme parameter budgets. Our work fundamentally challenges the static nature of the field, offering a more intelligent and efficient paradigm for adapting large pretrained models. The code is available at https://github.com/E-green11/DAF.
APA
Cao, Y., Liu, X., Zhang, F., Deng, S., Ren, Z. & Wang, J.. (2026). Beyond Static Allocation: Dynamic Sensitivity-Aware Fine-Tuning for Vision Transformers. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:11392-11417 Available from https://proceedings.mlr.press/v306/cao26o.html.

Related Material