Towards Understanding Steering Strength

Magamed Taimeskhanov, Samuel Vaiter, Damien Garreau
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:118084-118133, 2026.

Abstract

A popular approach to post-training control of large language models (LLMs) is the steering of intermediate latent representations. Namely, identify a well-chosen direction depending on the task at hand and perturbs representations along this direction at inference time. While many propositions exist to pick this direction, considerably less is understood about how to choose the magnitude of the move, whereas its importance is clear: too little and the intended behavior does not emerge, too much and the model’s performance degrades beyond repair. In this work, we propose the first theoretical analysis of steering strength. We characterize its effect on next token probability, presence of a concept, and cross-entropy, deriving precise qualitative laws governing these quantities. Our analysis reveals surprising behaviors, including non-monotonic effects of steering strength. We validate our theoretical predictions empirically on eleven language models, ranging from a small GPT architecture to modern models.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-taimeskhanov26a, title = {Towards Understanding Steering Strength}, author = {Taimeskhanov, Magamed and Vaiter, Samuel and Garreau, Damien}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {118084--118133}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/taimeskhanov26a/taimeskhanov26a.pdf}, url = {https://proceedings.mlr.press/v306/taimeskhanov26a.html}, abstract = {A popular approach to post-training control of large language models (LLMs) is the steering of intermediate latent representations. Namely, identify a well-chosen direction depending on the task at hand and perturbs representations along this direction at inference time. While many propositions exist to pick this direction, considerably less is understood about how to choose the magnitude of the move, whereas its importance is clear: too little and the intended behavior does not emerge, too much and the model’s performance degrades beyond repair. In this work, we propose the first theoretical analysis of steering strength. We characterize its effect on next token probability, presence of a concept, and cross-entropy, deriving precise qualitative laws governing these quantities. Our analysis reveals surprising behaviors, including non-monotonic effects of steering strength. We validate our theoretical predictions empirically on eleven language models, ranging from a small GPT architecture to modern models.} }
Endnote
%0 Conference Paper %T Towards Understanding Steering Strength %A Magamed Taimeskhanov %A Samuel Vaiter %A Damien Garreau %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-taimeskhanov26a %I PMLR %P 118084--118133 %U https://proceedings.mlr.press/v306/taimeskhanov26a.html %V 306 %X A popular approach to post-training control of large language models (LLMs) is the steering of intermediate latent representations. Namely, identify a well-chosen direction depending on the task at hand and perturbs representations along this direction at inference time. While many propositions exist to pick this direction, considerably less is understood about how to choose the magnitude of the move, whereas its importance is clear: too little and the intended behavior does not emerge, too much and the model’s performance degrades beyond repair. In this work, we propose the first theoretical analysis of steering strength. We characterize its effect on next token probability, presence of a concept, and cross-entropy, deriving precise qualitative laws governing these quantities. Our analysis reveals surprising behaviors, including non-monotonic effects of steering strength. We validate our theoretical predictions empirically on eleven language models, ranging from a small GPT architecture to modern models.
APA
Taimeskhanov, M., Vaiter, S. & Garreau, D.. (2026). Towards Understanding Steering Strength. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:118084-118133 Available from https://proceedings.mlr.press/v306/taimeskhanov26a.html.

Related Material