Suspicious Alignment of SGD:A Fine-Grained Step Size Condition Analysis

Shenyang Deng, Boyao Liao, Zhuoli Ouyang, Tianyu Pang, Minhak Song, Yaoqing Yang
Proceedings of The 37th International Conference on Algorithmic Learning Theory, PMLR 313:1-66, 2026.

Abstract

This paper explores the suspicious alignment phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and bulk subspaces. This phenomenon describes the behavior of gradient alignment in SGD updates. Specifically, during the initial phase of SGD updates, the alignment between the gradient and the dominant subspace tends to decrease. Subsequently, it enters a rising phase and eventually stabilizes in a high-alignment phase. The alignment is considered “suspicious” because, paradoxically, the projected gradient update along this highly-aligned dominant subspace proves ineffective at reducing the loss. The focus of this work is to give a fine-grained analysis in a high-dimensional quadratic setup about how step size selection produces this phenomenon. Our primary contribution can be summarized as follows: We propose a step-size condition theory revealing that in low-alignment regimes, an adaptive critical step size $\eta_t^\ast$ separates alignment-decreasing ($\eta_t < \eta_t^\ast$) from alignment-increasing ($\eta_t > \eta_t^\ast$) regimes, whereas in high-alignment regimes, the alignment is self-correcting and decreases regardless of the step size. We further show that under sufficient ill-conditioning, a step size interval exists where the loss in the bulk subspace decreases while the loss in the dominant subspace increases, which explains a recent empirical observation that projecting gradient updates to the dominant subspace is ineffective. Finally, based on this adaptive step-size theory, we prove that for a constant step size and large initialization, SGD exhibits this distinct two-phase behavior: an initial alignment-decreasing phase, followed by stabilization at high alignment.

Cite this Paper

BibTeX
@InProceedings{pmlr-v313-deng26a, title = {Suspicious Alignment of SGD:A Fine-Grained Step Size Condition Analysis}, author = {Deng, Shenyang and Liao, Boyao and Ouyang, Zhuoli and Pang, Tianyu and Song, Minhak and Yang, Yaoqing}, booktitle = {Proceedings of The 37th International Conference on Algorithmic Learning Theory}, pages = {1--66}, year = {2026}, editor = {Telgarsky, Matus and Ullman, Jonathan}, volume = {313}, series = {Proceedings of Machine Learning Research}, month = {23--26 Feb}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v313/main/assets/deng26a/deng26a.pdf}, url = {https://proceedings.mlr.press/v313/deng26a.html}, abstract = {This paper explores the suspicious alignment phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and bulk subspaces. This phenomenon describes the behavior of gradient alignment in SGD updates. Specifically, during the initial phase of SGD updates, the alignment between the gradient and the dominant subspace tends to decrease. Subsequently, it enters a rising phase and eventually stabilizes in a high-alignment phase. The alignment is considered “suspicious” because, paradoxically, the projected gradient update along this highly-aligned dominant subspace proves ineffective at reducing the loss. The focus of this work is to give a fine-grained analysis in a high-dimensional quadratic setup about how step size selection produces this phenomenon. Our primary contribution can be summarized as follows: We propose a step-size condition theory revealing that in low-alignment regimes, an adaptive critical step size $\eta_t^\ast$ separates alignment-decreasing ($\eta_t < \eta_t^\ast$) from alignment-increasing ($\eta_t > \eta_t^\ast$) regimes, whereas in high-alignment regimes, the alignment is self-correcting and decreases regardless of the step size. We further show that under sufficient ill-conditioning, a step size interval exists where the loss in the bulk subspace decreases while the loss in the dominant subspace increases, which explains a recent empirical observation that projecting gradient updates to the dominant subspace is ineffective. Finally, based on this adaptive step-size theory, we prove that for a constant step size and large initialization, SGD exhibits this distinct two-phase behavior: an initial alignment-decreasing phase, followed by stabilization at high alignment.} }
Endnote
%0 Conference Paper %T Suspicious Alignment of SGD:A Fine-Grained Step Size Condition Analysis %A Shenyang Deng %A Boyao Liao %A Zhuoli Ouyang %A Tianyu Pang %A Minhak Song %A Yaoqing Yang %B Proceedings of The 37th International Conference on Algorithmic Learning Theory %C Proceedings of Machine Learning Research %D 2026 %E Matus Telgarsky %E Jonathan Ullman %F pmlr-v313-deng26a %I PMLR %P 1--66 %U https://proceedings.mlr.press/v313/deng26a.html %V 313 %X This paper explores the suspicious alignment phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and bulk subspaces. This phenomenon describes the behavior of gradient alignment in SGD updates. Specifically, during the initial phase of SGD updates, the alignment between the gradient and the dominant subspace tends to decrease. Subsequently, it enters a rising phase and eventually stabilizes in a high-alignment phase. The alignment is considered “suspicious” because, paradoxically, the projected gradient update along this highly-aligned dominant subspace proves ineffective at reducing the loss. The focus of this work is to give a fine-grained analysis in a high-dimensional quadratic setup about how step size selection produces this phenomenon. Our primary contribution can be summarized as follows: We propose a step-size condition theory revealing that in low-alignment regimes, an adaptive critical step size $\eta_t^\ast$ separates alignment-decreasing ($\eta_t < \eta_t^\ast$) from alignment-increasing ($\eta_t > \eta_t^\ast$) regimes, whereas in high-alignment regimes, the alignment is self-correcting and decreases regardless of the step size. We further show that under sufficient ill-conditioning, a step size interval exists where the loss in the bulk subspace decreases while the loss in the dominant subspace increases, which explains a recent empirical observation that projecting gradient updates to the dominant subspace is ineffective. Finally, based on this adaptive step-size theory, we prove that for a constant step size and large initialization, SGD exhibits this distinct two-phase behavior: an initial alignment-decreasing phase, followed by stabilization at high alignment.
APA
Deng, S., Liao, B., Ouyang, Z., Pang, T., Song, M. & Yang, Y.. (2026). Suspicious Alignment of SGD:A Fine-Grained Step Size Condition Analysis. Proceedings of The 37th International Conference on Algorithmic Learning Theory, in Proceedings of Machine Learning Research 313:1-66 Available from https://proceedings.mlr.press/v313/deng26a.html.

Related Material