Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy

Zhendong Huang, Hengjie Cao, Fang Dong, Ruijun Huang, Mengyi Chen, Yifeng Yang, Xin Zhang, Anrui Chen, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert P. Dick, Yuan Cheng, Tun Lu, Fan Yang, Li Shang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:46441-46456, 2026.

Abstract

Gradient signals in LLM training are highly anisotropic: recurrent linguistic structure concentrates energy into a small set of dominant spectral directions, while context-specific information resides in a long tail. We show that this spike–tail separation persists throughout training, with the spike occupying only about 1.5% of directions yet dominating optimizer statistics. This dominance suppresses tail learning by contracting tail updates through second-moment normalization and tightening the globally stable learning-rate bound. Motivated by this analysis, we propose Spectra, a spike-aware optimizer that suppresses the dominant low-rank spike subspace without amplifying the noise-sensitive spectral tail. Spectra tracks the spike subspace via cached, warm-started power iteration and applies low-rank spectral shaping with negligible overhead and substantially reduced optimizer-state memory. Across Qwen3-0.6B trained on 100B tokens and LLaMA3-8B trained on 50B tokens, Spectra achieves the lowest final validation loss, improving average downstream accuracy by 1.41/0.89 and 1.62/0.66 points over AdamW/Muon, respectively. For wall-clock convergence, Spectra reaches matched loss targets up to 1.31$\times$, 1.34$\times$, and 1.24$\times$ faster than AdamW on Qwen3-0.6B, Qwen3-2B-A0.8B, and Qwen3-8B; its speedup over Muon grows as model scale increases from 0.6B to 8B. For computational efficiency, Spectra is 5.1$\times$ faster than Muon in optimizer processing time, cuts optimizer-state memory by 49.25%, and achieves the lowest measured end-to-end per-step runtime. Spectra’s Megatron integration is available at https://github.com/kimmichtank/spectra.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-huang26d, title = {Spectra: Rethinking Optimizers for {LLM}s Under Spectral Anisotropy}, author = {Huang, Zhendong and Cao, Hengjie and Dong, Fang and Huang, Ruijun and Chen, Mengyi and Yang, Yifeng and Zhang, Xin and Chen, Anrui and Dong, Mingzhi and Wang, Yujiang and Hou, Jinlong and Lv, Qin and Dick, Robert P. and Cheng, Yuan and Lu, Tun and Yang, Fan and Shang, Li}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {46441--46456}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/huang26d/huang26d.pdf}, url = {https://proceedings.mlr.press/v306/huang26d.html}, abstract = {Gradient signals in LLM training are highly anisotropic: recurrent linguistic structure concentrates energy into a small set of dominant spectral directions, while context-specific information resides in a long tail. We show that this spike–tail separation persists throughout training, with the spike occupying only about 1.5% of directions yet dominating optimizer statistics. This dominance suppresses tail learning by contracting tail updates through second-moment normalization and tightening the globally stable learning-rate bound. Motivated by this analysis, we propose Spectra, a spike-aware optimizer that suppresses the dominant low-rank spike subspace without amplifying the noise-sensitive spectral tail. Spectra tracks the spike subspace via cached, warm-started power iteration and applies low-rank spectral shaping with negligible overhead and substantially reduced optimizer-state memory. Across Qwen3-0.6B trained on 100B tokens and LLaMA3-8B trained on 50B tokens, Spectra achieves the lowest final validation loss, improving average downstream accuracy by 1.41/0.89 and 1.62/0.66 points over AdamW/Muon, respectively. For wall-clock convergence, Spectra reaches matched loss targets up to 1.31$\times$, 1.34$\times$, and 1.24$\times$ faster than AdamW on Qwen3-0.6B, Qwen3-2B-A0.8B, and Qwen3-8B; its speedup over Muon grows as model scale increases from 0.6B to 8B. For computational efficiency, Spectra is 5.1$\times$ faster than Muon in optimizer processing time, cuts optimizer-state memory by 49.25%, and achieves the lowest measured end-to-end per-step runtime. Spectra’s Megatron integration is available at https://github.com/kimmichtank/spectra.} }
Endnote
%0 Conference Paper %T Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy %A Zhendong Huang %A Hengjie Cao %A Fang Dong %A Ruijun Huang %A Mengyi Chen %A Yifeng Yang %A Xin Zhang %A Anrui Chen %A Mingzhi Dong %A Yujiang Wang %A Jinlong Hou %A Qin Lv %A Robert P. Dick %A Yuan Cheng %A Tun Lu %A Fan Yang %A Li Shang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-huang26d %I PMLR %P 46441--46456 %U https://proceedings.mlr.press/v306/huang26d.html %V 306 %X Gradient signals in LLM training are highly anisotropic: recurrent linguistic structure concentrates energy into a small set of dominant spectral directions, while context-specific information resides in a long tail. We show that this spike–tail separation persists throughout training, with the spike occupying only about 1.5% of directions yet dominating optimizer statistics. This dominance suppresses tail learning by contracting tail updates through second-moment normalization and tightening the globally stable learning-rate bound. Motivated by this analysis, we propose Spectra, a spike-aware optimizer that suppresses the dominant low-rank spike subspace without amplifying the noise-sensitive spectral tail. Spectra tracks the spike subspace via cached, warm-started power iteration and applies low-rank spectral shaping with negligible overhead and substantially reduced optimizer-state memory. Across Qwen3-0.6B trained on 100B tokens and LLaMA3-8B trained on 50B tokens, Spectra achieves the lowest final validation loss, improving average downstream accuracy by 1.41/0.89 and 1.62/0.66 points over AdamW/Muon, respectively. For wall-clock convergence, Spectra reaches matched loss targets up to 1.31$\times$, 1.34$\times$, and 1.24$\times$ faster than AdamW on Qwen3-0.6B, Qwen3-2B-A0.8B, and Qwen3-8B; its speedup over Muon grows as model scale increases from 0.6B to 8B. For computational efficiency, Spectra is 5.1$\times$ faster than Muon in optimizer processing time, cuts optimizer-state memory by 49.25%, and achieves the lowest measured end-to-end per-step runtime. Spectra’s Megatron integration is available at https://github.com/kimmichtank/spectra.
APA
Huang, Z., Cao, H., Dong, F., Huang, R., Chen, M., Yang, Y., Zhang, X., Chen, A., Dong, M., Wang, Y., Hou, J., Lv, Q., Dick, R.P., Cheng, Y., Lu, T., Yang, F. & Shang, L.. (2026). Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:46441-46456 Available from https://proceedings.mlr.press/v306/huang26d.html.

Related Material