Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction

Ziyang Wei, Wanrong Zhu, Jingyang Lyu, Wei Biao Wu
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:3565-3573, 2026.

Abstract

We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods—such as plug-in and batch-means estimators—are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence. To address these challenges, we propose a novel, fully online de-biased covariance estimator that eliminates the need for second-order derivatives while significantly improving estimation accuracy. Our method employs a bias-reduction technique to achieve a convergence rate of $n^{(\alpha-1)/2}\sqrt{\log n}$, outperforming existing Hessian-free alternatives.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-wei26c, title = { Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction }, author = {Wei, Ziyang and Zhu, Wanrong and Lyu, Jingyang and Wu, Wei Biao}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {3565--3573}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/wei26c/wei26c.pdf}, url = {https://proceedings.mlr.press/v300/wei26c.html}, abstract = { We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods—such as plug-in and batch-means estimators—are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence. To address these challenges, we propose a novel, fully online de-biased covariance estimator that eliminates the need for second-order derivatives while significantly improving estimation accuracy. Our method employs a bias-reduction technique to achieve a convergence rate of $n^{(\alpha-1)/2}\sqrt{\log n}$, outperforming existing Hessian-free alternatives. } }
Endnote
%0 Conference Paper %T Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction %A Ziyang Wei %A Wanrong Zhu %A Jingyang Lyu %A Wei Biao Wu %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-wei26c %I PMLR %P 3565--3573 %U https://proceedings.mlr.press/v300/wei26c.html %V 300 %X We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods—such as plug-in and batch-means estimators—are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence. To address these challenges, we propose a novel, fully online de-biased covariance estimator that eliminates the need for second-order derivatives while significantly improving estimation accuracy. Our method employs a bias-reduction technique to achieve a convergence rate of $n^{(\alpha-1)/2}\sqrt{\log n}$, outperforming existing Hessian-free alternatives.
APA
Wei, Z., Zhu, W., Lyu, J. & Wu, W.B.. (2026). Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:3565-3573 Available from https://proceedings.mlr.press/v300/wei26c.html.

Related Material