Differentially Private Clipped-SGD: High-Probability Convergence with Arbitrary Clipping Level

Saleh Vatan Khah, Savelii Chezhegov, Shahrokh Farahmand, Samuel Horváth, Eduard Gorbunov
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:4762-4770, 2026.

Abstract

Gradient clipping is a fundamental tool in Deep Learning, improving the high-probability convergence of stochastic first-order methods like SGD, AdaGrad, and Adam under heavy-tailed noise, which is common in training large language models. It is also a crucial component of Differential Privacy (DP) mechanisms. However, existing high-probability convergence analyses typically require the clipping threshold to increase with the number of optimization steps, which is incompatible with standard DP mechanisms like the Gaussian mechanism. In this work, we close this gap by providing the first high-probability convergence analysis for DP-Clipped-SGD with a fixed clipping level, applicable to both convex and non-convex smooth optimization under heavy-tailed noise, characterized by a bounded central $\alpha$-th moment assumption, $\alpha \in (1,2]$. Our results show that, with a fixed clipping level, the method converges to a neighborhood of the optimal solution with a \emph{faster rate} than the existing ones. The neighborhood can be balanced against the noise introduced by DP, providing a refined trade-off between convergence speed and privacy guarantees.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-khah26a, title = { Differentially Private Clipped-SGD: High-Probability Convergence with Arbitrary Clipping Level }, author = {Khah, Saleh Vatan and Chezhegov, Savelii and Farahmand, Shahrokh and Horv{\'a}th, Samuel and Gorbunov, Eduard}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {4762--4770}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/khah26a/khah26a.pdf}, url = {https://proceedings.mlr.press/v300/khah26a.html}, abstract = { Gradient clipping is a fundamental tool in Deep Learning, improving the high-probability convergence of stochastic first-order methods like SGD, AdaGrad, and Adam under heavy-tailed noise, which is common in training large language models. It is also a crucial component of Differential Privacy (DP) mechanisms. However, existing high-probability convergence analyses typically require the clipping threshold to increase with the number of optimization steps, which is incompatible with standard DP mechanisms like the Gaussian mechanism. In this work, we close this gap by providing the first high-probability convergence analysis for DP-Clipped-SGD with a fixed clipping level, applicable to both convex and non-convex smooth optimization under heavy-tailed noise, characterized by a bounded central $\alpha$-th moment assumption, $\alpha \in (1,2]$. Our results show that, with a fixed clipping level, the method converges to a neighborhood of the optimal solution with a \emph{faster rate} than the existing ones. The neighborhood can be balanced against the noise introduced by DP, providing a refined trade-off between convergence speed and privacy guarantees. } }
Endnote
%0 Conference Paper %T Differentially Private Clipped-SGD: High-Probability Convergence with Arbitrary Clipping Level %A Saleh Vatan Khah %A Savelii Chezhegov %A Shahrokh Farahmand %A Samuel Horváth %A Eduard Gorbunov %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-khah26a %I PMLR %P 4762--4770 %U https://proceedings.mlr.press/v300/khah26a.html %V 300 %X Gradient clipping is a fundamental tool in Deep Learning, improving the high-probability convergence of stochastic first-order methods like SGD, AdaGrad, and Adam under heavy-tailed noise, which is common in training large language models. It is also a crucial component of Differential Privacy (DP) mechanisms. However, existing high-probability convergence analyses typically require the clipping threshold to increase with the number of optimization steps, which is incompatible with standard DP mechanisms like the Gaussian mechanism. In this work, we close this gap by providing the first high-probability convergence analysis for DP-Clipped-SGD with a fixed clipping level, applicable to both convex and non-convex smooth optimization under heavy-tailed noise, characterized by a bounded central $\alpha$-th moment assumption, $\alpha \in (1,2]$. Our results show that, with a fixed clipping level, the method converges to a neighborhood of the optimal solution with a \emph{faster rate} than the existing ones. The neighborhood can be balanced against the noise introduced by DP, providing a refined trade-off between convergence speed and privacy guarantees.
APA
Khah, S.V., Chezhegov, S., Farahmand, S., Horváth, S. & Gorbunov, E.. (2026). Differentially Private Clipped-SGD: High-Probability Convergence with Arbitrary Clipping Level . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:4762-4770 Available from https://proceedings.mlr.press/v300/khah26a.html.

Related Material