FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction

Duc Dm, Thao Do, Minh Son Hoang, Anh Le Duc Tran, Daeyoung Kim, Huy Nguyen
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:25534-25566, 2026.

Abstract

Differentially private (DP) training protects individual examples by adding noise to gradients, but the injected noise interacts nontrivially with adaptive optimizers. Recent DP methods temporally filter privatized gradients to reduce variance; however, filtering also changes the DP noise statistics seen by AdamW’s second-moment accumulator. As a result, bias corrections derived for unfiltered DP noise (e.g., subtracting $\sigma_w^2$) can become miscalibrated when filtering is present. We propose FIBER, a DP optimizer designed for temporally filtered privatized gradients. FIBER (i) performs denoising in innovation space by filtering the residual stream and integrating it to form the filtered gradient estimate, (ii) decouples the two-point observation geometry from the innovation gain to enable independent tuning, and (iii) introduces a filter-aware second-moment calibration that subtracts the attenuated DP noise contribution $A(\omega)\sigma_w^2$, where $A(\omega)$ is derived in closed form for the innovation filter and can be computed for general stable linear filters. Across vision and language benchmarks, FIBER consistently demonstrates substantial improvements in the performance of DP optimizers, surpassing state-of-the-art results under equivalent privacy constraints on multiple tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-dm26a, title = {{FIBER}: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction}, author = {Dm, Duc and Do, Thao and Hoang, Minh Son and Tran, Anh Le Duc and Kim, Daeyoung and Nguyen, Huy}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {25534--25566}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/dm26a/dm26a.pdf}, url = {https://proceedings.mlr.press/v306/dm26a.html}, abstract = {Differentially private (DP) training protects individual examples by adding noise to gradients, but the injected noise interacts nontrivially with adaptive optimizers. Recent DP methods temporally filter privatized gradients to reduce variance; however, filtering also changes the DP noise statistics seen by AdamW’s second-moment accumulator. As a result, bias corrections derived for unfiltered DP noise (e.g., subtracting $\sigma_w^2$) can become miscalibrated when filtering is present. We propose FIBER, a DP optimizer designed for temporally filtered privatized gradients. FIBER (i) performs denoising in innovation space by filtering the residual stream and integrating it to form the filtered gradient estimate, (ii) decouples the two-point observation geometry from the innovation gain to enable independent tuning, and (iii) introduces a filter-aware second-moment calibration that subtracts the attenuated DP noise contribution $A(\omega)\sigma_w^2$, where $A(\omega)$ is derived in closed form for the innovation filter and can be computed for general stable linear filters. Across vision and language benchmarks, FIBER consistently demonstrates substantial improvements in the performance of DP optimizers, surpassing state-of-the-art results under equivalent privacy constraints on multiple tasks.} }
Endnote
%0 Conference Paper %T FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction %A Duc Dm %A Thao Do %A Minh Son Hoang %A Anh Le Duc Tran %A Daeyoung Kim %A Huy Nguyen %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-dm26a %I PMLR %P 25534--25566 %U https://proceedings.mlr.press/v306/dm26a.html %V 306 %X Differentially private (DP) training protects individual examples by adding noise to gradients, but the injected noise interacts nontrivially with adaptive optimizers. Recent DP methods temporally filter privatized gradients to reduce variance; however, filtering also changes the DP noise statistics seen by AdamW’s second-moment accumulator. As a result, bias corrections derived for unfiltered DP noise (e.g., subtracting $\sigma_w^2$) can become miscalibrated when filtering is present. We propose FIBER, a DP optimizer designed for temporally filtered privatized gradients. FIBER (i) performs denoising in innovation space by filtering the residual stream and integrating it to form the filtered gradient estimate, (ii) decouples the two-point observation geometry from the innovation gain to enable independent tuning, and (iii) introduces a filter-aware second-moment calibration that subtracts the attenuated DP noise contribution $A(\omega)\sigma_w^2$, where $A(\omega)$ is derived in closed form for the innovation filter and can be computed for general stable linear filters. Across vision and language benchmarks, FIBER consistently demonstrates substantial improvements in the performance of DP optimizers, surpassing state-of-the-art results under equivalent privacy constraints on multiple tasks.
APA
Dm, D., Do, T., Hoang, M.S., Tran, A.L.D., Kim, D. & Nguyen, H.. (2026). FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:25534-25566 Available from https://proceedings.mlr.press/v306/dm26a.html.

Related Material