Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data

Ahmed Mehdi Inane, Vincent Quirion, Gintare Karolina Dziugaite, Ioannis Mitliagkas
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:49733-49760, 2026.

Abstract

Noise-based certified machine unlearning currently faces a hard ceiling: the noise magnitude required to certify unlearning typically destroys model utility, particularly for large-scale deletion requests. While leveraging public data is a standard technique in differential privacy to relax this tension, its role in unlearning remains unexplored. We address this gap by introducing Asymmetric Langevin Unlearning (ALU), a framework that uses public data to mitigate privacy costs. We prove that public data injection suppresses the unlearning cost by a factor of $O(1/n_{\mathrm{pub}}^2)$, guaranteeing a strict computational advantage over retraining. This establishes a new control mechanism: practitioners can mitigate the need for high noise—and the associated utility loss—by increasing the volume of public data. Crucially, we analyze the realistic setting of distribution mismatch, explicitly characterizing how shifts between public and private sources impact utility. We show that ALU enables "mass unlearning” of constant dataset fractions – a regime where standard symmetric methods become impractical – while maintaining high utility. Empirical evaluations using variational Rényi divergence and membership inference attacks confirm that ALU effectively thwarts privacy attacks while preserving utility under reasonable distribution shifts.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-inane26a, title = {Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data}, author = {Inane, Ahmed Mehdi and Quirion, Vincent and Dziugaite, Gintare Karolina and Mitliagkas, Ioannis}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {49733--49760}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/inane26a/inane26a.pdf}, url = {https://proceedings.mlr.press/v306/inane26a.html}, abstract = {Noise-based certified machine unlearning currently faces a hard ceiling: the noise magnitude required to certify unlearning typically destroys model utility, particularly for large-scale deletion requests. While leveraging public data is a standard technique in differential privacy to relax this tension, its role in unlearning remains unexplored. We address this gap by introducing Asymmetric Langevin Unlearning (ALU), a framework that uses public data to mitigate privacy costs. We prove that public data injection suppresses the unlearning cost by a factor of $O(1/n_{\mathrm{pub}}^2)$, guaranteeing a strict computational advantage over retraining. This establishes a new control mechanism: practitioners can mitigate the need for high noise—and the associated utility loss—by increasing the volume of public data. Crucially, we analyze the realistic setting of distribution mismatch, explicitly characterizing how shifts between public and private sources impact utility. We show that ALU enables "mass unlearning” of constant dataset fractions – a regime where standard symmetric methods become impractical – while maintaining high utility. Empirical evaluations using variational Rényi divergence and membership inference attacks confirm that ALU effectively thwarts privacy attacks while preserving utility under reasonable distribution shifts.} }
Endnote
%0 Conference Paper %T Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data %A Ahmed Mehdi Inane %A Vincent Quirion %A Gintare Karolina Dziugaite %A Ioannis Mitliagkas %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-inane26a %I PMLR %P 49733--49760 %U https://proceedings.mlr.press/v306/inane26a.html %V 306 %X Noise-based certified machine unlearning currently faces a hard ceiling: the noise magnitude required to certify unlearning typically destroys model utility, particularly for large-scale deletion requests. While leveraging public data is a standard technique in differential privacy to relax this tension, its role in unlearning remains unexplored. We address this gap by introducing Asymmetric Langevin Unlearning (ALU), a framework that uses public data to mitigate privacy costs. We prove that public data injection suppresses the unlearning cost by a factor of $O(1/n_{\mathrm{pub}}^2)$, guaranteeing a strict computational advantage over retraining. This establishes a new control mechanism: practitioners can mitigate the need for high noise—and the associated utility loss—by increasing the volume of public data. Crucially, we analyze the realistic setting of distribution mismatch, explicitly characterizing how shifts between public and private sources impact utility. We show that ALU enables "mass unlearning” of constant dataset fractions – a regime where standard symmetric methods become impractical – while maintaining high utility. Empirical evaluations using variational Rényi divergence and membership inference attacks confirm that ALU effectively thwarts privacy attacks while preserving utility under reasonable distribution shifts.
APA
Inane, A.M., Quirion, V., Dziugaite, G.K. & Mitliagkas, I.. (2026). Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:49733-49760 Available from https://proceedings.mlr.press/v306/inane26a.html.

Related Material