Leak@$k$: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding

Hadi Reisizadeh, Jiajun Ruan, Yiwei Chen, Soumyadeep Pal, Sijia Liu, Mingyi Hong
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:104291-104315, 2026.

Abstract

Unlearning in large language models (LLMs) is critical for regulatory compliance and for building ethical generative AI systems that avoid producing private, toxic, illegal, or copyrighted content. Despite rapid progress, in this work, we show that almost all existing unlearning methods fail to achieve true forgetting in practice. Specifically, while evaluations of these ‘unlearned’ models under deterministic (greedy) decoding often suggest successful knowledge removal using standard benchmarks, we show that sensitive information reliably resurfaces when models are sampled with standard probabilistic decoding. To rigorously capture this vulnerability, we introduce leak@$k$, a new meta-evaluation metric that quantifies the likelihood of forgotten knowledge reappearing when generating $k$ samples from the model under realistic decoding strategies. Using three widely adopted benchmarks, TOFU, MUSE, and WMDP, we conduct the first large-scale, systematic study of unlearning reliability using leak@$k$ metric. Our findings demonstrate that knowledge leakage persists across methods and tasks, underscoring that current state-of-the-art (SOTA) unlearning techniques provide only limited forgetting. We propose an algorithm, termed Robust Unlearning under LEak@$k$ metric (RULE) to address this concern. We demonstrate that RULE provides an unlearned model for TOFU benchmark with no information leakage for a large number of generation samples. On the MUSE benchmark, RULE outperforms SOTA unlearning methods under the leak@$k$ metric across most sampling budgets $k$. Codes are available at https://github.com/OptimAI-Lab/Leak-k.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-reisizadeh26a, title = {Leak@$k$: Unlearning Does Not Make {LLM}s Forget Under Probabilistic Decoding}, author = {Reisizadeh, Hadi and Ruan, Jiajun and Chen, Yiwei and Pal, Soumyadeep and Liu, Sijia and Hong, Mingyi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {104291--104315}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/reisizadeh26a/reisizadeh26a.pdf}, url = {https://proceedings.mlr.press/v306/reisizadeh26a.html}, abstract = {Unlearning in large language models (LLMs) is critical for regulatory compliance and for building ethical generative AI systems that avoid producing private, toxic, illegal, or copyrighted content. Despite rapid progress, in this work, we show that almost all existing unlearning methods fail to achieve true forgetting in practice. Specifically, while evaluations of these ‘unlearned’ models under deterministic (greedy) decoding often suggest successful knowledge removal using standard benchmarks, we show that sensitive information reliably resurfaces when models are sampled with standard probabilistic decoding. To rigorously capture this vulnerability, we introduce leak@$k$, a new meta-evaluation metric that quantifies the likelihood of forgotten knowledge reappearing when generating $k$ samples from the model under realistic decoding strategies. Using three widely adopted benchmarks, TOFU, MUSE, and WMDP, we conduct the first large-scale, systematic study of unlearning reliability using leak@$k$ metric. Our findings demonstrate that knowledge leakage persists across methods and tasks, underscoring that current state-of-the-art (SOTA) unlearning techniques provide only limited forgetting. We propose an algorithm, termed Robust Unlearning under LEak@$k$ metric (RULE) to address this concern. We demonstrate that RULE provides an unlearned model for TOFU benchmark with no information leakage for a large number of generation samples. On the MUSE benchmark, RULE outperforms SOTA unlearning methods under the leak@$k$ metric across most sampling budgets $k$. Codes are available at https://github.com/OptimAI-Lab/Leak-k.} }
Endnote
%0 Conference Paper %T Leak@$k$: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding %A Hadi Reisizadeh %A Jiajun Ruan %A Yiwei Chen %A Soumyadeep Pal %A Sijia Liu %A Mingyi Hong %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-reisizadeh26a %I PMLR %P 104291--104315 %U https://proceedings.mlr.press/v306/reisizadeh26a.html %V 306 %X Unlearning in large language models (LLMs) is critical for regulatory compliance and for building ethical generative AI systems that avoid producing private, toxic, illegal, or copyrighted content. Despite rapid progress, in this work, we show that almost all existing unlearning methods fail to achieve true forgetting in practice. Specifically, while evaluations of these ‘unlearned’ models under deterministic (greedy) decoding often suggest successful knowledge removal using standard benchmarks, we show that sensitive information reliably resurfaces when models are sampled with standard probabilistic decoding. To rigorously capture this vulnerability, we introduce leak@$k$, a new meta-evaluation metric that quantifies the likelihood of forgotten knowledge reappearing when generating $k$ samples from the model under realistic decoding strategies. Using three widely adopted benchmarks, TOFU, MUSE, and WMDP, we conduct the first large-scale, systematic study of unlearning reliability using leak@$k$ metric. Our findings demonstrate that knowledge leakage persists across methods and tasks, underscoring that current state-of-the-art (SOTA) unlearning techniques provide only limited forgetting. We propose an algorithm, termed Robust Unlearning under LEak@$k$ metric (RULE) to address this concern. We demonstrate that RULE provides an unlearned model for TOFU benchmark with no information leakage for a large number of generation samples. On the MUSE benchmark, RULE outperforms SOTA unlearning methods under the leak@$k$ metric across most sampling budgets $k$. Codes are available at https://github.com/OptimAI-Lab/Leak-k.
APA
Reisizadeh, H., Ruan, J., Chen, Y., Pal, S., Liu, S. & Hong, M.. (2026). Leak@$k$: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:104291-104315 Available from https://proceedings.mlr.press/v306/reisizadeh26a.html.

Related Material