Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models

Yuefeng Peng, Parnian Afshar, Megan Ganji, Thomas Butler, Amir Houmansadr, Mingxian Wang, Dezhi Hong
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:97893-97911, 2026.

Abstract

Large language models can memorize information that must be removed—ranging from copyright-sensitive content (e.g., book chapters) to personally identifiable information (e.g., income)—to ensure responsible and compliant behavior. Unlearning has emerged as an efficient alternative to full retraining, aiming to remove specific knowledge. However, users may still expect model to leverage the removed information when it is re-introduced in the prompt. Existing evaluations of unlearning methods focus on (1) the extent of forgetting of the target knowledge (forget set) and (2) performance preservation on the retain set (i.e., utility), but overlook this critical usability dimension. Through a systematic evaluation of six state-of-the-art unlearning methods, we show that they consistently degrade such contextual utility—the model’s ability to use forgotten knowledge when it is provided in context. To address this, we augment unlearning objectives with a plug-in term that explicitly preserves contextual utility. Extensive experiments demonstrate that our approach restores contextual utility to near original levels while still maintaining effective forgetting and retain-set utility.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-peng26a, title = {Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models}, author = {Peng, Yuefeng and Afshar, Parnian and Ganji, Megan and Butler, Thomas and Houmansadr, Amir and Wang, Mingxian and Hong, Dezhi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {97893--97911}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/peng26a/peng26a.pdf}, url = {https://proceedings.mlr.press/v306/peng26a.html}, abstract = {Large language models can memorize information that must be removed—ranging from copyright-sensitive content (e.g., book chapters) to personally identifiable information (e.g., income)—to ensure responsible and compliant behavior. Unlearning has emerged as an efficient alternative to full retraining, aiming to remove specific knowledge. However, users may still expect model to leverage the removed information when it is re-introduced in the prompt. Existing evaluations of unlearning methods focus on (1) the extent of forgetting of the target knowledge (forget set) and (2) performance preservation on the retain set (i.e., utility), but overlook this critical usability dimension. Through a systematic evaluation of six state-of-the-art unlearning methods, we show that they consistently degrade such contextual utility—the model’s ability to use forgotten knowledge when it is provided in context. To address this, we augment unlearning objectives with a plug-in term that explicitly preserves contextual utility. Extensive experiments demonstrate that our approach restores contextual utility to near original levels while still maintaining effective forgetting and retain-set utility.} }
Endnote
%0 Conference Paper %T Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models %A Yuefeng Peng %A Parnian Afshar %A Megan Ganji %A Thomas Butler %A Amir Houmansadr %A Mingxian Wang %A Dezhi Hong %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-peng26a %I PMLR %P 97893--97911 %U https://proceedings.mlr.press/v306/peng26a.html %V 306 %X Large language models can memorize information that must be removed—ranging from copyright-sensitive content (e.g., book chapters) to personally identifiable information (e.g., income)—to ensure responsible and compliant behavior. Unlearning has emerged as an efficient alternative to full retraining, aiming to remove specific knowledge. However, users may still expect model to leverage the removed information when it is re-introduced in the prompt. Existing evaluations of unlearning methods focus on (1) the extent of forgetting of the target knowledge (forget set) and (2) performance preservation on the retain set (i.e., utility), but overlook this critical usability dimension. Through a systematic evaluation of six state-of-the-art unlearning methods, we show that they consistently degrade such contextual utility—the model’s ability to use forgotten knowledge when it is provided in context. To address this, we augment unlearning objectives with a plug-in term that explicitly preserves contextual utility. Extensive experiments demonstrate that our approach restores contextual utility to near original levels while still maintaining effective forgetting and retain-set utility.
APA
Peng, Y., Afshar, P., Ganji, M., Butler, T., Houmansadr, A., Wang, M. & Hong, D.. (2026). Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:97893-97911 Available from https://proceedings.mlr.press/v306/peng26a.html.

Related Material