From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents

Myeongseob Ko, Jihyun Jeong, Sumiran Singh Thakur, Gyuhak Kim, Ruoxi Jia
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:59572-59589, 2026.

Abstract

Anonymization is often assumed to protect privacy once explicit identifiers are removed, because re-identification has historically required specialized expertise, tailored algorithms, and manual corroboration. We show that LLM-based agents weaken this barrier: by combining scattered, individually non-identifying cues with public evidence, they reconstruct real-world identities, sometimes even during benign tasks. We evaluate this risk across three settings—classical linkage incidents, a controlled benchmark (InferLink) that varies fingerprint type, task framing, and attacker knowledge, and open-ended human–AI interaction traces. In the sparsest regime of the Netflix Prize deanonymization setting, agents reconstruct 79.2% of identities, against 56.0% for a classical matching baseline; on InferLink, they link individuals even without an explicit re-identification request, and more often once one is given. In redacted human–AI interaction traces, agents further resolve anonymized profiles to specific individuals by corroborating contextual cues with public evidence. These findings suggest that privacy evaluations for agentic systems should measure not only what information is accessed or disclosed, but also what identities can be inferred.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-ko26b, title = {From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in {LLM} Agents}, author = {Ko, Myeongseob and Jeong, Jihyun and Thakur, Sumiran Singh and Kim, Gyuhak and Jia, Ruoxi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {59572--59589}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ko26b/ko26b.pdf}, url = {https://proceedings.mlr.press/v306/ko26b.html}, abstract = {Anonymization is often assumed to protect privacy once explicit identifiers are removed, because re-identification has historically required specialized expertise, tailored algorithms, and manual corroboration. We show that LLM-based agents weaken this barrier: by combining scattered, individually non-identifying cues with public evidence, they reconstruct real-world identities, sometimes even during benign tasks. We evaluate this risk across three settings—classical linkage incidents, a controlled benchmark (InferLink) that varies fingerprint type, task framing, and attacker knowledge, and open-ended human–AI interaction traces. In the sparsest regime of the Netflix Prize deanonymization setting, agents reconstruct 79.2% of identities, against 56.0% for a classical matching baseline; on InferLink, they link individuals even without an explicit re-identification request, and more often once one is given. In redacted human–AI interaction traces, agents further resolve anonymized profiles to specific individuals by corroborating contextual cues with public evidence. These findings suggest that privacy evaluations for agentic systems should measure not only what information is accessed or disclosed, but also what identities can be inferred.} }
Endnote
%0 Conference Paper %T From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents %A Myeongseob Ko %A Jihyun Jeong %A Sumiran Singh Thakur %A Gyuhak Kim %A Ruoxi Jia %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ko26b %I PMLR %P 59572--59589 %U https://proceedings.mlr.press/v306/ko26b.html %V 306 %X Anonymization is often assumed to protect privacy once explicit identifiers are removed, because re-identification has historically required specialized expertise, tailored algorithms, and manual corroboration. We show that LLM-based agents weaken this barrier: by combining scattered, individually non-identifying cues with public evidence, they reconstruct real-world identities, sometimes even during benign tasks. We evaluate this risk across three settings—classical linkage incidents, a controlled benchmark (InferLink) that varies fingerprint type, task framing, and attacker knowledge, and open-ended human–AI interaction traces. In the sparsest regime of the Netflix Prize deanonymization setting, agents reconstruct 79.2% of identities, against 56.0% for a classical matching baseline; on InferLink, they link individuals even without an explicit re-identification request, and more often once one is given. In redacted human–AI interaction traces, agents further resolve anonymized profiles to specific individuals by corroborating contextual cues with public evidence. These findings suggest that privacy evaluations for agentic systems should measure not only what information is accessed or disclosed, but also what identities can be inferred.
APA
Ko, M., Jeong, J., Thakur, S.S., Kim, G. & Jia, R.. (2026). From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:59572-59589 Available from https://proceedings.mlr.press/v306/ko26b.html.

Related Material