Language Models May Verbatim Complete Text They Were Not Explicitly Trained On

Ken Liu, Christopher A. Choquette-Choo, Matthew Jagielski, Peter Kairouz, Sanmi Koyejo, Percy Liang, Nicolas Papernot
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:38210-38250, 2025.

Abstract

An important question today is whether a given text was used to train a large language model (LLM). A completion test is often employed: check if the LLM completes a sufficiently complex text. This, however, requires a ground-truth definition of membership; most commonly, it is defined as a member based on the n-gram overlap between the target text and any text in the dataset. In this work, we demonstrate that this n-gram based membership definition can be effectively gamed. We study scenarios where sequences are non-members for a given n and we find that completion tests still succeed. We find many natural cases of this phenomenon by retraining LLMs from scratch after removing all training samples that were completed; these cases include exact duplicates, near-duplicates, and even short overlaps. They showcase that it is difficult to find a single viable choice of n for membership definitions. Using these insights, we design adversarial datasets that can cause a given target sequence to be completed without containing it, for any reasonable choice of n. Our findings highlight the inadequacy of n-gram membership, suggesting membership definitions fail to account for auxiliary information available to the training algorithm.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-liu25h, title = {Language Models May Verbatim Complete Text They Were Not Explicitly Trained On}, author = {Liu, Ken and Choquette-Choo, Christopher A. and Jagielski, Matthew and Kairouz, Peter and Koyejo, Sanmi and Liang, Percy and Papernot, Nicolas}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {38210--38250}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/liu25h/liu25h.pdf}, url = {https://proceedings.mlr.press/v267/liu25h.html}, abstract = {An important question today is whether a given text was used to train a large language model (LLM). A completion test is often employed: check if the LLM completes a sufficiently complex text. This, however, requires a ground-truth definition of membership; most commonly, it is defined as a member based on the n-gram overlap between the target text and any text in the dataset. In this work, we demonstrate that this n-gram based membership definition can be effectively gamed. We study scenarios where sequences are non-members for a given n and we find that completion tests still succeed. We find many natural cases of this phenomenon by retraining LLMs from scratch after removing all training samples that were completed; these cases include exact duplicates, near-duplicates, and even short overlaps. They showcase that it is difficult to find a single viable choice of n for membership definitions. Using these insights, we design adversarial datasets that can cause a given target sequence to be completed without containing it, for any reasonable choice of n. Our findings highlight the inadequacy of n-gram membership, suggesting membership definitions fail to account for auxiliary information available to the training algorithm.} }
Endnote
%0 Conference Paper %T Language Models May Verbatim Complete Text They Were Not Explicitly Trained On %A Ken Liu %A Christopher A. Choquette-Choo %A Matthew Jagielski %A Peter Kairouz %A Sanmi Koyejo %A Percy Liang %A Nicolas Papernot %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-liu25h %I PMLR %P 38210--38250 %U https://proceedings.mlr.press/v267/liu25h.html %V 267 %X An important question today is whether a given text was used to train a large language model (LLM). A completion test is often employed: check if the LLM completes a sufficiently complex text. This, however, requires a ground-truth definition of membership; most commonly, it is defined as a member based on the n-gram overlap between the target text and any text in the dataset. In this work, we demonstrate that this n-gram based membership definition can be effectively gamed. We study scenarios where sequences are non-members for a given n and we find that completion tests still succeed. We find many natural cases of this phenomenon by retraining LLMs from scratch after removing all training samples that were completed; these cases include exact duplicates, near-duplicates, and even short overlaps. They showcase that it is difficult to find a single viable choice of n for membership definitions. Using these insights, we design adversarial datasets that can cause a given target sequence to be completed without containing it, for any reasonable choice of n. Our findings highlight the inadequacy of n-gram membership, suggesting membership definitions fail to account for auxiliary information available to the training algorithm.
APA
Liu, K., Choquette-Choo, C.A., Jagielski, M., Kairouz, P., Koyejo, S., Liang, P. & Papernot, N.. (2025). Language Models May Verbatim Complete Text They Were Not Explicitly Trained On. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:38210-38250 Available from https://proceedings.mlr.press/v267/liu25h.html.

Related Material