Token-Efficient Change Detection in LLM APIs

Timothee Chauvin, Clément Lalanne, Erwan Le Merrer, Jean-Michel Loubes, Francois Taiani, Gilles Tredan
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:13247-13277, 2026.

Abstract

Remote change detection in LLMs is a difficult problem. Existing methods are either too expensive for deployment at scale, or require initial white-box access to model weights or grey-box access to log probabilities. We aim to achieve both low cost and strict black-box operation, observing only output tokens. Our approach hinges on specific inputs we call Border Inputs, for which there exists more than one output top token. From a statistical perspective, optimal change detection depends on the model’s Jacobian and the Fisher information of the output distribution, whose analysis at low temperature regimes shows that border inputs enable powerful change detection tests. Building on this insight, we propose the Black-Box Border Input Tracking (B3IT) scheme. Extensive in-vivo and in-vitro experiments show that border inputs are easily found for non-reasoning tested endpoints, and present on-par performance with the best available grey-box approaches. B3IT reduces costs by $30\times$ compared to existing methods, while operating in a strict black-box setting.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chauvin26a, title = {Token-Efficient Change Detection in {LLM} {API}s}, author = {Chauvin, Timothee and Lalanne, Cl\'{e}ment and Le Merrer, Erwan and Loubes, Jean-Michel and Taiani, Francois and Tredan, Gilles}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {13247--13277}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chauvin26a/chauvin26a.pdf}, url = {https://proceedings.mlr.press/v306/chauvin26a.html}, abstract = {Remote change detection in LLMs is a difficult problem. Existing methods are either too expensive for deployment at scale, or require initial white-box access to model weights or grey-box access to log probabilities. We aim to achieve both low cost and strict black-box operation, observing only output tokens. Our approach hinges on specific inputs we call Border Inputs, for which there exists more than one output top token. From a statistical perspective, optimal change detection depends on the model’s Jacobian and the Fisher information of the output distribution, whose analysis at low temperature regimes shows that border inputs enable powerful change detection tests. Building on this insight, we propose the Black-Box Border Input Tracking (B3IT) scheme. Extensive in-vivo and in-vitro experiments show that border inputs are easily found for non-reasoning tested endpoints, and present on-par performance with the best available grey-box approaches. B3IT reduces costs by $30\times$ compared to existing methods, while operating in a strict black-box setting.} }
Endnote
%0 Conference Paper %T Token-Efficient Change Detection in LLM APIs %A Timothee Chauvin %A Clément Lalanne %A Erwan Le Merrer %A Jean-Michel Loubes %A Francois Taiani %A Gilles Tredan %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chauvin26a %I PMLR %P 13247--13277 %U https://proceedings.mlr.press/v306/chauvin26a.html %V 306 %X Remote change detection in LLMs is a difficult problem. Existing methods are either too expensive for deployment at scale, or require initial white-box access to model weights or grey-box access to log probabilities. We aim to achieve both low cost and strict black-box operation, observing only output tokens. Our approach hinges on specific inputs we call Border Inputs, for which there exists more than one output top token. From a statistical perspective, optimal change detection depends on the model’s Jacobian and the Fisher information of the output distribution, whose analysis at low temperature regimes shows that border inputs enable powerful change detection tests. Building on this insight, we propose the Black-Box Border Input Tracking (B3IT) scheme. Extensive in-vivo and in-vitro experiments show that border inputs are easily found for non-reasoning tested endpoints, and present on-par performance with the best available grey-box approaches. B3IT reduces costs by $30\times$ compared to existing methods, while operating in a strict black-box setting.
APA
Chauvin, T., Lalanne, C., Le Merrer, E., Loubes, J., Taiani, F. & Tredan, G.. (2026). Token-Efficient Change Detection in LLM APIs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:13247-13277 Available from https://proceedings.mlr.press/v306/chauvin26a.html.

Related Material