Same Question, Different Lies: Cross-Context Consistency (C$^3$) for Black-Box Sandbagging Detection

Lin Yulong, Pablo Bernabeu-Perez, Benjamin Arnav, Lennie Wells, Mary Phuong
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:152533-152571, 2026.

Abstract

As language models grow more capable, accurate capability evaluation becomes essential for safety decisions. If models can deliberately underperform on dangerous capability evaluations—a behavior known as sandbagging—they may evade safety measures designed for their true capability level. We introduce Cross-Context Consistency ($C^3$), a general framework for unsupervised black-box sandbagging detection that exploits a fundamental asymmetry: when a model truly lacks capability, its confusion manifests consistently across paraphrased questions, but when a capable model feigns incompetence, its strategic choices about how to appear weak create detectable inconsistencies. The framework is agnostic to the specific consistency metric and aggregation method; we present a simple instantiation using embedding cosine distance and mean aggregation that requires no training data or model-specific adaptation. We evaluate $C^3$ across prompted and fine-tuned sandbagging scenarios, across instructed and more naturalistic setups, maintaining a classification signal where other black-box methods fail. Our findings show the limitations of existing sandbagging detection methods, and reveal the efficacy of consistency-checking as a detection mechanism for sandbagging.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-yulong26a, title = {Same Question, Different Lies: Cross-Context Consistency ({C}${^3}$) for Black-Box Sandbagging Detection}, author = {Yulong, Lin and Bernabeu-Perez, Pablo and Arnav, Benjamin and Wells, Lennie and Phuong, Mary}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {152533--152571}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yulong26a/yulong26a.pdf}, url = {https://proceedings.mlr.press/v306/yulong26a.html}, abstract = {As language models grow more capable, accurate capability evaluation becomes essential for safety decisions. If models can deliberately underperform on dangerous capability evaluations—a behavior known as sandbagging—they may evade safety measures designed for their true capability level. We introduce Cross-Context Consistency ($C^3$), a general framework for unsupervised black-box sandbagging detection that exploits a fundamental asymmetry: when a model truly lacks capability, its confusion manifests consistently across paraphrased questions, but when a capable model feigns incompetence, its strategic choices about how to appear weak create detectable inconsistencies. The framework is agnostic to the specific consistency metric and aggregation method; we present a simple instantiation using embedding cosine distance and mean aggregation that requires no training data or model-specific adaptation. We evaluate $C^3$ across prompted and fine-tuned sandbagging scenarios, across instructed and more naturalistic setups, maintaining a classification signal where other black-box methods fail. Our findings show the limitations of existing sandbagging detection methods, and reveal the efficacy of consistency-checking as a detection mechanism for sandbagging.} }
Endnote
%0 Conference Paper %T Same Question, Different Lies: Cross-Context Consistency (C$^3$) for Black-Box Sandbagging Detection %A Lin Yulong %A Pablo Bernabeu-Perez %A Benjamin Arnav %A Lennie Wells %A Mary Phuong %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yulong26a %I PMLR %P 152533--152571 %U https://proceedings.mlr.press/v306/yulong26a.html %V 306 %X As language models grow more capable, accurate capability evaluation becomes essential for safety decisions. If models can deliberately underperform on dangerous capability evaluations—a behavior known as sandbagging—they may evade safety measures designed for their true capability level. We introduce Cross-Context Consistency ($C^3$), a general framework for unsupervised black-box sandbagging detection that exploits a fundamental asymmetry: when a model truly lacks capability, its confusion manifests consistently across paraphrased questions, but when a capable model feigns incompetence, its strategic choices about how to appear weak create detectable inconsistencies. The framework is agnostic to the specific consistency metric and aggregation method; we present a simple instantiation using embedding cosine distance and mean aggregation that requires no training data or model-specific adaptation. We evaluate $C^3$ across prompted and fine-tuned sandbagging scenarios, across instructed and more naturalistic setups, maintaining a classification signal where other black-box methods fail. Our findings show the limitations of existing sandbagging detection methods, and reveal the efficacy of consistency-checking as a detection mechanism for sandbagging.
APA
Yulong, L., Bernabeu-Perez, P., Arnav, B., Wells, L. & Phuong, M.. (2026). Same Question, Different Lies: Cross-Context Consistency (C$^3$) for Black-Box Sandbagging Detection. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:152533-152571 Available from https://proceedings.mlr.press/v306/yulong26a.html.

Related Material