How does information access affect LLM monitors’ ability to detect sabotage?

Rauno Arike, Raja Mehta Moreno, Rohan Subramani, Shubhorup Biswas, Francis Rhys Ward
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:3473-3526, 2026.

Abstract

Frontier language model agents can exhibit misaligned behaviors, including deception, exploiting reward hacks, and pursuing hidden objectives. To control such agents, we can use LLMs themselves to monitor for misbehavior. In this paper, we study how information access affects LLM monitor performance and find that contemporary systems often perform better with less information, a phenomenon we call the less-is-more effect for automated oversight. We demonstrate this phenomenon, and analyze the conditions under which it occurs, in three evaluation environments where agents must conduct sabotage while evading monitors. Leveraging this effect, we introduce extract-and-evaluate (EaE) monitoring—a new hierarchical approach where one monitor isolates relevant excerpts from the monitored agent’s trajectory and another monitor scores these excerpts. In BigCodeBench-Sabotage with a GPT-4.1-mini monitor, EaE improves sabotage detection rates by 16.8 percentage points over the next-best approach. In other settings, EaE either outperforms or is competitive with baselines. Additionally, we find that agents unaware of being monitored can be caught much more easily, that monitors scoring longer reasoning traces gain more from information filtering, and that monitor performance scales linearly with cost.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-arike26a, title = {How does information access affect {LLM} monitors’ ability to detect sabotage?}, author = {Arike, Rauno and Moreno, Raja Mehta and Subramani, Rohan and Biswas, Shubhorup and Ward, Francis Rhys}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {3473--3526}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/arike26a/arike26a.pdf}, url = {https://proceedings.mlr.press/v306/arike26a.html}, abstract = {Frontier language model agents can exhibit misaligned behaviors, including deception, exploiting reward hacks, and pursuing hidden objectives. To control such agents, we can use LLMs themselves to monitor for misbehavior. In this paper, we study how information access affects LLM monitor performance and find that contemporary systems often perform better with less information, a phenomenon we call the less-is-more effect for automated oversight. We demonstrate this phenomenon, and analyze the conditions under which it occurs, in three evaluation environments where agents must conduct sabotage while evading monitors. Leveraging this effect, we introduce extract-and-evaluate (EaE) monitoring—a new hierarchical approach where one monitor isolates relevant excerpts from the monitored agent’s trajectory and another monitor scores these excerpts. In BigCodeBench-Sabotage with a GPT-4.1-mini monitor, EaE improves sabotage detection rates by 16.8 percentage points over the next-best approach. In other settings, EaE either outperforms or is competitive with baselines. Additionally, we find that agents unaware of being monitored can be caught much more easily, that monitors scoring longer reasoning traces gain more from information filtering, and that monitor performance scales linearly with cost.} }
Endnote
%0 Conference Paper %T How does information access affect LLM monitors’ ability to detect sabotage? %A Rauno Arike %A Raja Mehta Moreno %A Rohan Subramani %A Shubhorup Biswas %A Francis Rhys Ward %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-arike26a %I PMLR %P 3473--3526 %U https://proceedings.mlr.press/v306/arike26a.html %V 306 %X Frontier language model agents can exhibit misaligned behaviors, including deception, exploiting reward hacks, and pursuing hidden objectives. To control such agents, we can use LLMs themselves to monitor for misbehavior. In this paper, we study how information access affects LLM monitor performance and find that contemporary systems often perform better with less information, a phenomenon we call the less-is-more effect for automated oversight. We demonstrate this phenomenon, and analyze the conditions under which it occurs, in three evaluation environments where agents must conduct sabotage while evading monitors. Leveraging this effect, we introduce extract-and-evaluate (EaE) monitoring—a new hierarchical approach where one monitor isolates relevant excerpts from the monitored agent’s trajectory and another monitor scores these excerpts. In BigCodeBench-Sabotage with a GPT-4.1-mini monitor, EaE improves sabotage detection rates by 16.8 percentage points over the next-best approach. In other settings, EaE either outperforms or is competitive with baselines. Additionally, we find that agents unaware of being monitored can be caught much more easily, that monitors scoring longer reasoning traces gain more from information filtering, and that monitor performance scales linearly with cost.
APA
Arike, R., Moreno, R.M., Subramani, R., Biswas, S. & Ward, F.R.. (2026). How does information access affect LLM monitors’ ability to detect sabotage?. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:3473-3526 Available from https://proceedings.mlr.press/v306/arike26a.html.

Related Material