Position: LLM-Safety Evaluations Lack Robustness

Tim Beyer, Sophie Xhonneux, Simon Geisler, Gauthier Gidel, Leo Schwinn, Stephan Günnemann
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:169466-169489, 2026.

Abstract

In this position paper, we argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise, such as small datasets, methodological inconsistencies, and unreliable evaluation setups. This can, at times, make it impossible to evaluate and compare attacks and defenses fairly, thereby slowing research progress. We systematically analyze the LLM safety evaluation pipeline, covering dataset curation, optimization strategies for automated red-teaming, response generation, and response evaluation using LLM judges. At each stage, we identify key issues and highlight their practical impact. We also propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers. Lastly, we offer an opposing perspective, highlighting practical reasons for existing limitations. We believe that addressing the outlined problems in future research will improve the field’s ability to generate easily comparable results and make measurable progress.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-beyer26b, title = {Position: {LLM}-Safety Evaluations Lack Robustness}, author = {Beyer, Tim and Xhonneux, Sophie and Geisler, Simon and Gidel, Gauthier and Schwinn, Leo and G\"{u}nnemann, Stephan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {169466--169489}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/beyer26b/beyer26b.pdf}, url = {https://proceedings.mlr.press/v306/beyer26b.html}, abstract = {In this position paper, we argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise, such as small datasets, methodological inconsistencies, and unreliable evaluation setups. This can, at times, make it impossible to evaluate and compare attacks and defenses fairly, thereby slowing research progress. We systematically analyze the LLM safety evaluation pipeline, covering dataset curation, optimization strategies for automated red-teaming, response generation, and response evaluation using LLM judges. At each stage, we identify key issues and highlight their practical impact. We also propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers. Lastly, we offer an opposing perspective, highlighting practical reasons for existing limitations. We believe that addressing the outlined problems in future research will improve the field’s ability to generate easily comparable results and make measurable progress.} }
Endnote
%0 Conference Paper %T Position: LLM-Safety Evaluations Lack Robustness %A Tim Beyer %A Sophie Xhonneux %A Simon Geisler %A Gauthier Gidel %A Leo Schwinn %A Stephan Günnemann %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-beyer26b %I PMLR %P 169466--169489 %U https://proceedings.mlr.press/v306/beyer26b.html %V 306 %X In this position paper, we argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise, such as small datasets, methodological inconsistencies, and unreliable evaluation setups. This can, at times, make it impossible to evaluate and compare attacks and defenses fairly, thereby slowing research progress. We systematically analyze the LLM safety evaluation pipeline, covering dataset curation, optimization strategies for automated red-teaming, response generation, and response evaluation using LLM judges. At each stage, we identify key issues and highlight their practical impact. We also propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers. Lastly, we offer an opposing perspective, highlighting practical reasons for existing limitations. We believe that addressing the outlined problems in future research will improve the field’s ability to generate easily comparable results and make measurable progress.
APA
Beyer, T., Xhonneux, S., Geisler, S., Gidel, G., Schwinn, L. & Günnemann, S.. (2026). Position: LLM-Safety Evaluations Lack Robustness. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:169466-169489 Available from https://proceedings.mlr.press/v306/beyer26b.html.

Related Material