ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, Anand Mudgerikar
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:136937-136980, 2026.

Abstract

We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real-world security analysts must sift through a large number of heterogeneous security logs, follow multi-hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-wu26bd, title = {{E}x{C}y{TI}n-Bench: Evaluating {LLM} agents on Cyber Threat Investigation}, author = {Wu, Yiran and Velazco, Mauricio and Zhao, Andrew and Luj\'{a}n, Manuel Ra\'{u}l Mel\'{e}ndez and Movva, Srisuma and Roy, Yogesh K and Nguyen, Quang and Rodriguez, Roberto and Wu, Qingyun and Albada, Michael and Kiseleva, Julia and Mudgerikar, Anand}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {136937--136980}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wu26bd/wu26bd.pdf}, url = {https://proceedings.mlr.press/v306/wu26bd.html}, abstract = {We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real-world security analysts must sift through a large number of heterogeneous security logs, follow multi-hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research.} }
Endnote
%0 Conference Paper %T ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation %A Yiran Wu %A Mauricio Velazco %A Andrew Zhao %A Manuel Raúl Meléndez Luján %A Srisuma Movva %A Yogesh K Roy %A Quang Nguyen %A Roberto Rodriguez %A Qingyun Wu %A Michael Albada %A Julia Kiseleva %A Anand Mudgerikar %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wu26bd %I PMLR %P 136937--136980 %U https://proceedings.mlr.press/v306/wu26bd.html %V 306 %X We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real-world security analysts must sift through a large number of heterogeneous security logs, follow multi-hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research.
APA
Wu, Y., Velazco, M., Zhao, A., Luján, M.R.M., Movva, S., Roy, Y.K., Nguyen, Q., Rodriguez, R., Wu, Q., Albada, M., Kiseleva, J. & Mudgerikar, A.. (2026). ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:136937-136980 Available from https://proceedings.mlr.press/v306/wu26bd.html.

Related Material