From Winning to Understanding: A Diagnostic Long-Horizon RTS Benchmark for LLMs

Jiacheng Li, Jiahui Liu, Yuqing Wang, Gaochen Cui, Xiao Zhang, Qianchuan Zhao, Ziyou Zhang, Chenghao Li
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:68721-68741, 2026.

Abstract

Large language models (LLMs) are increasingly used as decision modules, yet existing benchmarks provide limited coverage of long-horizon, adversarial interaction while faithfully acting on human instructions. We introduce a long-horizon Red Alert RTS benchmark with a hierarchical interface in which LLMs output budgeted, low-frequency macro/tactical intents that are executed deterministically for standardized comparison. The benchmark evaluates (i) robustness to “rules-as-variable” perturbations via rule-style shifts , (ii) competitive strength via Elo-style ratings from head-to-head matches, and (iii) human steerability via standardized language interventions. Beyond win/loss, we log economy growth/spending, combat loss ratio, and visibility coverage to diagnose long-horizon failure modes. Overall, the benchmark provides a reproducible and diagnostic testbed for robustness and controllability in long-horizon adversarial decision making.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-li26do, title = {From Winning to Understanding: A Diagnostic Long-Horizon {RTS} Benchmark for {LLM}s}, author = {Li, Jiacheng and Liu, Jiahui and Wang, Yuqing and Cui, Gaochen and Zhang, Xiao and Zhao, Qianchuan and Zhang, Ziyou and Li, Chenghao}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {68721--68741}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/li26do/li26do.pdf}, url = {https://proceedings.mlr.press/v306/li26do.html}, abstract = {Large language models (LLMs) are increasingly used as decision modules, yet existing benchmarks provide limited coverage of long-horizon, adversarial interaction while faithfully acting on human instructions. We introduce a long-horizon Red Alert RTS benchmark with a hierarchical interface in which LLMs output budgeted, low-frequency macro/tactical intents that are executed deterministically for standardized comparison. The benchmark evaluates (i) robustness to “rules-as-variable” perturbations via rule-style shifts , (ii) competitive strength via Elo-style ratings from head-to-head matches, and (iii) human steerability via standardized language interventions. Beyond win/loss, we log economy growth/spending, combat loss ratio, and visibility coverage to diagnose long-horizon failure modes. Overall, the benchmark provides a reproducible and diagnostic testbed for robustness and controllability in long-horizon adversarial decision making.} }
Endnote
%0 Conference Paper %T From Winning to Understanding: A Diagnostic Long-Horizon RTS Benchmark for LLMs %A Jiacheng Li %A Jiahui Liu %A Yuqing Wang %A Gaochen Cui %A Xiao Zhang %A Qianchuan Zhao %A Ziyou Zhang %A Chenghao Li %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-li26do %I PMLR %P 68721--68741 %U https://proceedings.mlr.press/v306/li26do.html %V 306 %X Large language models (LLMs) are increasingly used as decision modules, yet existing benchmarks provide limited coverage of long-horizon, adversarial interaction while faithfully acting on human instructions. We introduce a long-horizon Red Alert RTS benchmark with a hierarchical interface in which LLMs output budgeted, low-frequency macro/tactical intents that are executed deterministically for standardized comparison. The benchmark evaluates (i) robustness to “rules-as-variable” perturbations via rule-style shifts , (ii) competitive strength via Elo-style ratings from head-to-head matches, and (iii) human steerability via standardized language interventions. Beyond win/loss, we log economy growth/spending, combat loss ratio, and visibility coverage to diagnose long-horizon failure modes. Overall, the benchmark provides a reproducible and diagnostic testbed for robustness and controllability in long-horizon adversarial decision making.
APA
Li, J., Liu, J., Wang, Y., Cui, G., Zhang, X., Zhao, Q., Zhang, Z. & Li, C.. (2026). From Winning to Understanding: A Diagnostic Long-Horizon RTS Benchmark for LLMs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:68721-68741 Available from https://proceedings.mlr.press/v306/li26do.html.

Related Material