On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length

Sunghwan Kim, Junhee Cho, Beong-Woo Kwak, Taeyoon Kwon, Liang Wang, Nan Yang, Xingxing Zhang, Furu Wei, Jinyoung Yeo
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:57309-57337, 2026.

Abstract

Large language models (LLMs) have shown promise as interactive agents that solve tasks through extended sequences of environment interactions. While prior work has primarily focused on system-level optimizations or algorithmic improvements, the role of task horizon length in shaping training dynamics remains poorly understood. In this work, we present a systematic empirical study that examines horizon length through controlled task constructions. Specifically, we construct controlled tasks in which agents face identical decision rules and reasoning structures, but differ only in the length of action sequences required for successful completion. Our results reveal that increasing horizon length alone constitutes a training bottleneck, inducing severe training instability driven by exploration difficulties and credit assignment challenges. We demonstrate that horizon reduction is a key principle to address this limitation, stabilizing training and achieving better performance in long-horizon tasks. Moreover, we find that horizon reduction is related to stronger generalization across horizon lengths: models trained under reduced horizons generalize more effectively to longer-horizon variants at inference time, a phenomenon we refer to as horizon generalization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-kim26h, title = {On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length}, author = {Kim, Sunghwan and Cho, Junhee and Kwak, Beong-Woo and Kwon, Taeyoon and Wang, Liang and Yang, Nan and Zhang, Xingxing and Wei, Furu and Yeo, Jinyoung}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {57309--57337}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/kim26h/kim26h.pdf}, url = {https://proceedings.mlr.press/v306/kim26h.html}, abstract = {Large language models (LLMs) have shown promise as interactive agents that solve tasks through extended sequences of environment interactions. While prior work has primarily focused on system-level optimizations or algorithmic improvements, the role of task horizon length in shaping training dynamics remains poorly understood. In this work, we present a systematic empirical study that examines horizon length through controlled task constructions. Specifically, we construct controlled tasks in which agents face identical decision rules and reasoning structures, but differ only in the length of action sequences required for successful completion. Our results reveal that increasing horizon length alone constitutes a training bottleneck, inducing severe training instability driven by exploration difficulties and credit assignment challenges. We demonstrate that horizon reduction is a key principle to address this limitation, stabilizing training and achieving better performance in long-horizon tasks. Moreover, we find that horizon reduction is related to stronger generalization across horizon lengths: models trained under reduced horizons generalize more effectively to longer-horizon variants at inference time, a phenomenon we refer to as horizon generalization.} }
Endnote
%0 Conference Paper %T On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length %A Sunghwan Kim %A Junhee Cho %A Beong-Woo Kwak %A Taeyoon Kwon %A Liang Wang %A Nan Yang %A Xingxing Zhang %A Furu Wei %A Jinyoung Yeo %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-kim26h %I PMLR %P 57309--57337 %U https://proceedings.mlr.press/v306/kim26h.html %V 306 %X Large language models (LLMs) have shown promise as interactive agents that solve tasks through extended sequences of environment interactions. While prior work has primarily focused on system-level optimizations or algorithmic improvements, the role of task horizon length in shaping training dynamics remains poorly understood. In this work, we present a systematic empirical study that examines horizon length through controlled task constructions. Specifically, we construct controlled tasks in which agents face identical decision rules and reasoning structures, but differ only in the length of action sequences required for successful completion. Our results reveal that increasing horizon length alone constitutes a training bottleneck, inducing severe training instability driven by exploration difficulties and credit assignment challenges. We demonstrate that horizon reduction is a key principle to address this limitation, stabilizing training and achieving better performance in long-horizon tasks. Moreover, we find that horizon reduction is related to stronger generalization across horizon lengths: models trained under reduced horizons generalize more effectively to longer-horizon variants at inference time, a phenomenon we refer to as horizon generalization.
APA
Kim, S., Cho, J., Kwak, B., Kwon, T., Wang, L., Yang, N., Zhang, X., Wei, F. & Yeo, J.. (2026). On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:57309-57337 Available from https://proceedings.mlr.press/v306/kim26h.html.

Related Material