LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning

Sumeet Ramesh Motwani, Daniel Nichols, Charles London, Peggy Li, Fabio Pizzati, Acer Blake, Hasan Abed Al Kader Hammoud, Tavish Malcolm Mcdonald, Akshat Naik, Alesia Ivanova, Vignesh Baskaran, Ivan Laptev, Ruben Glatt, Tal Ben-Nun, Philip Torr, Ameya Prabhu, Brian R. Bartoldson, Bhavya Kailkhura, Christian Schroeder De Witt
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:90545-90579, 2026.

Abstract

As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this ability is planning and managing a long, complex chain-of-thought (CoT). We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models. Problems consist of a short input with a verifiable answer; solving them requires navigating a graph of interdependent steps that span tens to hundreds of thousands of reasoning tokens. Each local step is individually tractable for frontier models, so failures reflect long-horizon reasoning limitations. At release, the best models achieve $<$10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities. Overall, LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-motwani26a, title = {{L}ong{C}o{T}: Benchmarking Long-Horizon Chain-of-Thought Reasoning}, author = {Motwani, Sumeet Ramesh and Nichols, Daniel and London, Charles and Li, Peggy and Pizzati, Fabio and Blake, Acer and Hammoud, Hasan Abed Al Kader and Mcdonald, Tavish Malcolm and Naik, Akshat and Ivanova, Alesia and Baskaran, Vignesh and Laptev, Ivan and Glatt, Ruben and Ben-Nun, Tal and Torr, Philip and Prabhu, Ameya and Bartoldson, Brian R. and Kailkhura, Bhavya and Schroeder De Witt, Christian}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {90545--90579}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/motwani26a/motwani26a.pdf}, url = {https://proceedings.mlr.press/v306/motwani26a.html}, abstract = {As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this ability is planning and managing a long, complex chain-of-thought (CoT). We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models. Problems consist of a short input with a verifiable answer; solving them requires navigating a graph of interdependent steps that span tens to hundreds of thousands of reasoning tokens. Each local step is individually tractable for frontier models, so failures reflect long-horizon reasoning limitations. At release, the best models achieve $<$10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities. Overall, LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods.} }
Endnote
%0 Conference Paper %T LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning %A Sumeet Ramesh Motwani %A Daniel Nichols %A Charles London %A Peggy Li %A Fabio Pizzati %A Acer Blake %A Hasan Abed Al Kader Hammoud %A Tavish Malcolm Mcdonald %A Akshat Naik %A Alesia Ivanova %A Vignesh Baskaran %A Ivan Laptev %A Ruben Glatt %A Tal Ben-Nun %A Philip Torr %A Ameya Prabhu %A Brian R. Bartoldson %A Bhavya Kailkhura %A Christian Schroeder De Witt %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-motwani26a %I PMLR %P 90545--90579 %U https://proceedings.mlr.press/v306/motwani26a.html %V 306 %X As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this ability is planning and managing a long, complex chain-of-thought (CoT). We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models. Problems consist of a short input with a verifiable answer; solving them requires navigating a graph of interdependent steps that span tens to hundreds of thousands of reasoning tokens. Each local step is individually tractable for frontier models, so failures reflect long-horizon reasoning limitations. At release, the best models achieve $<$10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities. Overall, LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods.
APA
Motwani, S.R., Nichols, D., London, C., Li, P., Pizzati, F., Blake, A., Hammoud, H.A.A.K., Mcdonald, T.M., Naik, A., Ivanova, A., Baskaran, V., Laptev, I., Glatt, R., Ben-Nun, T., Torr, P., Prabhu, A., Bartoldson, B.R., Kailkhura, B. & Schroeder De Witt, C.. (2026). LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:90545-90579 Available from https://proceedings.mlr.press/v306/motwani26a.html.

Related Material