Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens

Wei-Lin Chen, Liqian Peng, Tian Tan, Chao Zhao, Jianhang Chen, Ziqian Lin, Alec Go, Yu Meng
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:16342-16360, 2026.

Abstract

Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal “overthinking,” leading to performance degradation. In this work, we quantify inference-time effort by identifying deep-thinking tokens—tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that deep-thinking ratio (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@$n$, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@$n$ matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26di, title = {Think Deep, Not Just Long: Measuring {LLM} Reasoning Effort via Deep-Thinking Tokens}, author = {Chen, Wei-Lin and Peng, Liqian and Tan, Tian and Zhao, Chao and Chen, Jianhang and Lin, Ziqian and Go, Alec and Meng, Yu}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {16342--16360}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26di/chen26di.pdf}, url = {https://proceedings.mlr.press/v306/chen26di.html}, abstract = {Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal “overthinking,” leading to performance degradation. In this work, we quantify inference-time effort by identifying deep-thinking tokens—tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that deep-thinking ratio (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@$n$, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@$n$ matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.} }
Endnote
%0 Conference Paper %T Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens %A Wei-Lin Chen %A Liqian Peng %A Tian Tan %A Chao Zhao %A Jianhang Chen %A Ziqian Lin %A Alec Go %A Yu Meng %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26di %I PMLR %P 16342--16360 %U https://proceedings.mlr.press/v306/chen26di.html %V 306 %X Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal “overthinking,” leading to performance degradation. In this work, we quantify inference-time effort by identifying deep-thinking tokens—tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that deep-thinking ratio (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@$n$, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@$n$ matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.
APA
Chen, W., Peng, L., Tan, T., Zhao, C., Chen, J., Lin, Z., Go, A. & Meng, Y.. (2026). Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:16342-16360 Available from https://proceedings.mlr.press/v306/chen26di.html.

Related Material