SpecExit: Accelerating Large Reasoning Model via Speculative Exit

Rubing Yang, Huajun Bai, Song Liu, Guanghua Yu, Runzhi Fan, Yanbin Dang, Zhang Jiejing, Kai Liu, Jianchen Zhu, Peng Chen
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:144187-144210, 2026.

Abstract

Despite their strong performance on reasoning tasks, Large reasoning models (LRMs) often suffer from overthinking, producing unnecessarily long outputs and incurring high end-to-end latency, a significant limitation to their real-world deployment. To address overthinking, early-exit mechanisms have been proposed to terminate reasoning before typical completion, showing that this approach can effectively shorten generation length with minimal impact on accuracy. However, their reliance on probing mechanisms introduces a detection overhead that limits their end-to-end latency gains and compromises their generalizability across diverse problems. Inspired by the use of hidden states in speculative decoding, we propose SpecExit, a novel framework that predicts both future tokens and an early-exit signal directly from a lightweight draft model without probing overhead. Our method offers significant improvements, achieving up to 66% generation length reduction and 2.5$\times$ end-to-end speedup compared with the speculative decoding baseline, without compromising accuracy. Our method leverages the inherent signals from hidden states to provide effective early-exit signals, suggesting broader use of hidden states for efficient reasoning. Our code is available at: https://anonymous.4open.science/r/SpecExit-B802.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-yang26a, title = {{S}pec{E}xit: Accelerating Large Reasoning Model via Speculative Exit}, author = {Yang, Rubing and Bai, Huajun and Liu, Song and Yu, Guanghua and Fan, Runzhi and Dang, Yanbin and Jiejing, Zhang and Liu, Kai and Zhu, Jianchen and Chen, Peng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {144187--144210}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yang26a/yang26a.pdf}, url = {https://proceedings.mlr.press/v306/yang26a.html}, abstract = {Despite their strong performance on reasoning tasks, Large reasoning models (LRMs) often suffer from overthinking, producing unnecessarily long outputs and incurring high end-to-end latency, a significant limitation to their real-world deployment. To address overthinking, early-exit mechanisms have been proposed to terminate reasoning before typical completion, showing that this approach can effectively shorten generation length with minimal impact on accuracy. However, their reliance on probing mechanisms introduces a detection overhead that limits their end-to-end latency gains and compromises their generalizability across diverse problems. Inspired by the use of hidden states in speculative decoding, we propose SpecExit, a novel framework that predicts both future tokens and an early-exit signal directly from a lightweight draft model without probing overhead. Our method offers significant improvements, achieving up to 66% generation length reduction and 2.5$\times$ end-to-end speedup compared with the speculative decoding baseline, without compromising accuracy. Our method leverages the inherent signals from hidden states to provide effective early-exit signals, suggesting broader use of hidden states for efficient reasoning. Our code is available at: https://anonymous.4open.science/r/SpecExit-B802.} }
Endnote
%0 Conference Paper %T SpecExit: Accelerating Large Reasoning Model via Speculative Exit %A Rubing Yang %A Huajun Bai %A Song Liu %A Guanghua Yu %A Runzhi Fan %A Yanbin Dang %A Zhang Jiejing %A Kai Liu %A Jianchen Zhu %A Peng Chen %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yang26a %I PMLR %P 144187--144210 %U https://proceedings.mlr.press/v306/yang26a.html %V 306 %X Despite their strong performance on reasoning tasks, Large reasoning models (LRMs) often suffer from overthinking, producing unnecessarily long outputs and incurring high end-to-end latency, a significant limitation to their real-world deployment. To address overthinking, early-exit mechanisms have been proposed to terminate reasoning before typical completion, showing that this approach can effectively shorten generation length with minimal impact on accuracy. However, their reliance on probing mechanisms introduces a detection overhead that limits their end-to-end latency gains and compromises their generalizability across diverse problems. Inspired by the use of hidden states in speculative decoding, we propose SpecExit, a novel framework that predicts both future tokens and an early-exit signal directly from a lightweight draft model without probing overhead. Our method offers significant improvements, achieving up to 66% generation length reduction and 2.5$\times$ end-to-end speedup compared with the speculative decoding baseline, without compromising accuracy. Our method leverages the inherent signals from hidden states to provide effective early-exit signals, suggesting broader use of hidden states for efficient reasoning. Our code is available at: https://anonymous.4open.science/r/SpecExit-B802.
APA
Yang, R., Bai, H., Liu, S., Yu, G., Fan, R., Dang, Y., Jiejing, Z., Liu, K., Zhu, J. & Chen, P.. (2026). SpecExit: Accelerating Large Reasoning Model via Speculative Exit. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:144187-144210 Available from https://proceedings.mlr.press/v306/yang26a.html.

Related Material