SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification

Kanghoon Yoon, Minsub Kim, Sungjae Lee, Joonhyung Lee, Sunghyeon Woo, Yeonjun In, Se Jung Kwon, Chanyoung Park, Dongsoo Lee
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:149677-149695, 2026.

Abstract

Speculative decoding accelerates LLM inference by verifying candidate tokens from a draft model against a larger target model. Recent "judge” decoding boosts this process by relaxing verification criteria by accepting draft tokens that may exhibit minor discrepancies from target model output, but existing methods are restricted by their reliance on human annotations or tasks with verifiable ground truths, limiting generalizability across diverse NLP tasks. We propose SelfJudge, which trains judge verifiers via self-supervision of the target model. Our method measures semantic preservation by assessing whether token-substituted responses preserve the meaning of original responses, enabling automatic verifier training across diverse NLP tasks. Our experiments show SelfJudge achieves superior inference-accuracy trade-offs than judge decoding baselines, offering a broadly applicable solution for faster LLM inference.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-yoon26d, title = {{S}elf{J}udge: Faster Speculative Decoding via Self-Supervised Judge Verification}, author = {Yoon, Kanghoon and Kim, Minsub and Lee, Sungjae and Lee, Joonhyung and Woo, Sunghyeon and In, Yeonjun and Kwon, Se Jung and Park, Chanyoung and Lee, Dongsoo}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {149677--149695}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yoon26d/yoon26d.pdf}, url = {https://proceedings.mlr.press/v306/yoon26d.html}, abstract = {Speculative decoding accelerates LLM inference by verifying candidate tokens from a draft model against a larger target model. Recent "judge” decoding boosts this process by relaxing verification criteria by accepting draft tokens that may exhibit minor discrepancies from target model output, but existing methods are restricted by their reliance on human annotations or tasks with verifiable ground truths, limiting generalizability across diverse NLP tasks. We propose SelfJudge, which trains judge verifiers via self-supervision of the target model. Our method measures semantic preservation by assessing whether token-substituted responses preserve the meaning of original responses, enabling automatic verifier training across diverse NLP tasks. Our experiments show SelfJudge achieves superior inference-accuracy trade-offs than judge decoding baselines, offering a broadly applicable solution for faster LLM inference.} }
Endnote
%0 Conference Paper %T SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification %A Kanghoon Yoon %A Minsub Kim %A Sungjae Lee %A Joonhyung Lee %A Sunghyeon Woo %A Yeonjun In %A Se Jung Kwon %A Chanyoung Park %A Dongsoo Lee %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yoon26d %I PMLR %P 149677--149695 %U https://proceedings.mlr.press/v306/yoon26d.html %V 306 %X Speculative decoding accelerates LLM inference by verifying candidate tokens from a draft model against a larger target model. Recent "judge” decoding boosts this process by relaxing verification criteria by accepting draft tokens that may exhibit minor discrepancies from target model output, but existing methods are restricted by their reliance on human annotations or tasks with verifiable ground truths, limiting generalizability across diverse NLP tasks. We propose SelfJudge, which trains judge verifiers via self-supervision of the target model. Our method measures semantic preservation by assessing whether token-substituted responses preserve the meaning of original responses, enabling automatic verifier training across diverse NLP tasks. Our experiments show SelfJudge achieves superior inference-accuracy trade-offs than judge decoding baselines, offering a broadly applicable solution for faster LLM inference.
APA
Yoon, K., Kim, M., Lee, S., Lee, J., Woo, S., In, Y., Kwon, S.J., Park, C. & Lee, D.. (2026). SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:149677-149695 Available from https://proceedings.mlr.press/v306/yoon26d.html.

Related Material