NITP: Next Implicit Token Prediction for LLM Pre-training

Xiangdong Zhang, Debing Zhang, Shaofeng Zhang, Xiaohan Qin, Yu Cheng, Junchi Yan
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:160773-160793, 2026.

Abstract

Standard Next-Token Prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse, one-hot supervision leaves the latent representation space under-constrained, allowing hidden states to drift into degenerate and anisotropic configurations that limit generalization. To address this issue, we propose Next Implicit Token Prediction (NITP), which augments discrete prediction with dense, continuous supervision directly in the representation space. NITP requires the model to predict the implicit semantic content of the next token, using shallow-layer representations from the same model as stable self-supervised targets. Theoretically, we show that NITP regularizes the optimization landscape by mitigating under-constrained degrees of freedom and enforcing a compact, structured representation geometry. Empirically, across dense and MoE models ranging from 0.5B to 9B parameters, NITP consistently improves downstream performance with negligible computational overhead. Notably, on the 9B MoE model, NITP achieves a 5.7% absolute improvement on MMLU-Pro, along with gains of 6.4% on C3 and 4.3% on CommonsenseQA, with $\sim$2% additional training FLOPs and no additional inference cost. Our implementation is available at https://github.com/aHapBean/NITP.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zhang26ix, title = {{NITP}: Next Implicit Token Prediction for {LLM} Pre-training}, author = {Zhang, Xiangdong and Zhang, Debing and Zhang, Shaofeng and Qin, Xiaohan and Cheng, Yu and Yan, Junchi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {160773--160793}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26ix/zhang26ix.pdf}, url = {https://proceedings.mlr.press/v306/zhang26ix.html}, abstract = {Standard Next-Token Prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse, one-hot supervision leaves the latent representation space under-constrained, allowing hidden states to drift into degenerate and anisotropic configurations that limit generalization. To address this issue, we propose Next Implicit Token Prediction (NITP), which augments discrete prediction with dense, continuous supervision directly in the representation space. NITP requires the model to predict the implicit semantic content of the next token, using shallow-layer representations from the same model as stable self-supervised targets. Theoretically, we show that NITP regularizes the optimization landscape by mitigating under-constrained degrees of freedom and enforcing a compact, structured representation geometry. Empirically, across dense and MoE models ranging from 0.5B to 9B parameters, NITP consistently improves downstream performance with negligible computational overhead. Notably, on the 9B MoE model, NITP achieves a 5.7% absolute improvement on MMLU-Pro, along with gains of 6.4% on C3 and 4.3% on CommonsenseQA, with $\sim$2% additional training FLOPs and no additional inference cost. Our implementation is available at https://github.com/aHapBean/NITP.} }
Endnote
%0 Conference Paper %T NITP: Next Implicit Token Prediction for LLM Pre-training %A Xiangdong Zhang %A Debing Zhang %A Shaofeng Zhang %A Xiaohan Qin %A Yu Cheng %A Junchi Yan %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26ix %I PMLR %P 160773--160793 %U https://proceedings.mlr.press/v306/zhang26ix.html %V 306 %X Standard Next-Token Prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse, one-hot supervision leaves the latent representation space under-constrained, allowing hidden states to drift into degenerate and anisotropic configurations that limit generalization. To address this issue, we propose Next Implicit Token Prediction (NITP), which augments discrete prediction with dense, continuous supervision directly in the representation space. NITP requires the model to predict the implicit semantic content of the next token, using shallow-layer representations from the same model as stable self-supervised targets. Theoretically, we show that NITP regularizes the optimization landscape by mitigating under-constrained degrees of freedom and enforcing a compact, structured representation geometry. Empirically, across dense and MoE models ranging from 0.5B to 9B parameters, NITP consistently improves downstream performance with negligible computational overhead. Notably, on the 9B MoE model, NITP achieves a 5.7% absolute improvement on MMLU-Pro, along with gains of 6.4% on C3 and 4.3% on CommonsenseQA, with $\sim$2% additional training FLOPs and no additional inference cost. Our implementation is available at https://github.com/aHapBean/NITP.
APA
Zhang, X., Zhang, D., Zhang, S., Qin, X., Cheng, Y. & Yan, J.. (2026). NITP: Next Implicit Token Prediction for LLM Pre-training. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:160773-160793 Available from https://proceedings.mlr.press/v306/zhang26ix.html.

Related Material