AdaHC: Accelerating Multi-Token Prediction with Adaptive Head Chunking with Pipeline Parallelism

Yan Wang, Chang Si, Kaiming Yang, Zhipeng Zhang, Weijian Liu, Man Yuan, Mingzhen Li, Yong Li, Weile Jia
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:129892-129907, 2026.

Abstract

Multi-token prediction (MTP) architecture is widely adopted in LLMs. MTP blocks can be appended to the tail of model to predict additional tokens. However, when training with pipeline parallel, MTP leads to more pipeline bubbles and deteriorates the pipeline efficiency. Based on in-depth analysis of MTP architectures and loss functions, we have identified the parallel nature of the MTP blocks, and leverage it for superior pipeline scheduling. We propose AdaHC, an adaptive pipeline scheduling framework for accelerating LLMs training with MTP block(s). AdaHC splits the output heads into chunks and reassembles the chunks to generate balanced pipeline stages, and performs adaptive activation forwarding to preserve the numerical equivalence. Experimental results show that AdaHC improves the training throughput of SOTA LLMs with diverse MTP configurations by 1.35$\times$ on average. This work paves a new direction for practical pipeline training.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-wang26gy, title = {{A}da{HC}: Accelerating Multi-Token Prediction with Adaptive Head Chunking with Pipeline Parallelism}, author = {Wang, Yan and Si, Chang and Yang, Kaiming and Zhang, Zhipeng and Liu, Weijian and Yuan, Man and Li, Mingzhen and Li, Yong and Jia, Weile}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {129892--129907}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wang26gy/wang26gy.pdf}, url = {https://proceedings.mlr.press/v306/wang26gy.html}, abstract = {Multi-token prediction (MTP) architecture is widely adopted in LLMs. MTP blocks can be appended to the tail of model to predict additional tokens. However, when training with pipeline parallel, MTP leads to more pipeline bubbles and deteriorates the pipeline efficiency. Based on in-depth analysis of MTP architectures and loss functions, we have identified the parallel nature of the MTP blocks, and leverage it for superior pipeline scheduling. We propose AdaHC, an adaptive pipeline scheduling framework for accelerating LLMs training with MTP block(s). AdaHC splits the output heads into chunks and reassembles the chunks to generate balanced pipeline stages, and performs adaptive activation forwarding to preserve the numerical equivalence. Experimental results show that AdaHC improves the training throughput of SOTA LLMs with diverse MTP configurations by 1.35$\times$ on average. This work paves a new direction for practical pipeline training.} }
Endnote
%0 Conference Paper %T AdaHC: Accelerating Multi-Token Prediction with Adaptive Head Chunking with Pipeline Parallelism %A Yan Wang %A Chang Si %A Kaiming Yang %A Zhipeng Zhang %A Weijian Liu %A Man Yuan %A Mingzhen Li %A Yong Li %A Weile Jia %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wang26gy %I PMLR %P 129892--129907 %U https://proceedings.mlr.press/v306/wang26gy.html %V 306 %X Multi-token prediction (MTP) architecture is widely adopted in LLMs. MTP blocks can be appended to the tail of model to predict additional tokens. However, when training with pipeline parallel, MTP leads to more pipeline bubbles and deteriorates the pipeline efficiency. Based on in-depth analysis of MTP architectures and loss functions, we have identified the parallel nature of the MTP blocks, and leverage it for superior pipeline scheduling. We propose AdaHC, an adaptive pipeline scheduling framework for accelerating LLMs training with MTP block(s). AdaHC splits the output heads into chunks and reassembles the chunks to generate balanced pipeline stages, and performs adaptive activation forwarding to preserve the numerical equivalence. Experimental results show that AdaHC improves the training throughput of SOTA LLMs with diverse MTP configurations by 1.35$\times$ on average. This work paves a new direction for practical pipeline training.
APA
Wang, Y., Si, C., Yang, K., Zhang, Z., Liu, W., Yuan, M., Li, M., Li, Y. & Jia, W.. (2026). AdaHC: Accelerating Multi-Token Prediction with Adaptive Head Chunking with Pipeline Parallelism. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:129892-129907 Available from https://proceedings.mlr.press/v306/wang26gy.html.

Related Material