Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch

Ziyang Zhang, Xinheng Ding, Jiayi Yuan, Rixin Liu, Huizi Mao, Jiarong Xing, Zirui Liu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:155026-155042, 2026.

Abstract

Deterministic inference is increasingly critical for Large language model (LLM) applications such as LLM-as-a-judge evaluation, multi-agent systems, and reinforcement learning (RL). However, existing LLM serving frameworks can produce different outputs for identical inputs when tensor parallel (TP) size or batch size changes, even under greedy decoding. This arises from the non-associativity of floating-point arithmetic and inconsistent reduction orders across GPUs. While prior work has addressed batch-size–related nondeterminism through batch-invariant kernels, determinism across different TP sizes remains an open problem, particularly in RL settings, where the training engine typically uses Fully Sharded Data Parallel (FSDP) (i.e., TP = 1) while the rollout engine relies on multi-GPU TP to maximize the inference throughput, creating a probability mismatch that can degrade or even collapse training. We identify and analyze the root causes of TP-induced inconsistency and propose Tree-Based Invariant Kernels (TBIK), a set of custom matrix multiplication and reduction kernels that guarantee bit-wise identical results across TP sizes. Our key insight is to enforce a consistent reduction order across and within GPUs. We implement TBIK in Triton and integrate it into vLLM and FSDP, achieving bit-wise deterministic inference across different TP sizes and zero probability divergence between rollout and training engines in RL pipelines. By eliminating mismatches caused by different parallelization strategies, TBIK enables true on-policy RL at scale for the first time, leading to improved model performance and faster convergence.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-zhang26ag, title = {Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch}, author = {Zhang, Ziyang and Ding, Xinheng and Yuan, Jiayi and Liu, Rixin and Mao, Huizi and Xing, Jiarong and Liu, Zirui}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {155026--155042}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26ag/zhang26ag.pdf}, url = {https://proceedings.mlr.press/v306/zhang26ag.html}, abstract = {Deterministic inference is increasingly critical for Large language model (LLM) applications such as LLM-as-a-judge evaluation, multi-agent systems, and reinforcement learning (RL). However, existing LLM serving frameworks can produce different outputs for identical inputs when tensor parallel (TP) size or batch size changes, even under greedy decoding. This arises from the non-associativity of floating-point arithmetic and inconsistent reduction orders across GPUs. While prior work has addressed batch-size–related nondeterminism through batch-invariant kernels, determinism across different TP sizes remains an open problem, particularly in RL settings, where the training engine typically uses Fully Sharded Data Parallel (FSDP) (i.e., TP = 1) while the rollout engine relies on multi-GPU TP to maximize the inference throughput, creating a probability mismatch that can degrade or even collapse training. We identify and analyze the root causes of TP-induced inconsistency and propose Tree-Based Invariant Kernels (TBIK), a set of custom matrix multiplication and reduction kernels that guarantee bit-wise identical results across TP sizes. Our key insight is to enforce a consistent reduction order across and within GPUs. We implement TBIK in Triton and integrate it into vLLM and FSDP, achieving bit-wise deterministic inference across different TP sizes and zero probability divergence between rollout and training engines in RL pipelines. By eliminating mismatches caused by different parallelization strategies, TBIK enables true on-policy RL at scale for the first time, leading to improved model performance and faster convergence.} }
Endnote
%0 Conference Paper %T Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch %A Ziyang Zhang %A Xinheng Ding %A Jiayi Yuan %A Rixin Liu %A Huizi Mao %A Jiarong Xing %A Zirui Liu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26ag %I PMLR %P 155026--155042 %U https://proceedings.mlr.press/v306/zhang26ag.html %V 306 %X Deterministic inference is increasingly critical for Large language model (LLM) applications such as LLM-as-a-judge evaluation, multi-agent systems, and reinforcement learning (RL). However, existing LLM serving frameworks can produce different outputs for identical inputs when tensor parallel (TP) size or batch size changes, even under greedy decoding. This arises from the non-associativity of floating-point arithmetic and inconsistent reduction orders across GPUs. While prior work has addressed batch-size–related nondeterminism through batch-invariant kernels, determinism across different TP sizes remains an open problem, particularly in RL settings, where the training engine typically uses Fully Sharded Data Parallel (FSDP) (i.e., TP = 1) while the rollout engine relies on multi-GPU TP to maximize the inference throughput, creating a probability mismatch that can degrade or even collapse training. We identify and analyze the root causes of TP-induced inconsistency and propose Tree-Based Invariant Kernels (TBIK), a set of custom matrix multiplication and reduction kernels that guarantee bit-wise identical results across TP sizes. Our key insight is to enforce a consistent reduction order across and within GPUs. We implement TBIK in Triton and integrate it into vLLM and FSDP, achieving bit-wise deterministic inference across different TP sizes and zero probability divergence between rollout and training engines in RL pipelines. By eliminating mismatches caused by different parallelization strategies, TBIK enables true on-policy RL at scale for the first time, leading to improved model performance and faster convergence.
APA
Zhang, Z., Ding, X., Yuan, J., Liu, R., Mao, H., Xing, J. & Liu, Z.. (2026). Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:155026-155042 Available from https://proceedings.mlr.press/v306/zhang26ag.html.

Related Material