SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100k+ GPUs

Jin Lee, Zhonghao Chen, Xuhang He, Robert Underwood, Bogdan Nicolae, Franck Cappello, Xiaoyi Lu, Sheng Di, Zheng Zhang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:63450-63470, 2026.

Abstract

In large-scale LLM pretraining systems with $100\mathrm{k}+$ GPUs, failures become the norm rather than the exception, and restart costs can dominate wall-clock training time. However, existing fault-tolerance mechanisms are largely unprepared for this restart-dominant regime. To address this challenge, we propose SPARe—Stacked Parallelism with Adaptive Reordering—a fault-tolerance framework that masks node failures during gradient synchronization by stacking redundant data shards across parallelism groups and adaptively reordering execution. SPARe achieves availability comparable to traditional replication while maintaining near-constant computation overhead of only $2\sim3\times$, even under high redundancy where traidional replication would require linearly inflating overhead. We derive closed-form expressions for endurable failure count and computation overhead, validate them via SimGrid-based discrete-event simulation, and jointly optimize redundancy and checkpointing to minimize training time. At extreme scale with up to $600\mathrm{k}$ GPUs, SPARe reduces time-to-train by $40\sim50$% compared to traditional replication.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-lee26c, title = {{SPAR}e: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant {LLM} Pretraining Systems with 100k+ {GPU}s}, author = {Lee, Jin and Chen, Zhonghao and He, Xuhang and Underwood, Robert and Nicolae, Bogdan and Cappello, Franck and Lu, Xiaoyi and Di, Sheng and Zhang, Zheng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {63450--63470}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/lee26c/lee26c.pdf}, url = {https://proceedings.mlr.press/v306/lee26c.html}, abstract = {In large-scale LLM pretraining systems with $100\mathrm{k}+$ GPUs, failures become the norm rather than the exception, and restart costs can dominate wall-clock training time. However, existing fault-tolerance mechanisms are largely unprepared for this restart-dominant regime. To address this challenge, we propose SPARe—Stacked Parallelism with Adaptive Reordering—a fault-tolerance framework that masks node failures during gradient synchronization by stacking redundant data shards across parallelism groups and adaptively reordering execution. SPARe achieves availability comparable to traditional replication while maintaining near-constant computation overhead of only $2\sim3\times$, even under high redundancy where traidional replication would require linearly inflating overhead. We derive closed-form expressions for endurable failure count and computation overhead, validate them via SimGrid-based discrete-event simulation, and jointly optimize redundancy and checkpointing to minimize training time. At extreme scale with up to $600\mathrm{k}$ GPUs, SPARe reduces time-to-train by $40\sim50$% compared to traditional replication.} }
Endnote
%0 Conference Paper %T SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100k+ GPUs %A Jin Lee %A Zhonghao Chen %A Xuhang He %A Robert Underwood %A Bogdan Nicolae %A Franck Cappello %A Xiaoyi Lu %A Sheng Di %A Zheng Zhang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-lee26c %I PMLR %P 63450--63470 %U https://proceedings.mlr.press/v306/lee26c.html %V 306 %X In large-scale LLM pretraining systems with $100\mathrm{k}+$ GPUs, failures become the norm rather than the exception, and restart costs can dominate wall-clock training time. However, existing fault-tolerance mechanisms are largely unprepared for this restart-dominant regime. To address this challenge, we propose SPARe—Stacked Parallelism with Adaptive Reordering—a fault-tolerance framework that masks node failures during gradient synchronization by stacking redundant data shards across parallelism groups and adaptively reordering execution. SPARe achieves availability comparable to traditional replication while maintaining near-constant computation overhead of only $2\sim3\times$, even under high redundancy where traidional replication would require linearly inflating overhead. We derive closed-form expressions for endurable failure count and computation overhead, validate them via SimGrid-based discrete-event simulation, and jointly optimize redundancy and checkpointing to minimize training time. At extreme scale with up to $600\mathrm{k}$ GPUs, SPARe reduces time-to-train by $40\sim50$% compared to traditional replication.
APA
Lee, J., Chen, Z., He, X., Underwood, R., Nicolae, B., Cappello, F., Lu, X., Di, S. & Zhang, Z.. (2026). SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100k+ GPUs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:63450-63470 Available from https://proceedings.mlr.press/v306/lee26c.html.

Related Material