Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation

Alan Baade, Eric Ryan Chan, Kyle Sargent, Changan Chen, Justin Johnson, Ehsan Adeli, Li Fei-Fei
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:4694-4707, 2026.

Abstract

Latent diffusion models excel at generating high-quality images but lose the benefits of end-to-end modeling. They discard information during image encoding, require a separately trained decoder, and model an auxiliary distribution to the raw data. In this paper, we propose Latent Forcing, a simple modification to existing architectures that achieves the efficiency of latent diffusion while operating on raw natural images. Our approach orders the denoising trajectory by jointly processing latents and pixels with separately tuned noise schedules. This allows the latents to act as a scratchpad for intermediate computation before high-frequency pixel features are generated. We find that the order of conditioning signals is critical, and we analyze this to explain differences between REPA distillation in the tokenizer and the diffusion model, as well as conditional and unconditional generation. Applied to pixel-space diffusion on ImageNet, Latent Forcing achieves a new state of the art for diffusion transformer-based pixel generation at our compute scale. Code and checkpoints at https://github.com/AlanBaade/LatentForcing

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-baade26a, title = {Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation}, author = {Baade, Alan and Chan, Eric Ryan and Sargent, Kyle and Chen, Changan and Johnson, Justin and Adeli, Ehsan and Fei-Fei, Li}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {4694--4707}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/baade26a/baade26a.pdf}, url = {https://proceedings.mlr.press/v306/baade26a.html}, abstract = {Latent diffusion models excel at generating high-quality images but lose the benefits of end-to-end modeling. They discard information during image encoding, require a separately trained decoder, and model an auxiliary distribution to the raw data. In this paper, we propose Latent Forcing, a simple modification to existing architectures that achieves the efficiency of latent diffusion while operating on raw natural images. Our approach orders the denoising trajectory by jointly processing latents and pixels with separately tuned noise schedules. This allows the latents to act as a scratchpad for intermediate computation before high-frequency pixel features are generated. We find that the order of conditioning signals is critical, and we analyze this to explain differences between REPA distillation in the tokenizer and the diffusion model, as well as conditional and unconditional generation. Applied to pixel-space diffusion on ImageNet, Latent Forcing achieves a new state of the art for diffusion transformer-based pixel generation at our compute scale. Code and checkpoints at https://github.com/AlanBaade/LatentForcing} }
Endnote
%0 Conference Paper %T Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation %A Alan Baade %A Eric Ryan Chan %A Kyle Sargent %A Changan Chen %A Justin Johnson %A Ehsan Adeli %A Li Fei-Fei %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-baade26a %I PMLR %P 4694--4707 %U https://proceedings.mlr.press/v306/baade26a.html %V 306 %X Latent diffusion models excel at generating high-quality images but lose the benefits of end-to-end modeling. They discard information during image encoding, require a separately trained decoder, and model an auxiliary distribution to the raw data. In this paper, we propose Latent Forcing, a simple modification to existing architectures that achieves the efficiency of latent diffusion while operating on raw natural images. Our approach orders the denoising trajectory by jointly processing latents and pixels with separately tuned noise schedules. This allows the latents to act as a scratchpad for intermediate computation before high-frequency pixel features are generated. We find that the order of conditioning signals is critical, and we analyze this to explain differences between REPA distillation in the tokenizer and the diffusion model, as well as conditional and unconditional generation. Applied to pixel-space diffusion on ImageNet, Latent Forcing achieves a new state of the art for diffusion transformer-based pixel generation at our compute scale. Code and checkpoints at https://github.com/AlanBaade/LatentForcing
APA
Baade, A., Chan, E.R., Sargent, K., Chen, C., Johnson, J., Adeli, E. & Fei-Fei, L.. (2026). Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:4694-4707 Available from https://proceedings.mlr.press/v306/baade26a.html.

Related Material