Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?

Xinchen Yan, Chen Liang, Lijun Yu, Adams Wei Yu, Yifeng Lu, Quoc V Le
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:143795-143815, 2026.

Abstract

This paper investigates the scaling properties of autoregressive next-pixel prediction, a simple, end-to-end yet under-explored framework for unified vision models. Starting with images at resolutions of 32$\times$ 32, we train a family of Transformers using IsoFLOP profiles across compute budgets up to 7e19 FLOPs and evaluate three distinct target metrics: next-pixel prediction objective, ImageNet classification accuracy, and generation-based completion where top half of the image serves as a spatial prompt. First, optimal scaling strategy is critically task-dependent. At a fixed resolution of 32 $\times$ 32 alone, the optimal scaling properties for image classification and image generation diverge, where generation optimal setup requires the data size grow three to five times faster than for the classification optimal setup. Second, as image resolution increases, the optimal scaling strategy indicates that the model size must grow much faster than data size. Surprisingly, by projecting our findings, we discover that the primary bottleneck is compute rather than the amount of training data. As compute continues to grow four to five times annually, we forecast the feasibility of pixel-by-pixel modeling of images within the next five years.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-yan26j, title = {Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?}, author = {Yan, Xinchen and Liang, Chen and Yu, Lijun and Yu, Adams Wei and Lu, Yifeng and Le, Quoc V}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {143795--143815}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yan26j/yan26j.pdf}, url = {https://proceedings.mlr.press/v306/yan26j.html}, abstract = {This paper investigates the scaling properties of autoregressive next-pixel prediction, a simple, end-to-end yet under-explored framework for unified vision models. Starting with images at resolutions of 32$\times$ 32, we train a family of Transformers using IsoFLOP profiles across compute budgets up to 7e19 FLOPs and evaluate three distinct target metrics: next-pixel prediction objective, ImageNet classification accuracy, and generation-based completion where top half of the image serves as a spatial prompt. First, optimal scaling strategy is critically task-dependent. At a fixed resolution of 32 $\times$ 32 alone, the optimal scaling properties for image classification and image generation diverge, where generation optimal setup requires the data size grow three to five times faster than for the classification optimal setup. Second, as image resolution increases, the optimal scaling strategy indicates that the model size must grow much faster than data size. Surprisingly, by projecting our findings, we discover that the primary bottleneck is compute rather than the amount of training data. As compute continues to grow four to five times annually, we forecast the feasibility of pixel-by-pixel modeling of images within the next five years.} }
Endnote
%0 Conference Paper %T Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction? %A Xinchen Yan %A Chen Liang %A Lijun Yu %A Adams Wei Yu %A Yifeng Lu %A Quoc V Le %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yan26j %I PMLR %P 143795--143815 %U https://proceedings.mlr.press/v306/yan26j.html %V 306 %X This paper investigates the scaling properties of autoregressive next-pixel prediction, a simple, end-to-end yet under-explored framework for unified vision models. Starting with images at resolutions of 32$\times$ 32, we train a family of Transformers using IsoFLOP profiles across compute budgets up to 7e19 FLOPs and evaluate three distinct target metrics: next-pixel prediction objective, ImageNet classification accuracy, and generation-based completion where top half of the image serves as a spatial prompt. First, optimal scaling strategy is critically task-dependent. At a fixed resolution of 32 $\times$ 32 alone, the optimal scaling properties for image classification and image generation diverge, where generation optimal setup requires the data size grow three to five times faster than for the classification optimal setup. Second, as image resolution increases, the optimal scaling strategy indicates that the model size must grow much faster than data size. Surprisingly, by projecting our findings, we discover that the primary bottleneck is compute rather than the amount of training data. As compute continues to grow four to five times annually, we forecast the feasibility of pixel-by-pixel modeling of images within the next five years.
APA
Yan, X., Liang, C., Yu, L., Yu, A.W., Lu, Y. & Le, Q.V.. (2026). Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:143795-143815 Available from https://proceedings.mlr.press/v306/yan26j.html.

Related Material